I am working on the file "DataPreprocessing_results.ipynb".
The first task is:
When working with CNN, we should add an axis for the channels: 3 (R,G,B) for image processing and in 1 in this case as the spectrogram can be regarded as a monochromatic image. Thus, we use np.newaxis to add a new axis.
Different deep learning frameworks require different order for the axes. You can change it with np.swapaxes .
We can plot the first example, which is different every time because of the batch shuffling.
Can you write Python code for it?


I am working on the file "DataQuickLook/Quicklook NuSTAR data with Stingray.ipynb".
The first task is:
Nice Quasi-periodic oscillations there! Note that at high frequencies the white noise level increases. This is not real variability, but an effect of dead time. The easiest way to get a flat periodogram at high frequencies is using the cospectrum instead of the power density spectrum. For this, we use separately the events from the two detectors. The cospectrum calculation is slightly slower than the power spectrum.
For an accurate way to correct the power density spectrum from dead time, see the documentation of stingray.deadtime and the Frequency Amplitude Difference (FAD) correction.
Can you write Python code for it?

cs = AveragedCrossspectrum.from_events(evA, evB, segment_size=256, dt=0.001, norm='leahy')

plt.figure(figsize=(10,7))
plt.semilogx(cs.freq, cs.power.real)


I am working on the file "DataQuickLook/Quicklook NuSTAR data with Stingray.ipynb".
The first task is:
Quicklook NuSTAR data with Stingray
Let us load the data from two event lists corresponding to the two detectors onboard NuSTAR. fmt='hea' indicates event data produced by HEAsoft tools or compatible (e.g. XMM-Newton).
Can you write Python code for it?

evA = EventList.read('nustar_A_src.evt', 'hea')
evB = EventList.read('nustar_B_src.evt', 'hea')


I am working on the file "DataQuickLook/Quicklook NuSTAR data with Stingray.ipynb".
The first task is:
To improve the plot, we can rebin the data logarithmically
Can you write Python code for it?

cs_reb = cs.rebin_log(0.02)

plt.figure(figsize=(10,7))
plt.loglog(cs_reb.freq, cs_reb.power.real)
plt.ylim([1e-3, None])
plt.xlabel("Frequency")
plt.ylabel("Cospectrum Power")


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
**kwargs
Similarly, we can define a dictionary of named parameters
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
*args and **kwargs
In Python, there is a special deferencing scheme that allows for defining and calling functions with argument lists or dictionaries.
*args
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
fillna is a dataframe function that replaces all nan values with either a scalar value, a series of values with the same indices as found in the dataframe, or a dataframe that is indexed by the columns of the target dataframe.
cancer_data.mean().round() will take the mean of each column (this computation ignores the currently present nan values), then round, and return a dataframe indexed by the columns of the original dataframe:
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
A plot begins with the ggplot function.  Here, we pass in the cancer_data pandas dataframe and a special function called aes (short for aesthetic).  The values provided to aes change depending on which type of plot is being used.  Here we are going to make a histogram from the clump_thickness column in cancer_data, so that column name needs to be passed as the x parameter to aes.
The grammar of graphics is based off of a concept of "geoms" (short for geometric objects).  These geoms provide granular control of the plot and are progressively added to the base call to ggplot with + syntax.
Lets say we wanted to show the mean clump_thickness on this plot.  We could do something like the following
Can you write Python code for it?

plt = ggplot(aes(x = 'clump_thickness'), data = cancer_data) + \
        geom_histogram(binwidth = 1, fill = 'steelblue') + \
        geom_vline(xintercept = [cancer_data['clump_thickness'].mean()], linetype='dashed')
print plt


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
An important concept in machine learning is to split the data set into training data and testing data.  The machine learning algorithm will use the subset of training data to build a classifier to predict labels.  We then test the accuracy of this classifier on the subset of testing data.  This is done in order to prevent overfitting the classifier to one given set of data.
Overfitting is a major concern in the design of machine learning algorithms.  Conceptually, overfitting is when a classifier is really good at predicting the data used to build it, but isn't robust or general enough to predict new, yet unseen data all that well.
To perform machine learning, we will use a package called sci-kit learn (sklearn for short).  The sklearn cross_validation module contains a function called train_test_split that will take in features and labels, and randomly select values into the training and testing subsets
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

features_train, features_test, labels_train, labels_test = train_test_split(features,
                                                                            labels,
                                                                            test_size = 0.3,
                                                                            random_state = 42)


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
And make a call to download_file
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Arbitrary indentation can be used within a code block, as long as the indentation is consistent.
Can you write Python code for it?

if (1 == 1):
    print "We're in "
        print "Deep Trouble:"


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
As you can see, each geom has its own set of parameters specific to the appearance of that geom (also called aesthetics).
Lets try a scatter plot to get some multi-variable action
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei'), data = cancer_data) + \
        geom_point()
print plt


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Conditionals
Python supports the standard if-else-if conditional expression
Can you write Python code for it?

a = 2; b = 3

if a < b:
    print "a is less than b"

if a > b:
    print "a is greater than b"

else:
    print "a is less than or equal to b"

a = 3; b = 3

if a < b:
    print "a is less than b"
elif a > b:
    print "a is greater than b"
else:
    print "a is equal to b"


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Dictionaries
Python supports dictionaries which can be thought of as an unordered list of key, value pairs.  Keys can be any immutable type and are typically integers or strings.  Values can be any object, even other dictionaries.
Dictionaries are created with curly braces {}
Can you write Python code for it?

# mydict

# Florida

# Deletes

# Assignment

# Appending

# Keys


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Foreach
As it turns out, counting loops are just foreach loops in Python.  The range function returns a list of integers over which for in iterates.  This can be extended to any other iterable type
Can you write Python code for it?

for asdf in [1,["a","b","c"],3,4,5,"potato"]:
    print asdf

anotherdict = {"one" : 1, "two" : 2, "three" : 3}

for key in anotherdict.keys():
    print key, anotherdict[key]


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Functions can have optional arguments if a default value is provided in the function signature
Can you write Python code for it?

def player(name, number, team = 'Florida'): # optional team argument
    print "#" + str(number), name, team

player("Kasey Hill", 0) # no team argument supplied

player("Aaron Harrison", 2, "Kentucky") # supplying all three arguments


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Here we call values on the dataframe to extract the values stored in the dataframe as an array of numpy arrays with the same dimensions as our subsetted dataframe.  Numpy is a powerful, high performance scientific computing package that implements arrays.  It is used internally by pandas.  We will use labels and features later on in our machine learning classifier
Can you write Python code for it?

labels = cancer_data['class'].values
features = cancer_data[cancer_features].values


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Here we have attempted to convert the bare_nuclei series to a numeric type.  Lets see what the unique values are now.
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Just as before, we can define a function taking an arbitrary dictionary
Can you write Python code for it?

def foo(**kwargs):
    for key in kwargs.keys():
        print key, kwargs[key]

foo(**kwargs)


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Lets take a look at some simple statistics for the clump_thickness column
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Looks like nothing was found. -1 is returned by default. We'll talk about -1 when we talk about truth later.
lower and upper
Can you write Python code for it?

print mystring.lower()
print mystring.upper()


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Loops
Python supports for, foreach, and while loops
For (counting)
Traditional counting loops are accomplished in Python with a combination of the for key word and the range function
Can you write Python code for it?

for x in range(10):
    print x

for x in range(2,10):
    print x

for x in range(2,10,2):
    print x

for a in range(10,0,-1):
    print a

for b in range(2, 1, 1): # with a positive step, all values are less than 1. No integers are produced
    print b

for c in range(1, 2, -1): # same goes for a negative step as all values are less than 2
    print c


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Modules and Import
Files with a .py extension are known as Modules in Python.  Modules are used to store functions, variables, and class definitions.
Modules that are not part of the standard Python library are included in your program using the import statement.
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Next, we create a variable to store the classifier
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Notice that replace returned a new string every time.  Nothing was modified in place. Strings (and most basic data types) are immutable in Python.
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Notice that the method is called on the string, not on the list. Lists don't have a join method. This is discussed often on StackOverflow.
For more information on string functions:
https://docs.python.org/2/library/stdtypes.html#string-methods
Data Structures
Lists
The Python standard library does not have traditional C-style fixed-memory fixed-type arrays.  Instead, lists are used and can contain a mix of any type.
Lists are created with square brackets []
Lists are also mutable, we can change them.
Can you write Python code for it?

# mylist List of 5

# append 6

mylist.extend([8, 'nine']) # extend the list with the contents of another list
print mylist

mylist.insert(6, 7) # insert the number 7 at index 6
print mylist
# Insert 20 at 20

mylist.remove('five') # removes the first matching occurence
print mylist

popped = mylist.pop() # by default, the last item in the list is removed and returned
print popped
print mylist

popped2 = mylist.pop(4) # pops at at index
print popped2
print mylist

# len()

mylist.extend(range(-3, 0)) # the range function returns a list from -3 inclusive to 0 non inclusive
print mylist

# default list sorting. When more complex objects are in the list, arguments can be used to customize how to sort
mylist.sort()
print mylist

mylist.reverse() # reverse the list
print mylist


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Now this might seem like overkill for downloading a single, small csv file, but we can use this same function to access countless APIs available on the World Wide Web by building an API request in the url.
Wrangling the Data
Now that we have some data, lets get it into a useful form.  For this task we will use a package called pandas. pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for Python.  The most fundamental data structure in pandas is the dataframe, which is similar to the data.frame data structure found in the R statistical programming language.
For more information: http://pandas.pydata.org
pandas dataframes are a 2-dimensional labeled data structures with columns of potentially different types.  Dataframes can be thought of as similar to a spreadsheet or SQL table.
There are numerous ways to build a dataframe with pandas.  Since we have already attained a csv file, we can use a parser built into pandas called read_csv which will read the contents of a csv file directly into a data frame.
For more information: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.io.parsers.read_csv.html
Can you write Python code for it?

import pandas as pd # import the module and alias it as pd

cancer_data = pd.read_csv('breast-cancer-wisconsin.csv')
cancer_data.head() # show the first few rows of the data


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Printing values uses the print statement which can be passed arguments in different ways
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Python does not have a construct for a do-while loop, though it can be accomplished using the break statement
Functions
Functions in Python do not have a distinction between those that do and do not return a value.  If a value is returned, the type is not declared.
Functions can be declared in any module without any distinction between static and non-static.  Functions can even be declared within other functions
The syntax is the following
Can you write Python code for it?

# define function

# define player, name, number


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Rather than set the color equal to the class, we have created two plots based off of the class.  With a facet, we can get very detailed.  Lets through some more variables into the mix
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        facet_grid('clump_thickness', 'marginal_adhesion')
print plt


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Referring to the documentation link above about the data, the count, range of values (min = 1, max = 10), and data type (dtype = float64) look correct.
Lets take a look at another column, this time bare_nuclei
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Strings
As you may expect, Python has a powerful, full featured string module.
Substrings
Python strings can be substringed using bracket syntax
Can you write Python code for it?

mystring = "Go Gators, Come on Gators, Get up and go!"
# Print 1


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
The classifier is now ready to make some predictions.  We can use the score function to see how accurate the classifier is on the test data.  The score function will take the data in features_test, make a prediction of benign or malignant based on the decision tree that was fit to the training data, and compare that prediction to the true values in labels_test
Can you write Python code for it?

print "Accuracy score:", clf.score(features_test,labels_test)


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Then we have to fit the classifier to the training data. Both the training features (uniformity_cell_shape and bare_nuclei) and the labels (benign vs. malignant) are passed to the fit function
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Unfortunately, legends for faceting are not yet implemented in the Python ggplot package.  In this example we faceted on the x-axis with clump_thickness and along the y-axis with marginal_adhesion, then created 100 plots of uniformity_cell_shape vs. bare_nuclei effect on class.
I highly encourage you to check out https://ggplot.yhathq.com/docs/index.html to see all of the available geoms.  The best way to learn is to play with and visualize the data with many different plots and aesthetics.
Machine Learning
So now that we've acquired, audited, cleaned, and visualized our data, we have arrived at machine learning.  By formal definition from Tom Mitchell:

A computer program is set to learn from an experience E with
respect to some task T and some performance measure P if its performance
on T as measured by P improves with experience E.

Okay, thats a bit ridiculous.  Essentially machine learning is the science of building algorithms that learn from data in order make predictions about the data.  There are two main classes of machine learning: supervised and unsupervised.
In supervised learning, an algorithm will use the features of the data given to make a prediction about a known label.  For example, we will use supervised learning here to take features such as bare_nuclei and uniformity_cell_shape and predict a tumor class (benign or malignant).  This type of machine learning is called supervised because the class labels (benign or malignant) are a known quantity during learning, so we are supervising the algorithm with the "correct" answer.
In unsupervised learning, an algorithm will use the features of the data to discover what types of labels there could be.  The "correct" answer is not known.
In this session we will be mostly focused on supervised learning as we attempt to predict whether a tumor is benign or malignant.  We will also be focused on doing some practical machine learning, and will glaze over the algorithmic details.
The first thing we have to do is to extract the class labels and features...
Can you write Python code for it?

cancer_features = ['uniformity_cell_shape', 'bare_nuclei']


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Using negative values, you can count backwards from the end of the string.
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Visualizing the Data
Another important tool in the data scientist's toolbox is the ability to create visualizations from data.  Visualizing data is often the most logical place to start getting a deeper intuition of the data.  This intuition will shape and drive your analysis.
Even more important than visualizing data for your own personal benefit, it is often the job of the data scientist to use the data to tell a story.  Creating illustrative visuals that succinctly convey an idea are the best way to tell that story, especially to stakeholders with less technical skillsets.
Here we will be using a Python package called ggplot (https://ggplot.yhathq.com).  The ggplot package is an attempt to bring visuals following the guidelines outlayed in the grammar of graphics (http://vita.had.co.nz/papers/layered-grammar.html) to Python.  It is based off of and intended to mimic the features of the ggplot2 library found in R.  Additionally, ggplot is designed to work with Pandas dataframes, making things nice and simple.
We'll start by doing a bit of setup
Can you write Python code for it?

# The following line is NOT Python code, but a special syntax for enabling inline plotting in IPython
%matplotlib inline

from ggplot import *

import warnings

# ggplot usage of pandas throws a future warning
warnings.filterwarnings('ignore')


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
Whoops.  Importing the math module allows us access to all of its functions, but we must call them properly namespaced
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
With a simple aesthetic addition, we can see how these two variables play into our cancer classification
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter')
print plt


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
join
The join is useful for building strings from lists or other iterables.  Call join on the desired separator
Can you write Python code for it?


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
replace
Can you write Python code for it?

print mystring.replace("Gators", "Seminoles") # replaces all occurences of Gators with Seminoles

print mystring

print mystring.replace("Gators", "Seminoles", 1) # limit the number of replacements


I am working on the file "DataSciUF - Python I.ipynb".
The first task is:
return
In Python functions, an arbitrary number of values can be returned
Can you write Python code for it?

# def sum, return a + b

def sum_and_product(x,y):
    return x + y, x * y # return two values

mysum, myproduct = sum_and_product(1,2)
print mysum, myproduct


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
fillna is a dataframe function that replaces all nan values with either a scalar value, a series of values with the same indices as found in the dataframe, or a dataframe that is indexed by the columns of the target dataframe.
cancer_data.mean().round() will take the mean of each column (this computation ignores the currently present nan values), then round, and return a dataframe indexed by the columns of the original dataframe:
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
inplace=True allows us to make this modification directly on the dataframe, without having to do any assignment.
Now that we have figured out how to impute these missing values in a single column, lets start over and quickly apply this technique to the entire dataframe.
Can you write Python code for it?

cancer_data = pd.read_csv('breast-cancer-wisconsin.csv', header=None, names=cancer_header)
cancer_data = cancer_data.convert_objects(convert_numeric=True)
cancer_data.fillna(cancer_data.mean().round(), inplace=True)
cancer_data["bare_nuclei"].describe()

cancer_data["bare_nuclei"].unique()


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
A plot begins with the ggplot function.  Here, we pass in the cancer_data pandas dataframe and a special function called aes (short for aesthetic).  The values provided to aes change depending on which type of plot is being used.  Here we are going to make a histogram from the clump_thickness column in cancer_data, so that column name needs to be passed as the x parameter to aes.
The grammar of graphics is based off of a concept of "geoms" (short for geometric objects).  These geoms provide granular control of the plot and are progressively added to the base call to ggplot with + syntax.
Lets say we wanted to show the mean clump_thickness on this plot.  We could do something like the following
Can you write Python code for it?

plt = ggplot(aes(x = 'clump_thickness'), data = cancer_data) + \
        geom_histogram(binwidth = 1, fill = 'steelblue') + \
        geom_vline(xintercept = [cancer_data['clump_thickness'].mean()], linetype='dashed')
print plt


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
And make a call to download_file
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
As you can see, each geom has its own set of parameters specific to the appearance of that geom (also called aesthetics).
Lets try a scatter plot to get some multi-variable action
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
By adding color = 'class' as a parameter to the aes function, we now give a color to each unique value found in that column and automatically get a legend.  Remember, 2 is benign and 4 is malignant.
We can also do things such as add a title or change the axis labeling with geoms
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        ylab("Amount of Bare Nuclei") + \
        xlab("Uniformity in Cell shape")
print plt


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Data Science Tutorial
Now that we've covered some Python basics, we will begin a tutorial going through many tasks a data scientist may perform.  We will obtain real world data and go through the process of auditing, analyzing, visualing, and building classifiers from the data.
We will use a database of breast cancer data obtained from the University of Wisconsin Hospitals, Madison from Dr. William H. Wolberg.  The data is a collection of samples from Dr. Wolberg's clinical cases with attributes pertaining to tumors and a class labeling the sample as benign or malignant.
| Attribute                      | Domain                          |
|--------------------------------|---------------------------------|
| 1. Sample code number          | id number                       |
| 2. Clump Thickness             | 1 - 10                          |
| 3. Uniformity of Cell Size     | 1 - 10                          |
| 4. Uniformity of Cell Shape    | 1 - 10                          |
| 5. Marginal Adhesion           | 1 - 10                          |
| 6. Single Epithelial Cell Size | 1 - 10                          |
| 7. Bare Nuclei                 | 1 - 10                          |
| 8. Bland Chromatin             | 1 - 10                          |
| 9. Normal Nucleoli             | 1 - 10                          |
| 10. Mitoses                    | 1 - 10                          |
| 11. Class                      | (2 for benign, 4 for malignant) |
For more information on this data set:
https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29
Obtaining the Data
Lets begin by programmatically obtaining the data.  Here I'll define a function we can use to make HTTP requests and download the data
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
For this example, we will build a Decision Tree Classifier.  The goal of a decision tree is to create a prediction by outlining a simple tree of decision rules.  These rules are built from the training data by slicing the data on simple boundaries and trying to minimize the prediction error of that boundary.  More details on decision trees can be found here: http://scikit-learn.org/stable/modules/tree.html
The first step is to import the classifier from the sklearn.tree module.
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Here we call values on the dataframe to extract the values stored in the dataframe as an array of numpy arrays with the same dimensions as our subsetted dataframe.  Numpy is a powerful, high performance scientific computing package that implements arrays.  It is used internally by pandas.  We will use labels and features later on in our machine learning classifier
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Here we have attempted to convert the bare_nuclei series to a numeric type.  Lets see what the unique values are now.
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Lets take a look at some simple statistics for the clump_thickness column
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Lets try the import again, this time specifying the names.  When specifying names, the read_csv function requires us to set the header row number to None
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Nearly all classifiers, decision trees included, will have paremeters that can be tuned to build a more accurate model.  Without any parameter tuning and using just two features we have made a pretty accurate prediction.  Good job!
To get a better idea of what is going on, I have included a helper function to plot our test data along with the decision boundary
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Next, we create a variable to store the classifier
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Rather than set the color equal to the class, we have created two plots based off of the class.  With a facet, we can get very detailed.  Lets through some more variables into the mix
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Referring to the documentation link above about the data, the count, range of values (min = 1, max = 10), and data type (dtype = float64) look correct.
Lets take a look at another column, this time bare_nuclei
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
So we enabled plotting in IPython and imported everything from the ggplot package.  Now we'll create a plot and then break down the components
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Sometimes when working with integer data, or data that takes on a limited range of values, it is easier to visualize the plot with added jitter to the points.  We can do that by adding an aesthetic to geom_point.
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Structurally, Pandas dataframes are a collection of Series objects sharing a common index.  In general, the Series object and Dataframe object share a large number of functions with some behavioral differences.  In other words, whatever computation you can do on a single column can generally be applied to the entire dataframe.
Now we can use the dataframe version of describe to get an overview of all of our data
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Then we have to fit the classifier to the training data. Both the training features (uniformity_cell_shape and bare_nuclei) and the labels (benign vs. malignant) are passed to the fit function
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
There is definitely some patterning going on in that plot.
A slightly different way to convey this idea is to use faceting.  Faceting is the creation of multiple related plots arranged by the values of a given faceted variable
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        facet_grid('class')
print plt


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Unfortunately, legends for faceting are not yet implemented in the Python ggplot package.  In this example we faceted on the x-axis with clump_thickness and along the y-axis with marginal_adhesion, then created 100 plots of uniformity_cell_shape vs. bare_nuclei effect on class.
I highly encourage you to check out https://ggplot.yhathq.com/docs/index.html to see all of the available geoms.  The best way to learn is to play with and visualize the data with many different plots and aesthetics.
Machine Learning
So now that we've acquired, audited, cleaned, and visualized our data, we have arrived at machine learning.  By formal definition from Tom Mitchell:

A computer program is set to learn from an experience E with
respect to some task T and some performance measure P if its performance
on T as measured by P improves with experience E.

Okay, thats a bit ridiculous.  Essentially machine learning is the science of building algorithms that learn from data in order make predictions about the data.  There are two main classes of machine learning: supervised and unsupervised.
In supervised learning, an algorithm will use the features of the data given to make a prediction about a known label.  For example, we will use supervised learning here to take features such as bare_nuclei and uniformity_cell_shape and predict a tumor class (benign or malignant).  This type of machine learning is called supervised because the class labels (benign or malignant) are a known quantity during learning, so we are supervising the algorithm with the "correct" answer.
In unsupervised learning, an algorithm will use the features of the data to discover what types of labels there could be.  The "correct" answer is not known.
In this session we will be mostly focused on supervised learning as we attempt to predict whether a tumor is benign or malignant.  We will also be focused on doing some practical machine learning, and will glaze over the algorithmic details.
The first thing we have to do is to extract the class labels and features...
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Well at least the count is correct.  We were expecting no more than 10 unique values and now the data type is an object.
Whats up with our data?
We have arrived at arguably the most important part of performing data science: dealing with messy data.  One of most important tools in a data scientist's toolbox is the ability to audit, clean, and reshape data.  The real world is full of messy data and your sources may not always have data in the exact format you desire.
In this case we are working with csv data, which is a relatively straightforward format, but this will not always be the case when performing real world data science.  Data comes in all varieties from csv all the way to something as unstructured as a collection of emails or documents.  A data scientist must be versed in a wide variety of technologies and methodologies in order to be successful.
Now, lets do a little bit of digging into why were are not getting a numeric pandas column
Can you write Python code for it?


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
Whoops, looks like our csv file did not contain a header row.  read_csv assumes the first row of the csv is the header by default.
Lets check out the file located here: https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.names
This contains information about the data set including the names of the attributes.
Lets create a list of these attribute names to use when reading the csv file
Can you write Python code for it?

# \ allows multi line wrapping
cancer_header = [ \
                 'sample_code_number', \
                 'clump_thickness', \
                 'uniformity_cell_size', \
                 'uniformity_cell_shape', \
                 'marginal_adhesion', \
                 'single_epithelial_cell_size', \
                 'bare_nuclei', \
                 'bland_chromatin', \
                 'normal_nucleoli', \
                 'mitoses', \
                 'class']


I am working on the file "DataSciUF - Python II.ipynb".
The first task is:
With a simple aesthetic addition, we can see how these two variables play into our cancer classification
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
**kwargs
Similarly, we can define a dictionary of named parameters
Can you write Python code for it?

kwargs = {'name' : 'Michael Frazier II', 'number' : 20}

player(**kwargs) # calling player with the dereferenced kwargs dictionary.  The team argument will be defaulted


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
*args and **kwargs
In Python, there is a special deferencing scheme that allows for defining and calling functions with argument lists or dictionaries.
*args
Can you write Python code for it?

args = ['Michael Frazier II', 20, 'Florida']

player(*args) # calling player with the dereferenced argument list


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
And make a call to download_file
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Arbitrary indentation can be used within a code block, as long as the indentation is consistent.
Can you write Python code for it?

if (1 == 1):
        print "We're in "
            print "Deep Trouble:"

if (0 > -1):
            print "This works "
            print "just fine."


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Argument lists can also be used in defining a function as such
Can you write Python code for it?

def foo(*args):
    for someFoo in args:
        print someFoo

foo('la', 'dee', 'da') # supports an arbitrary number of arguments


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
By adding color = 'class' as a parameter to the aes function, we now give a color to each unique value found in that column and automatically get a legend.  Remember, 2 is benign and 4 is malignant.
We can also do things such as add a title or change the axis labeling with geoms
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Data Science Tutorial
Now that we've covered some Python basics, we will begin a tutorial going through many tasks a data scientist may perform.  We will obtain real world data and go through the process of auditing, analyzing, visualing, and building classifiers from the data.
We will use a database of breast cancer data obtained from the University of Wisconsin Hospitals, Madison from Dr. William H. Wolberg.  The data is a collection of samples from Dr. Wolberg's clinical cases with attributes pertaining to tumors and a class labeling the sample as benign or malignant.
| Attribute                      | Domain                          |
|--------------------------------|---------------------------------|
| 1. Sample code number          | id number                       |
| 2. Clump Thickness             | 1 - 10                          |
| 3. Uniformity of Cell Size     | 1 - 10                          |
| 4. Uniformity of Cell Shape    | 1 - 10                          |
| 5. Marginal Adhesion           | 1 - 10                          |
| 6. Single Epithelial Cell Size | 1 - 10                          |
| 7. Bare Nuclei                 | 1 - 10                          |
| 8. Bland Chromatin             | 1 - 10                          |
| 9. Normal Nucleoli             | 1 - 10                          |
| 10. Mitoses                    | 1 - 10                          |
| 11. Class                      | (2 for benign, 4 for malignant) |
For more information on this data set:
https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29
Obtaining the Data
Lets begin by programmatically obtaining the data.  Here I'll define a function we can use to make HTTP requests and download the data
Can you write Python code for it?

def download_file(url, local_filename):
    import requests

    # stream = True allows downloading of large files; prevents loading entire file into memory
    r = requests.get(url, stream = True)
    with open(local_filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)
                f.flush()


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
For more information on Lists:
https://docs.python.org/2/tutorial/datastructures.html#more-on-lists
Tuples
Python supports n-tuple sequences.  These are non-mutable
Can you write Python code for it?

mytuple = 'Tim', 'Tebow', 15 # Created with commas
print mytuple
print type(mytuple)

print mytuple[1] # access an item

mytuple[1] = "Winston" # results in error


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
For more information on string functions:
https://docs.python.org/2/library/stdtypes.html#string-methods
Data Structures
Lists
The Python standard library does not have traditional C-style fixed-memory fixed-type arrays.  Instead, lists are used and can contain a mix of any type.
Lists are created with square brackets []
Can you write Python code for it?

mylist = [1, 2, 3, 4, 'five']
print mylist

mylist.append(6.0) # add an item to the end of the list
print mylist

mylist.extend([8, 'nine']) # extend the list with the contents of another list
print mylist

mylist.insert(6, 7) # insert the number 7 at index 6
print mylist

mylist.remove('five') # removes the first matching occurence
print mylist

popped = mylist.pop() # by default, the last item in the list is removed and returned
print popped
print mylist

popped2 = mylist.pop(4) # pops at at index
print popped2
print mylist

print len(mylist) # returns the length of any iterable such as lists and strings

mylist.extend(range(-3, 0)) # the range function returns a list from -3 inclusive to 0 non inclusive
print mylist

# default list sorting. When more complex objects are in the list, arguments can be used to customize how to sort
mylist.sort()
print mylist

mylist.reverse() # reverse the list
print mylist


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Foreach
As it turns out, counting loops are just foreach loops in Python.  The range function returns a list of integers over which for in iterates.  This can be extended to any other iterable type
Can you write Python code for it?

for i in ['foo', 'bar']: # iterate over a list of strings
    print i

anotherdict = {'one' : 1, 'two' : 2, 'three' : 3}

for key in anotherdict.keys(): # iterate over a dictionary.  Order is not guaranteed
    print key, anotherdict[key]


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Here we have attempted to convert the bare_nuclei series to a numeric type.  Lets see what the unique values are now.
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Just as before, we can define a function taking an arbitrary dictionary
Can you write Python code for it?

def foo(**kwargs):
    for key in kwargs.keys():
        print key, kwargs[key]

foo(**kwargs)


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Lets take a look at some simple statistics for the clump_thickness column
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Looks like nothing was found.  -1 is returned by default.
Can you write Python code for it?

print mystring.find("Seminoles") # no Seminoles here


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Loops
Python supports for, foreach, and while loops
For (counting)
Traditional counting loops are accomplished in Python with a combination of the for key word and the range function
Can you write Python code for it?

for x in range(10): # with one argument, range produces integers from 0 to 9
    print x

for y in range(5, 12): # with two argumentts, range produces integers from 5 to 11
    print y

for z in range(1, 12, 3): # with three arguments, range starts at 1 and goes in steps of 3 until greater than 12
    print z

for a in range(10, 1, -5): # can use a negative step size as well
    print a

for b in range(2, 1, 1): # with a positive step, all values are less than 1. No integers are produced
    print b

for c in range(1, 2, -1): # same goes for a negative step as all values are less than 2
    print c


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Nearly all classifiers, decision trees included, will have paremeters that can be tuned to build a more accurate model.  Without any parameter tuning and using just two features we have made a pretty accurate prediction.  Good job!
To get a better idea of what is going on, I have included a helper function to plot our test data along with the decision boundary
Can you write Python code for it?

from class_vis import prettyPicture # helper class

prettyPicture(clf, features_test, labels_test)


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Now this might seem like overkill for downloading a single, small csv file, but we can use this same function to access countless APIs available on the World Wide Web by building an API request in the url.
Wrangling the Data
Now that we have some data, lets get it into a useful form.  For this task we will use a package called pandas. pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for Python.  The most fundamental data structure in pandas is the dataframe, which is similar to the data.frame data structure found in the R statistical programming language.
For more information: http://pandas.pydata.org
pandas dataframes are a 2-dimensional labeled data structures with columns of potentially different types.  Dataframes can be thought of as similar to a spreadsheet or SQL table.
There are numerous ways to build a dataframe with pandas.  Since we have already attained a csv file, we can use a parser built into pandas called read_csv which will read the contents of a csv file directly into a data frame.
For more information: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.io.parsers.read_csv.html
Can you write Python code for it?

import pandas as pd # import the module and alias it as pd

cancer_data = pd.read_csv('breast-cancer-wisconsin.csv')
cancer_data.head() # show the first few rows of the data


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Now we'll specify the url of the file and the file name we will save to
Can you write Python code for it?

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data'
filename = 'breast-cancer-wisconsin.csv'


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Python does not have a construct for a do-while loop, though it can be accomplished using the break statement
Can you write Python code for it?

a = 1; b = 10

while True: # short circuit the while condition
    a *= 2
    print a
    if a > b:
        break


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Python is a 0-index based language.  Generally whenever forming a range of values in Python, the first argument is inclusive whereas the second is not, i.e. mystring[11:25] returns characters 11 through 24.
You can omit the first or second argument
Can you write Python code for it?

print mystring[:9] # all characters before the 9th index

print mystring[27:] # all characters at or after the 27th

print mystring[:] # you can even omit both arguments


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Rather than set the color equal to the class, we have created two plots based off of the class.  With a facet, we can get very detailed.  Lets through some more variables into the mix
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        facet_grid('clump_thickness', 'marginal_adhesion')
print plt


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Sets
Python includes the set data structure which is an unordered collection with no duplicates
Can you write Python code for it?

schools = ['Florida', 'Florida State', 'Miami', 'Florida']
myset = set(schools) # the set is built from the schools list
print myset

print 'Georgia' in myset # membership test

print 'Florida' in myset

badschools = set(['Florida State', 'Miami'])
print myset - badschools # set arithmetic

print myset & badschools # AND

print myset | set(['Miami', 'Stetson']) # OR

print myset ^ set(['Miami', 'Stetson']) # XOR


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
So we enabled plotting in IPython and imported everything from the ggplot package.  Now we'll create a plot and then break down the components
Can you write Python code for it?

plt = ggplot(aes(x = 'clump_thickness'), data = cancer_data) + \
        geom_histogram(binwidth = 1, fill = 'steelblue')

# using print gets the plot to show up here within the notebook.
# In normal Python environment without using print, the plot appears in a window
print plt


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Sometimes when working with integer data, or data that takes on a limited range of values, it is easier to visualize the plot with added jitter to the points.  We can do that by adding an aesthetic to geom_point.
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei'), data = cancer_data) + \
        geom_point(position = 'jitter')
print plt


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
String Functions
Here are some more useful string functions
find
Can you write Python code for it?

print mystring.find("Gators") # returns the index of the first occurence of Gators

print mystring.find("Gators", 4) # specify an index on which to begin searching

print mystring.find("Gators", 4, 19) # specify both begin and end indexes to search


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Strings can be declared with either single or double quotes.
Can you write Python code for it?

c1 = "Go "
c2 = 'Gators'
c3 = c1 + c2
print c3
print type(c3)


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
The classifier is now ready to make some predictions.  We can use the score function to see how accurate the classifier is on the test data.  The score function will take the data in features_test, make a prediction of benign or malignant based on the decision tree that was fit to the training data, and compare that prediction to the true values in labels_test
Can you write Python code for it?

print "Accuracy score:", clf.score(features_test,labels_test)


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
There is definitely some patterning going on in that plot.
A slightly different way to convey this idea is to use faceting.  Faceting is the creation of multiple related plots arranged by the values of a given faceted variable
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        facet_grid('class')
print plt


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Using negative values, you can count positions backwards
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Using the from statement we can import everything from the math module.
Disclaimer: many Pythonistas discourage doing this for performance reasons.  Just import what you need
Can you write Python code for it?

from math import *
print sin(pi/2) # now we don't have to make a call to math


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Variables and Types
Variables can be given alphanumeric names beginning with an underscore or letter.  Variable types do not have to be declared and are inferred at run time.
Can you write Python code for it?

a = 1
print type(a) # Built in function

b = 2.5
print type(b)


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Well at least the count is correct.  We were expecting no more than 10 unique values and now the data type is an object.
Whats up with our data?
We have arrived at arguably the most important part of performing data science: dealing with messy data.  One of most important tools in a data scientist's toolbox is the ability to audit, clean, and reshape data.  The real world is full of messy data and your sources may not always have data in the exact format you desire.
In this case we are working with csv data, which is a relatively straightforward format, but this will not always be the case when performing real world data science.  Data comes in all varieties from csv all the way to something as unstructured as a collection of emails or documents.  A data scientist must be versed in a wide variety of technologies and methodologies in order to be successful.
Now, lets do a little bit of digging into why were are not getting a numeric pandas column
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
While
Python supports standard while loops
Can you write Python code for it?

a = 1; b = 4; c = 7; d = 5;

while (a < b) and (c > d): # example of and condition
    print c - a
    a += 1 # example of incrementing
    c -= 1 # decrementing


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
Whoops, looks like our csv file did not contain a header row.  read_csv assumes the first row of the csv is the header by default.
Lets check out the file located here: https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.names
This contains information about the data set including the names of the attributes.
Lets create a list of these attribute names to use when reading the csv file
Can you write Python code for it?

# \ allows multi line wrapping
cancer_header = [ \
                 'sample_code_number', \
                 'clump_thickness', \
                 'uniformity_cell_size', \
                 'uniformity_cell_shape', \
                 'marginal_adhesion', \
                 'single_epithelial_cell_size', \
                 'bare_nuclei', \
                 'bland_chromatin', \
                 'normal_nucleoli', \
                 'mitoses', \
                 'class']


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
With a simple aesthetic addition, we can see how these two variables play into our cancer classification
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter')
print plt


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
join
The join is useful for building strings from lists or other iterables.  Call join on the desired separator
Can you write Python code for it?


I am working on the file "DataSciUF Python I - Instructor.ipynb".
The first task is:
lower and upper
Can you write Python code for it?

print mystring.lower()
print mystring.upper()


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
fillna is a dataframe function that replaces all nan values with either a scalar value, a series of values with the same indices as found in the dataframe, or a dataframe that is indexed by the columns of the target dataframe.
cancer_data.mean().round() will take the mean of each column (this computation ignores the currently present nan values), then round, and return a dataframe indexed by the columns of the original dataframe:
Can you write Python code for it?


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
inplace=True allows us to make this modification directly on the dataframe, without having to do any assignment.
Now that we have figured out how to impute these missing values in a single column, lets start over and quickly apply this technique to the entire dataframe.
Can you write Python code for it?

cancer_data = pd.read_csv('breast-cancer-wisconsin.csv', header=None, names=cancer_header)
cancer_data = cancer_data.convert_objects(convert_numeric=True)
cancer_data.fillna(cancer_data.mean().round(), inplace=True)
cancer_data["bare_nuclei"].describe()

cancer_data["bare_nuclei"].unique()


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
A plot begins with the ggplot function.  Here, we pass in the cancer_data pandas dataframe and a special function called aes (short for aesthetic).  The values provided to aes change depending on which type of plot is being used.  Here we are going to make a histogram from the clump_thickness column in cancer_data, so that column name needs to be passed as the x parameter to aes.
The grammar of graphics is based off of a concept of "geoms" (short for geometric objects).  These geoms provide granular control of the plot and are progressively added to the base call to ggplot with + syntax.
Lets say we wanted to show the mean clump_thickness on this plot.  We could do something like the following
Can you write Python code for it?

plt = ggplot(aes(x = 'clump_thickness'), data = cancer_data) + \
        geom_histogram(binwidth = 1, fill = 'steelblue') + \
        geom_vline(xintercept = [cancer_data['clump_thickness'].mean()], linetype='dashed')
print plt


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
By adding color = 'class' as a parameter to the aes function, we now give a color to each unique value found in that column and automatically get a legend.  Remember, 2 is benign and 4 is malignant.
We can also do things such as add a title or change the axis labeling with geoms
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        ylab("Amount of Bare Nuclei") + \
        xlab("Uniformity in Cell shape")
print plt


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
For this example, we will build a Decision Tree Classifier.  The goal of a decision tree is to create a prediction by outlining a simple tree of decision rules.  These rules are built from the training data by slicing the data on simple boundaries and trying to minimize the prediction error of that boundary.  More details on decision trees can be found here: http://scikit-learn.org/stable/modules/tree.html
The first step is to import the classifier from the sklearn.tree module.
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Here we call values on the dataframe to extract the values stored in the dataframe as an array of numpy arrays with the same dimensions as our subsetted dataframe.  Numpy is a powerful, high performance scientific computing package that implements arrays.  It is used internally by pandas.  We will use labels and features later on in our machine learning classifier
Can you write Python code for it?

labels = cancer_data['class'].values
features = cancer_data[cancer_features].values


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Here we have attempted to convert the bare_nuclei series to a numeric type.  Lets see what the unique values are now.
Can you write Python code for it?


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Lets try the import again, this time specifying the names.  When specifying names, the read_csv function requires us to set the header row number to None
Can you write Python code for it?

cancer_data = pd.read_csv('breast-cancer-wisconsin.csv', header=None, names=cancer_header)
cancer_data.head()


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Nearly all classifiers, decision trees included, will have paremeters that can be tuned to build a more accurate model.  Without any parameter tuning and using just two features we have made a pretty accurate prediction.  Good job!
To get a better idea of what is going on, I have included a helper function to plot our test data along with the decision boundary
Can you write Python code for it?

from class_vis import prettyPicture # helper class

prettyPicture(clf, features_test, labels_test)


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Next, we create a variable to store the classifier
Can you write Python code for it?


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Now we'll specify the url of the file and the file name we will save to
Can you write Python code for it?

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data'
filename = 'breast-cancer-wisconsin.csv'


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Rather than set the color equal to the class, we have created two plots based off of the class.  With a facet, we can get very detailed.  Lets through some more variables into the mix
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        facet_grid('clump_thickness', 'marginal_adhesion')
print plt


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Referring to the documentation link above about the data, the count, range of values (min = 1, max = 10), and data type (dtype = float64) look correct.
Lets take a look at another column, this time bare_nuclei
Can you write Python code for it?


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
So we enabled plotting in IPython and imported everything from the ggplot package.  Now we'll create a plot and then break down the components
Can you write Python code for it?

plt = ggplot(aes(x = 'clump_thickness'), data = cancer_data) + \
        geom_histogram(binwidth = 1, fill = 'steelblue')

# using print gets the plot to show up here within the notebook.
# In normal Python environment without using print, the plot appears in a window
print plt


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Sometimes when working with integer data, or data that takes on a limited range of values, it is easier to visualize the plot with added jitter to the points.  We can do that by adding an aesthetic to geom_point.
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei'), data = cancer_data) + \
        geom_point(position = 'jitter')
print plt


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Then we have to fit the classifier to the training data. Both the training features (uniformity_cell_shape and bare_nuclei) and the labels (benign vs. malignant) are passed to the fit function
Can you write Python code for it?


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
There is definitely some patterning going on in that plot.
A slightly different way to convey this idea is to use faceting.  Faceting is the creation of multiple related plots arranged by the values of a given faceted variable
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei'), data = cancer_data) + \
        geom_point(position = 'jitter') + \
        ggtitle("The Effect of the Bare Nuclei and Cell Shape Uniformity on Classification") + \
        facet_grid('class')
print plt


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Unfortunately, legends for faceting are not yet implemented in the Python ggplot package.  In this example we faceted on the x-axis with clump_thickness and along the y-axis with marginal_adhesion, then created 100 plots of uniformity_cell_shape vs. bare_nuclei effect on class.
I highly encourage you to check out https://ggplot.yhathq.com/docs/index.html to see all of the available geoms.  The best way to learn is to play with and visualize the data with many different plots and aesthetics.
Machine Learning
So now that we've acquired, audited, cleaned, and visualized our data, we have arrived at machine learning.  By formal definition from Tom Mitchell:

A computer program is set to learn from an experience E with
respect to some task T and some performance measure P if its performance
on T as measured by P improves with experience E.

Okay, thats a bit ridiculous.  Essentially machine learning is the science of building algorithms that learn from data in order make predictions about the data.  There are two main classes of machine learning: supervised and unsupervised.
In supervised learning, an algorithm will use the features of the data given to make a prediction about a known label.  For example, we will use supervised learning here to take features such as bare_nuclei and uniformity_cell_shape and predict a tumor class (benign or malignant).  This type of machine learning is called supervised because the class labels (benign or malignant) are a known quantity during learning, so we are supervising the algorithm with the "correct" answer.
In unsupervised learning, an algorithm will use the features of the data to discover what types of labels there could be.  The "correct" answer is not known.
In this session we will be mostly focused on supervised learning as we attempt to predict whether a tumor is benign or malignant.  We will also be focused on doing some practical machine learning, and will glaze over the algorithmic details.
The first thing we have to do is to extract the class labels and features...
Can you write Python code for it?

cancer_features = ['uniformity_cell_shape', 'bare_nuclei']


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Using unique we can see that '?' is one of the distinct values that appears in this series.  Looking again at the documentation for this data set, we find the following:

Missing attribute values: 16
There are 16 instances in Groups 1 to 6 that contain a single missing
  (i.e., unavailable) attribute value, now denoted by "?".

It was so nice of them to tell us to expect these missing values, but as a data scientist that will almost never be the case.  Lets see what we can do with these missing values.
Can you write Python code for it?


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Visualizing the Data
Another important tool in the data scientist's toolbox is the ability to create visualizations from data.  Visualizing data is often the most logical place to start getting a deeper intuition of the data.  This intuition will shape and drive your analysis.
Even more important than visualizing data for your own personal benefit, it is often the job of the data scientist to use the data to tell a story.  Creating illustrative visuals that succinctly convey an idea are the best way to tell that story, especially to stakeholders with less technical skillsets.
Here we will be using a Python package called ggplot (https://ggplot.yhathq.com).  The ggplot package is an attempt to bring visuals following the guidelines outlayed in the grammar of graphics (http://vita.had.co.nz/papers/layered-grammar.html) to Python.  It is based off of and intended to mimic the features of the ggplot2 library found in R.  Additionally, ggplot is designed to work with Pandas dataframes, making things nice and simple.
We'll start by doing a bit of setup
Can you write Python code for it?

# The following line is NOT Python code, but a special syntax for enabling inline plotting in IPython
%matplotlib inline

from ggplot import *

import warnings

# ggplot usage of pandas throws a future warning
warnings.filterwarnings('ignore')


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
Whoops, looks like our csv file did not contain a header row.  read_csv assumes the first row of the csv is the header by default.
Lets check out the file located here: https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.names
This contains information about the data set including the names of the attributes.
Lets create a list of these attribute names to use when reading the csv file
Can you write Python code for it?

# \ allows multi line wrapping
cancer_header = [ \
                 'sample_code_number', \
                 'clump_thickness', \
                 'uniformity_cell_size', \
                 'uniformity_cell_shape', \
                 'marginal_adhesion', \
                 'single_epithelial_cell_size', \
                 'bare_nuclei', \
                 'bland_chromatin', \
                 'normal_nucleoli', \
                 'mitoses', \
                 'class']


I am working on the file "DataSciUF Python II - Instructor.ipynb".
The first task is:
With a simple aesthetic addition, we can see how these two variables play into our cancer classification
Can you write Python code for it?

plt = ggplot(aes(x = 'uniformity_cell_shape', y = 'bare_nuclei', color = 'class'), data = cancer_data) + \
        geom_point(position = 'jitter')
print plt


I am working on the file "DataScience/.ipynb_checkpoints/MapReduce-Introduction-checkpoint.ipynb".
The first task is:
Example Problem using MapReduce
As Python implements the functional programming paradigm too, it provides the functions required to implement the map-reduce paradigm builtin.
The foundation tools to implement map reduce are:

"mapper" which is in charge of mapping each input value to a corresponding output value
"reducer" which is in charge of merging multiple mapper outputs into a single output.

Both phases can be called multiple times (the output of a reducer can become the input of another reducer and a mapper can call other mappers).
Many MapReduce implementations also have additional phases like "combination" and "aggregation" which are executed after the mapper or the reducer to further cleanup their output.
Can you write Python code for it?

numbers = range(1000)

def mapper(value):
    return value*2

def reducer(*values):
    return sum(values)
#first_step = map(mapper, numbers)
#print first_step
result = reduce(reducer, map(mapper, numbers))
print result


I am working on the file "DataScience/.ipynb_checkpoints/MapReduce-Introduction-checkpoint.ipynb".
The first task is:
The previous mapreduce implementation takes a Mapper and a Reducer and splits them across num_workers until it gets back the final result
Can you write Python code for it?

numbers = range(1000)

def mapper(value):
    return value*2

def reducer(values):
    return sum(values)

mapreduce = ParallelMapReduce(mapper, reducer, 10)
print mapreduce(numbers)


I am working on the file "DataScience/Day2_MongoDB_PySpark.ipynb".
The first task is:
Updating the document replaces the whole original document
Can you write Python code for it?

col.find_one({'name': 'Giovanni'}

col.find({'name': 'Alessandro'}).count()

col.update({'name': 'Giovanni'}, {'$set': {'phone': '+23244322'}}) #

col.find_one({'name': 'Giovanni'})

doc = col.find_one({'name': 'Giovanni'})
doc['phone'] = 5
col.save(doc)
print doc


I am working on the file "DataScience/Day4_MongoDB.ipynb".
The first task is:
Exporting from MongoDB
There are cases you might want to export the results of a mongodb query to make possible to process them into another system, this might be the case for an EMR job which has to perform operations on data stored on MongoDB.
The most simple solution to export those data in a format recognized by EMR and MrJob is using the mongoexport tool provided with mongodb itself. The tool is able to export data in a format recognized by MrJob JSONValueProcotol so you can upload it to S3 and directly process it from EMR.
For example, exporting all the data for the web tag, can be easily done using:
$ ./mongoexport -d phonebook -c blog -o /tmp/data.json -q '{"tags": "web"}'

This will write the data to /tmp/data.json in a format recognized by JSONValueProtocol, full list of options can be seen using --help, in the previous example the following options were used:

-d -> to specify the database
-c -> to specify the collection in the database
-o -> to write output to /tmp/data.json
-q -> to filter output by the provided query

Sharding
Sharding, or horizontal scaling, divides the data set and distributes the data over multiple servers, or shards. Each shard is an independent database, and collectively, the shards make up a single logical database.
Chunk
The whole set of data is divided in Chunks, chunk are then distributed as equally as possible through all the nodes
Shard Key
The shard key is the Document property on which chunks are decided, the range of shard key possible values is divided in chunks and each chunk is assigned to a node. Document which near values for the shard key will end up being in the same chunk and so on the same node.
Shard
Each MongoDB node or ReplicaSet that contains part of the sharded data.
Router
The routers is the interface to the cluster, each query and operation will be performed against the router. The router is then in charge of forwarding the operation to one or multiple shards and gather the results.
Config Server
The config servers keep track of chunks distribution,...
Can you write Python code for it?

from pymongo import MongoClient
client = MongoClient('mongodb://localhost:27017/')
db = client.twitter

db.tweets.map_reduce(
    map='''function() {
        var tags = this.entities.hashtags;
        for(var i=0; i<tags.length; i++)
            emit(tags[i].text, 1);
    }''',
    reduce='''function(key, values) {
        return Array.sum(values);
    }''',
    out='tagsfrequency'
)

print(list(
    db.tagsfrequency.find({'value': {'$gt': 10}})
))


I am working on the file "DataScience/Day4_MongoDB.ipynb".
The first task is:
Fetching back inserted document can be done using find and find_one methods of collections. Both methods accept a query expression that filters the returned documents. Omitting it means retrieving all the documents (or in case of find_one the first document).
Can you write Python code for it?


I am working on the file "DataScience/Day4_MongoDB.ipynb".
The first task is:
Filters in mongodb are described by Documents themselves, so in case of PyMongo they are dictionaries too.
A filter can be specified in the form {'field': value}.
By default filtering is performed by equality comparison, this can be changed by specifying a query operator in place of the value.
Query operators by convention start with a $ sign and can be specified as {'field': {'operator': value}}.
Full list of query operators is available at http://docs.mongodb.org/manual/reference/operator/query/
For example if we want to find each person that has an object id greather than 53b30ff57ab71c051823b031 we can achieve that with:
Can you write Python code for it?

from bson import ObjectId
db.people.find_one({'_id': {'$gt':  ObjectId('55893a1d7ab71c669f4c149e')}})


I am working on the file "DataScience/Day4_MongoDB.ipynb".
The first task is:
MapReduce
MongoDB is powered by the V8 javascript engine, this means that each mongod node is able to run javascript code.
With an high enough number of mongod nodes, you actually end up with a powerful execution environment for distributed code that also copes with the major problem of data locality.
For this reason MongoDB exposes a mapreduce function which can be leveraged in shareded environments to run map reduce jobs.
Note that the Aggregation Pipeline is usually faster than the mapReduce feature, and it scales with the number of nodes as mapReduce, so you should rely on MapReduce only when the algorithm cannot be efficiently expressed with the Aggregation Pipeline.
Can you write Python code for it?

freqs = db.tweets.map_reduce(
    map='''function() {
        var tags = this.entities.hashtags;
        for(var i=0; i<tags.length; i++)
            emit(tags[i].text, 1);
    }''',
    reduce='''function(key, values) {
        return Array.sum(values);
    }''',
    out='tagsfrequency'
)

print(list(
    db.tagsfrequency.find({'value': {'$gt': 10}}).sort([('value', -1)])
))

print freqs

db.tweets.find_one()

freqs = db.tweets.map_reduce(
    map='''function() {
        var tags = this.user.screen_name;
        emit(tags, 1);
    }''',
    reduce='''function(key, values) {
        return Array.sum(values);
    }''',
    out='namefrequency'
)

print(list(
    db.namefrequency.find().sort([('value', -1)]).limit(10)
))


I am working on the file "DataScience/Day4_MongoDB.ipynb".
The first task is:
SubDocuments
The real power of mongodb is released when you use subdocuments.
As each mongodb document is a JSON object (actually BSON, but that doesn't change much for the user), it can contain any data which is valid in JSON. Including other documents and arrays. This replaces "relations" between collections in multiple use cases and it's heavily more efficient as it returns all the data in a single query instead of having to perform multiple queries to retrieve related data.
As MongoDB fully supports subdocuments it is also possible to query on sub document fields and even query on arrays using the dot notation.
For example if you want to store a blog post in mongodb you might actually store everything, including author data and tags inside the blogpost itself:
Can you write Python code for it?

db.blog.insert({'title': 'MongoDB is great!',
                'author': {'name': 'Alessandro',
                           'surname': 'Molina',
                           'avatar': 'http://www.gravatar.com/avatar/7a952cebb086d2114080b4b39ed83cad.png'},
                'tags': ['mongodb', 'web', 'scaling']})

db.blog.find_one({'title': 'MongoDB is great!'})

db.blog.find_one({'tags': 'mongodb'})

db.blog.find_one({'author.name': 'Alessandro'})

TAGS = ['mongodb', 'web', 'scaling', 'cooking']

import random
for postnum in range(1, 5):
    db.blog.insert({'title': 'Post %s' % postnum,
                    'author': {'name': 'Alessandro',
                               'surname': 'Molina',
                               'avatar': 'http://www.gravatar.com/avatar/7a952cebb086d2114080b4b39ed83cad.png'},
                    'tags': random.sample(TAGS, 2)})

for post in db.blog.find({'tags': {'$in': ['scaling', 'cooking']}}):
    print post['title'], '->', ', '.join(post['tags'])


I am working on the file "DataScience/Day4_MongoDB.ipynb".
The first task is:
You will notice that the map_reduce command has now been properly split across the nodes of our cluster. Our shards should report in their logs something like:
2015-06-28T12:31:58.231+0200 I COMMAND  [conn4] command twitter.$cmd command: mapReduce { mapreduce: "tweets", map: "function() {
        var tags = this.entities.hashtags;
        for(var i=0; i<tags.length; i++)
            emit(tags[i].text, 1);
    }", reduce: "function(key, values) {
        return Array.sum(values);
    }", out: "tmp.mrs.tweets_1435487518_0", shardedFirstPass: true } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:151 locks:{ Global: { acquireCount: { r: 2225, w: 1068, W: 3 } }, MMAPV1Journal: { acquireCount: { r: 575, w: 2130 } }, Database: { acquireCount: { r: 535, w: 1060, R: 42, W: 11 } }, Collection: { acquireCount: { R: 535, W: 1063 } }, Metadata: { acquireCount: { W: 8 } } } 102ms
Performances
Journal Performances

Write performance is reduced by 5-30%
For apps that are write-heavy (1000+ writes per  server) there can be slowdown due to mix of  journal and data flushes.

To avoid Journal Overhead save the journal on a separate DISK from data, it will lower the journal overhead down to 3%.
Fragmentation
Files can get fragmented over time if remove() and update() are issued.
* It gets worse if documents have varied sizes
* Fragmentation wastes disk space and RAM
* Also makes writes scattered and slower (have to lookup for an empty slot in extent)
* Fragmentation can be checked by comparing size to storageSize in the collection’s stats
* nmoved=1 in logs means document has been resized and moved to another extent
PowerOf2Allocation is default on 2.6, is more efficient in case of updates/remove as each record has a size in bytes that is a power of 2 (e.g. 32, 64, 128, 256, 512...) so when updating documents they probably have not need to be moved (if document was 200bytes it will have up to 56 more bytes before needing to be reallocated) and when deleted it will leave a slot that can be reused for another...
Can you write Python code for it?

sc

data = sc.parallelize([1,2,3,4,5,6])

data.first()

def multiply(v):
    return v*2

multdata = data.map(multiply)
print multdata

multdata.collect()

sc.defaultParallelism

values = range(20)
print values

rddvalues = sc.parallelize(values)

def sumall(i):
    yield list(i) # returns the list on each core (4 cores available)

rddvalues.mapPartitions(sumall).collect()

def summap(v):
    return v+1

rddvalues.map(summap).collect()

def sumtwo(a, b):
    return a+b

rddvalues.reduce(sumtwo)


I am working on the file "DataScience/Essai_PySpark.ipynb".
The first task is:
New exercice: Baby names
======================
dataset found here
to launch mongodb server: ./bin/mongod --dbpath=./data
Can you write Python code for it?

import pymongo
import csv
cli = pymongo.MongoClient('mongodb://localhost:27017')

with open('/Users/adrianopagano/Desktop/Big_Dive/BigDive5/Data/Baby_Names__Beginning_2007.csv') as csvfile:
    babies = csv.reader(csvfile)
    next(babies)
    for row in babies:
        cli.testdb.babies.insert({'year': row[0],
                                  'firstname': row[1],
                                  'county': row[2],
                                  'sex': row[3],
                                  'count': row[4]})

cli.testdb.babies.count()

babies_rdd = sc.mongoRDD('mongodb://localhost:27017/testdb.babies') # this creates a Spark RDD from the mongoDB collections

babies_rdd.take(2)

babies_rdd.filter(lambda d: d['firstname']=='MICHAEL').take(5)

babies

names_to_counties = babies_rdd.map(lambda d: (d['firstname'], d['county']))
names_to_counties.take(5)

grouped_names=names_to_counties.groupByKey()
grouped_names.take(5)

grouped_names.map(lambda pair: (pair[0], len(pair[1]))).sortBy(lambda pair: pair[1], False).take(10) # pair[1] is a list of all counties where the name appeared

names_freqs = babies_rdd.map(lambda d: (d['firstname'], int(d['count']))).reduceByKey(lambda x,y: x+y)
names_freqs.sortBy(lambda pair: pair[1], False).take(10)

babies_rdd.map(lambda d: (int(d['year']), int(d['count']))).reduceByKey(lambda x,y: x+y).sortBy(lambda pair: pair[1], False).sortBy(lambda pair: pair[0], True).collect()


I am working on the file "DataScience/MapReduce-Introduction.ipynb".
The first task is:
The previous mapreduce implementation takes a Mapper and a Reducer and splits them across num_workers until it gets back the final result
Can you write Python code for it?


I am working on the file "DataScience/MrJob.ipynb".
The first task is:
Final output will be something like:
Creating temp directory /var/folders/_x/g5brlyv963vclshf_kffdm440000gn/T/01_text_info.alexcomu.20160610.150420.364592
Running step 1 of 1...
Streaming final output from /var/folders/_x/g5brlyv963vclshf_kffdm440000gn/T/01_text_info.alexcomu.20160610.150420.364592/output...
"characters"    1258
"phrases"   21
"words" 144

MultiStep Jobs
There are cases when it is convenient to run multiple MapReduce steps on a single input to actually provide the expected output.
If you want to have multiple steps instead of the plain mapper, reducer steps you can specify them in MrJob using steps() method which will be called by MrJob to know the actual MapReduce Steps to run in place of the standard ones.
We are going to create a multistep job that gets the most frequent word in the text and only returns that one.
It's clear that the first loop through MapReduce will be our previous word frequency counter, then we need to filter the most frequent word out of all the counted ones.
Can you write Python code for it?

from mrjob.job import MRJob
from mrjob.step import MRStep

class MRMostFreqWord(MRJob):
    def steps(self):
        return [
            MRStep(mapper=self.mapper_wordcount,
                    reducer=self.reducer_wordcount),
            MRStep(mapper=self.mapper_freq,
                    reducer=self.reducer_freq),
            MRStep(mapper=self.mapper_most,
                    reducer=self.reducer_most)
        ]

    def mapper_wordcount(self, _, line):
        for word in line.split():
            if len(word)>2:
                yield word.lower(), 1 # return only word with at least 3 letters

    def reducer_wordcount(self, word, counts):
        yield word, sum(counts)  # Sum occurrences of each word to get frequency

    def mapper_freq(self, word, total):
        if total > 1:  # Only get words that appear more than once
            yield total, word  # Group them by frequency

    def reducer_freq(self, total, words):
        yield total, words.next()  # .next() gets the first element, so we emit only one word for each frequency

    def mapper_most(self, freq, word):
        yield 'most_used', [freq, word]  # Group all the words together in a list of (frequency, word) tuples

    def reducer_most(self, _, freqs):
        yield 'most_used', max(freqs)  # Get only the most frequent word

if __name__ == '__main__':
    try:
        MRMostFreqWord.run()
    except TypeError:
        print 'MrJob cannot work inside iPython Notebook as it is not saved as a standalone .py file'


I am working on the file "DataScience/MrJob.ipynb".
The first task is:
In this case we will have a better result:
"most_used" [4, "vitae"]
Let's Play!
Exercise 1
From the lorem.txt file calculate how many words starts for each letter of the alphabet.
Exercise 2
From the lorem.txt file calculate how many words starts for each letter of the alphabet and print out the max and the min.
Exercise 3
Write a MapReduce job that report the most frequent word grouped by word length.
ES:
5 Chars -> hello -> 8 occurrences
3 Chars -> cat -> 4 occurrences
2 Chars -> at -> 7 occurrences
Programmatically Running Jobs
While starting jobs from the command line is a good way to test them, it is often the case that you have to visualize the resulting data. So we need to be able to start the MapReduce job from our software, get the output and send it to the HTML layer for visualization.
This can be achieved using MrJob Runners, which permit to start the execution programmatically and read the output.
Keep in mind that, as the MrJob process must live in a separate .py file, the runner must be kept separate from it.
So the runner will rely in our application, while the actual MrJob class will be in a separate module that can be sent to Hadoop for execution.
Our runner for the WordFreqCount job will look like (check the folder 06_runner):
Can you write Python code for it?

from wordcount import MRWordFreqCount

mr_job = MRWordFreqCount()
mr_job.stdin = open('lorem.txt').readlines()

with mr_job.make_runner() as runner:
    runner.run()
    for line in runner.stream_output():
        key, value = mr_job.parse_output_line(line)
        print 'Word:', key, 'Count:', value


I am working on the file "DataScienceProgramming/01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
Keeping the respository updated

Once you cloned the repository you can download any updates with a single command git pull
Note: your working directory has to be the repository or one of its subdirectories. Navigate into the right directory before executing the command
Warning: Make sure to rename a file before editing. If you edit a file that exists in the repository your pull request will fail. If that happens rename or move the offending file and pull again.

The Concept of Pipes and Redirections

Source: https://en.wikipedia.org/wiki/Pipeline_(Unix)
Examples
- ls -l | more
- ls -R > all_my_files.txt
- cat

The standard input STDIN can also be connect to a file using the < symbol
The standard output STDOUT can be redirected to a file using the > or >> symbols. Hereby >> appends the new content to an existing file.
The standard error STDERR can be redirected to file using 2> and 2>>. The symbol 2>&1 merges STDOUT and STDERR into one stream.

Examples

ls -l | more
ls -R > all_my_files.txt
cat all_my_files.txt | grep foo > data.dat
sort < data.dat
echo -n "The number of README files in this directory is: "; find . -name "README*" | wc -l
echo -n "There are `find . -name "README*" | wc -l` README files in this directory tree."
Can you write Python code for it?

%%sh
echo -n "There are `find .. -name "README*" | wc -l` README files in the parent directory tree."


I am working on the file "DataScienceProgramming/01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
Let's look at the file in the terminal using the commands more or less
There is a lot of "junk":
- There is a lot of other text like legal notices included.
- Special character and even upper and lower case words will affect the analysis.
Tricks

Use tr -s ' ' to squash duplicate characters (banks in this case)
Filter entire repeating passages by
copy those lines into a file
run grep -v -f data/legalnotice.txt < data/pg100.txt

Filter empty/blank lines with grep -v -e '^[[:space:]]*$' this those are the little things you find on StackOverflow https://stackoverflow.com/questions/3432555/remove-blank-lines-with-grep ... usually after some digging.

Some questions?

How often do the terms "love", "hate", "murder", "faith" appear in the text?
What are the most frequent words
Can you write Python code for it?

%%sh
cat data/pg100.txt | grep hate | head -20
echo
echo -n "The answer is: "
cat data/pg100.txt | grep hate | wc -l

%%sh
cat data/pg100.txt | tr ' ' '\n' | grep hate | sort | uniq -c | sort -rn | head -20
echo
echo -n "The answer is: "
cat data/pg100.txt | tr ' ' '\n' | grep hate | wc -l


I am working on the file "DataScienceProgramming/01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
News Groups Dataset
http://qwone.com/~jason/20Newsgroups/
he 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.

DO NOT DOWNLOAD THESE FILES ONTO THE CLUSTER!

Instead, use the shared data directory /home/data/20_newsgroup/
Can you write Python code for it?

%%sh
DATADIR=/home/data/20_newsgroup/
ls $DATADIR | while read TOPIC; do
echo -n "Topic $TOPIC: number of documents "
ls $DATADIR/$TOPIC | wc -l
done | cat -b


I am working on the file "DataScienceProgramming/01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
UNIX Commands/Tools for Data Manipulation and Analysis

grep
sort
uniq
tr
wc
cut

Advanced

awk
sed

The Works  by Shakespeare
Let's see what we can do with Shakespeare's collected body of work.
We can download everything from http://www.gutenberg.org/ebooks/100 in plain text format
Can you write Python code for it?

%%sh
mkdir -p data
cd data
wget http://www.gutenberg.org/cache/epub/100/pg100.txt 2> /dev/null
cd ..
ls -l data


I am working on the file "DataScienceProgramming/02-Python/WordFrequencies.ipynb".
The first task is:
Dictonaries
Dictionaries serve as associative arrays that binds keys to values. These can be used to keep track of the individual words. However, retrieving values from their keys can be time consuming.
Can you write Python code for it?

help(dict)

f = { 'one': 1, 'two': 2}
f['a'] = 0

f

f['one']

f.keys()

f.values()

Ω = 17

Δ

'a' in f.keys()

f['b']


I am working on the file "DataScienceProgramming/02-Python/WordFrequencies.ipynb".
The first task is:
For example: load the collection of Shakespear's work and print a couple of rows. (The first 244 lines of this particular document are copyright information, and should be skipped.)
Can you write Python code for it?

with urlopen('http://ocw.mit.edu/ans7870/6/6.006/s08/lecturenotes/files/t8.shakespeare.txt') as src:
    txt = src.readlines()
    for t in txt[244:250]:
        print(t.decode())


I am working on the file "DataScienceProgramming/02-Python/WordFrequencies.ipynb".
The first task is:
In order to read the text files within an entire directory we have to first get thg list, and then iterate through it.
Can you write Python code for it?

dir_list = client.list('/user/pmolnar/data/20news/20news-bydate-test/talk.politics.mideast/')
dir_list[0:10]

text_docs = []
for f in dir_list:
    with client.read('/user/pmolnar/data/20news/20news-bydate-test/talk.politics.mideast/%s' % f) as reader:
        txt = zlib.decompress(reader.read(), 16+zlib.MAX_WBITS).decode()
        text_docs.append(txt)
print("Read %d text files." % len(text_docs))

text_docs[1:3]


I am working on the file "DataScienceProgramming/02-Python/WordFrequencies.ipynb".
The first task is:
Pull text from Hadoop File System (HDFS)
We're usually interested in fairly big data sets which we keep on the Hadoop File System. All Hadoop and Spark functions can uncompress text files on the fly. Therefore they are stored in a compressed format (.gz).
Can you write Python code for it?

import zlib
from hdfs import InsecureClient
client = InsecureClient('http://backend-0-0:50070')

with client.read('/user/pmolnar/data/20news/20news-bydate-test/talk.politics.mideast/77239.gz') as reader:
  txt = zlib.decompress(reader.read(), 16+zlib.MAX_WBITS).decode()
txt[0:100]

txt.split('\n')


I am working on the file "DataScienceProgramming/03-NumPy-and-Linear-Algebra/APD-Crime-Data_orig.ipynb".
The first task is:
Crime per year
Let's look at the
Can you write Python code for it?

crime_year = dff[dff.occur_year.between(2009, 2015)].groupby(by=['UC2 Literal', 'occur_year']).offense_id.count()

%matplotlib inline
fig = plt.figure(figsize=(40,30))
crime_types = crime_year.index.levels[0]
years = crime_year.index.levels[1]
for c in range(len(crime_types)):
    y_max = max(crime_year.loc[crime_types[c]])

    plt.subplot(4,3,c+1)
    plt.hlines(crime_year.loc[crime_types[c]].iloc[-1]*100/y_max, years[0], years[-1], linestyles="dashed", color="r")
    plt.bar(crime_year.loc[crime_types[c]].index, crime_year.loc[crime_types[c]]*100/y_max, label=crime_types[c], alpha=0.5)
    ##plt.legend()
    plt.ylim(0, 100)
    plt.xticks(years+0.4, [str(int(y)) for y in years], rotation=0, fontsize=24)
    plt.yticks([0,20,40,60,80,100], ['0%','20%','40%','60%','80%','100%'], fontsize=24)
    plt.title(crime_types[c], fontsize=30)
    None