I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Ever since I discovered Watermark, I love it because it automatically documents the package versions, and information about the machine I ran the code on.  This is great for reproducibility when sharing your work/results with others.
Why Pre-Process the data?
Scikit Learn - Machine Learning Package for python
- Very easy to run machine learning algorithms
- More difficult to use the algorithms correctly
- Garbage in, garbage out!

So scikit learn is a machine learning package for python.  I think it is the gold standard for the way packages should be in the scientific community.  It is so easy to pick up quickly and use to build awesome models and create machine learning algorithms with.    So it's easy to use, but the real work is in 2 areas, first we need to make sure our data is cleaned and properly formatted for the various machine learning methods, and we need to make sure that the data looks the way that it should for the given model.  We want to make sure that we are satisfying the assumptions of the model before using it.  Garbage in, garbage out.   So this is why we need to preprocess our data.
Various issues in the data can include:
- Noisy data (out-of-range, impossible combinations, human error, etc)
- Missing Data (General noise/error, MCAR, MNAR, MAR)
- Data of different types (numeric, categorical)
- Too many attributes
- Too much data
- Data does not fit the assumed characteristic for a given method

For each of these issues, we have different solutions:
- Imputing (filling in missing values)
- Binary Features
- Categorical
- Binarizing
- Correlation Analysis/Chi-Squared Test of Independance
- Aggregating
- Random Sampling
- Scaling/Normalizing

So what kinds of issues can we find in our data?  Data has been known to "lie".  It comes from various sources such as people, sensors, the internet, and all these have been known to lie sometimes.  We can have noisy data, missing data, different types of data, too much data, and data values not as expected.  I'll be going over a few examples of...
Can you write Python code for it?

from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
import numpy
import seaborn as sns
import pandas as pd
%matplotlib inline
from sklearn.pipeline import Pipeline
from sklearn import preprocessing
sns.set()

iris = load_iris()

iris.feature_names

iris.data[0:5]


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
I like seaborn's visualization capability and integration with pandas so I'm going to download the dataset from there instead.
Can you write Python code for it?

df = sns.load_dataset("iris")

df.head()


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
It's much easier to visualize what is happening using pandas, so I'll include that here as well.
Can you write Python code for it?


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Let's make a deep copy of the exercise data frame so we can start modifying it.
Can you write Python code for it?

exercise_numeric_df = exercise.copy()

exercise.columns

exercise.head()


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Looking at this data, another method could be to use clustering or regression to model the data without missing values and then see where the data with some missing feature values would be.  The thing to note about imputing the data in a more specialized way is that the preprocessing.impute() function would need to be called on the data itself and the Imputer could not be part of the pipeline.
Dealing with Different Types (Numeric, Categorical):
Binarizing:
Binarizing is the process of converting a variable to a 0 or 1 given a certain threshold.
To show an example for binarizing, I wanted to have a data set with both categorical and numeric data.  I downloaded an 'exercise' dataset from seaborn.
Can you write Python code for it?

sns.set(style="ticks")
exercise = sns.load_dataset("exercise")

exercise.head()


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Now something to think about might be whether want to set the mean of all the data as the imputed value.  Looking at the data by the given species, for petal length, the means vary vastly between the 3 species.  So we many want to impute using the mean within the species and not over the whole data set.
Can you write Python code for it?


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Now we need to convert the diet, kind, and time columns into "dummy variables":
Can you write Python code for it?

one_hot_encoder = preprocessing.OneHotEncoder(categorical_features=[2,4,5])

one_hot_encoder

exercise_numeric_encoded_matrix = one_hot_encoder.fit_transform(exercise_numeric_df.values)

exercise_numeric_encoded_matrix.toarray()[0:10,:]

exercise_numeric_encoded_matrix.shape


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Obviously this is a simple example, we could just easily do this with a one line pandas call, but again the advantage of doing this through scikit-learn is that it can be part of the pipeline.
pandas one liner: exercise['high_pulse'] = exercise.pulse>120
Create numerical features from the Categorical:
Since we can't just plug in the exercise data as in into a machine learning algorithm, we need to transform the data so that it only contains numerical data.  So there are 2 primary ways of doing this.  One is to create a numeric value for each of the categories in a given column, or another way is to create new features very similar to what is called "creating dummy variables" in statistics.

LabelEncoder()
OneHotEncoder()
Can you write Python code for it?

encoder = preprocessing.LabelEncoder()

exercise.diet.unique()

encoder.fit_transform(exercise.diet)


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
OneHotEncoder expects the data to be numeric, so LabelEncoder would need to be applied first to convert everything to a numeric value.
Way to do it in pandas, because 1 and 0 don't have any meaning really, it doesn't matter which one got the 1 label and which one got the 0 label.
Can you write Python code for it?


I am working on the file "Pre-Model Workflow.ipynb".
The first task is:
Strategy can take values 'mean', 'median', 'most_frequent'.  Can also give the Imputer function what the missing value looks like in the data.  Sometimes people use -1 or 99999.
Can you write Python code for it?

imputed_df = df.copy()

imputed_data = imputer.fit_transform(df.ix[:,0:4])

imputed_df.ix[:,0:4] = imputed_data

imputed_df.head(15)

print df.mean()
df.groupby('species').mean()


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
2. Simple Math in the Jupyter Notebook
Now that we have the Jupyter notebook up and running, we're going to do a short breakout exploring some of the mathematical functionality that Python offers.
Please open Breakout-Simple-Math.ipynb, find a partner, and make your way through that notebook, typing and executing code along the way.
3. Loading data with pandas
With this simple Python computation experience under our belt, we can now move to doing some more interesting analysis.
Python's Data Science Ecosystem
In addition to Python's built-in modules like the math module we explored above, there are also many often-used third-party modules that are core tools for doing data science with Python.
Some of the most important ones are:
numpy: Numerical Python
Numpy is short for "Numerical Python", and contains tools for efficient manipulation of arrays of data.
If you have used other computational tools like IDL or MatLab, Numpy should feel very familiar.
scipy: Scientific Python
Scipy is short for "Scientific Python", and contains a wide range of functionality for accomplishing common scientific tasks, such as optimization/minimization, numerical integration, interpolation, and much more.
We will not look closely at Scipy today, but we will use its functionality later in the course.
pandas: Labeled Data Manipulation in Python
Pandas is short for "Panel Data", and contains tools for doing more advanced manipulation of labeled data in Python, in particular with a columnar data structure called a Data Frame.
If you've used the R statistical language (and in particular the so-called "Hadley Stack"), much of the functionality in Pandas should feel very familiar.
matplotlib: Visualization in Python
Matplotlib started out as a Matlab plotting clone in Python, and has grown from there in the 15 years since its creation. It is the most popular data visualization tool currently in the Python data world (though other recent packages are starting to encroach on its monopoly).
Installing Pandas & friends
Because the above...
Can you write Python code for it?


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
4. Manipulating data with pandas
Here we'll cover some key features of manipulating data with pandas
Access columns by name using square-bracket indexing:
Can you write Python code for it?

df_small = df[ 'stoptime']

type(df_small)


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
Columns can be created (or overwritten) with the assignment operator.
Let's create a tripminutes column with the number of minutes for each trip
More complicated mathematical operations can be done with tools in the numpy package:
Working with Times
One trick to know when working with columns of times is that Pandas DateTimeIndex provides a nice interface for working with columns of times.
For a dataset of this size, using pd.to_datetime and specifying the date format can make things much faster (from the strftime reference, we see that the pronto data has format "%m/%d/%Y %I:%M:%S %p"
(Note: you can also use infer_datetime_format=True in most cases to automatically infer the correct format, though due to a bug it doesn't work when AM/PM are present)
With it, we can extract, the hour of the day, the day of the week, the month, and a wide range of other views of the time:
Simple Grouping of Data
The real power of Pandas comes in its tools for grouping and aggregating data. Here we'll look at value counts and the basics of group-by operations.
Value Counts
Pandas includes an array of useful functionality for manipulating and analyzing tabular data.
We'll take a look at two of these here.
The pandas.value_counts returns statistics on the unique values within each column.
We can use it, for example, to break down rides by gender:
Or to break down rides by age:
By default, the values rather than the index are sorted. Use sort=False to turn this behavior off:
We can explore other things as well: day of week, hour of day, etc.
Group-by Operation
One of the killer features of the Pandas dataframe is the ability to do group-by operations.
You can visualize the group-by like this (image borrowed from the Python Data Science Handbook)
Can you write Python code for it?

df.head()

df_count = df.groupby(['from_station_id']).count()
df_count.head()

df_count1 = df_count[['trip_id']]
df_count2 = df_count1.rename(columns={'trip_id': 'count'})
df_count2['new'] = 1
df_count2.head()

df_mean = df.groupby(['from_station_id']).mean()
df_mean.head()

dfgroup = df.groupby(['from_station_id'])
dfgroup.groups


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
Loading Data with Pandas
Because we'll use it so much, we often import under a shortened name using the import ... as ... pattern:
Can you write Python code for it?

import pandas as pd
df = pd.read_csv('pronto.csv')


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
Now we can simply call the plot() method of any series or dataframe to get a reasonable view of the data:
Can you write Python code for it?

import matplotlib.pyplot as plt
df['tripduration'].hist()


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
The columns attribute gives us the column names
The index attribute gives us the index names
The dtypes attribute gives the data types of each column:
Can you write Python code for it?


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
The shape attribute shows us the number of elements:
Can you write Python code for it?


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
The simplest version of a groupby looks like this, and you can use almost any aggregation function you wish (mean, median, sum, minimum, maximum, standard deviation, count, etc.)
<data object>.groupby(<grouping values>).<aggregate>()
for example, we can group by gender and find the average of all numerical columns:
It's also possible to indes the grouped object like it is a dataframe:
You can even group by multiple values: for example we can look at the trip duration by time of day and by gender:
The unstack() operation can help make sense of this type of multiply-grouped data. What this technically does is split a multiple-valued index into an index plus columns:
5. Visualizing data with pandas
Of course, looking at tables of data is not very intuitive.
Fortunately Pandas has many useful plotting functions built-in, all of which make use of the matplotlib library to generate plots.
Whenever you do plotting in the IPython notebook, you will want to first run this magic command which configures the notebook to work well with plots:
Can you write Python code for it?


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-And-Data-Spring-2018.ipynb".
The first task is:
uncomment this to download the data:
Can you write Python code for it?

#!curl -o pronto.csv https://data.seattle.gov/api/views/tw7j-dfaw/rows.csv?accessType=DOWNLOAD


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-and-Data.ipynb".
The first task is:
Now we can simply call the plot() method of any series or dataframe to get a reasonable view of the data:
Adjusting the Plot Style
The default formatting is not very nice; I often make use of the Seaborn library for better plotting defaults.
First you'll have to
$ conda install seaborn
and then you can do this:
Can you write Python code for it?


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-and-Data.ipynb".
The first task is:
So, for example, we can use this to find the average length of a ride as a function of time of day:
The simplest version of a groupby looks like this, and you can use almost any aggregation function you wish (mean, median, sum, minimum, maximum, standard deviation, count, etc.)
<data object>.groupby(<grouping values>).<aggregate>()
You can even group by multiple values: for example we can look at the trip duration by time of day and by gender:
The unstack() operation can help make sense of this type of multiply-grouped data:
5. Visualizing data with pandas
Of course, looking at tables of data is not very intuitive.
Fortunately Pandas has many useful plotting functions built-in, all of which make use of the matplotlib library to generate plots.
Whenever you do plotting in the IPython notebook, you will want to first run this magic command which configures the notebook to work well with plots:
Can you write Python code for it?


I am working on the file "PreFall2018/02-Python-and-Data/Lecture-Python-and-Data.ipynb".
The first task is:
Software Engineering for Data Scientists
Manipulating Data with Python
CSE 599 B1
Today's Objectives
1. Opening & Navigating the IPython Notebook
2. Simple Math in the IPython Notebook
3. Loading data with pandas
4. Cleaning and Manipulating data with pandas
5. Visualizing data with pandas
1. Opening and Navigating the IPython Notebook
We will start today with the interactive environment that we will be using often through the course: the IPython/Jupyter Notebook.
We will walk through the following steps together:

Download miniconda (be sure to get Version 3.5) and install it on your system (hopefully you have done this before coming to class)

Use the conda command-line tool to update your package listing and install the IPython notebook:

Update conda's listing of packages for your system:
   $ conda update conda
Install IPython notebook and all its requirements
   $ conda install ipython-notebook

Navigate to the directory containing the course material. For example:

$ cd ~/courses/CSE599/
You should see a number of files in the directory, including these:
$ ls
   ...
   Breakout-Simple-Math.ipynb
   CSE599_Lecture_2.ipynb
   ...

Type ipython notebook in the terminal to start the notebook

$ ipython notebook
If everything has worked correctly, it should automatically launch your default browser

Click on CSE599_Lecture_2.ipynb to open the notebook containing the content for this lecture.

With that, you're set up to use the IPython notebook!
2. Simple Math in the IPython Notebook
Now that we have the IPython notebook up and running, we're going to do a short breakout exploring some of the mathematical functionality that Python offers.
Please open Breakout-Simple-Math.ipynb, find a partner, and make your way through that notebook, typing and executing code along the way.
3. Loading data with pandas
With this simple Python computation experience under our belt, we can now move to doing some more interesting analysis.
Python's Data Science Ecosystem
In addition to Python's built-in modules like...
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:

Hacky Hack Time with Ifs, Fors, Lists, and imports!
Objective: Replace the bash magic bits for downloading the Pronto data and uncompressing it with Python code.  Since the download is big, check if the zip file exists first before downloading it again.  Then load it into a pandas dataframe.
Notes:
* The os package has tools for checking if a file exists: os.path.exists
import os
filename = 'open_data_year_two.zip'
if os.path.exists(filename):
    print("wahoo!")
* Use the requests package to get the file given a url (got this from the requests docs)
import requests
url = 'https://s3.amazonaws.com/pronto-data/open_data_year_two.zip'
req = requests.get(url)
assert req.status_code == 200 # if the download failed, this line will generate an error
with open(filename, 'wb') as f:
    f.write(req.content)
* Use the zipfile package to decompress the file while reading it into pandas
import pandas as pd
import zipfile
csv_filename = '2016_trip_data.csv'
zf = zipfile.ZipFile(filename)
data = pd.read_csv(zf.open(csv_filename))
Now, use your code from above for the following URLs and filenames
| URL | filename | csv_filename |
|-----|----------|--------------|
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set1.zip?raw=true | open_data_year_two_set1.zip | 2016_trip_data_set1.csv |
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set2.zip?raw=true | open_data_year_two_set2.zip | 2016_trip_data_set2.csv |
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set3.zip?raw=true | open_data_year_two_set3.zip | 2016_trip_data_set3.csv |
What pieces of the data structures and flow control that we talked about earlier can you use?
How did you solve this problem?

Functions
For loops let you repeat some code for every item in a list.  Functions are similar in that they run the same lines of code for new values of some variable.  They are different in that functions are not limited to looping over items.
Functions are a critical part of writing easy to...
Can you write Python code for it?

def print_string(str):
    """This prints out a string passed as the parameter."""
    print(str)
    return


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Any questions about lists before we move on?
<hr>
Dictionaries
Dictionaries are similar to tuples and lists in that they hold a collection of objects.  Dictionaries, however, allow an additional indexing mode: keys.  Think of a real dictionary where the elements in it are the definitions of the words and the keys to retrieve the entries are the words themselves.
| word | definition |
|------|------------|
| tuple | An immutable collection of ordered objects |
| list | A mutable collection of ordered objects |
| dictionary | A mutable collection of named objects |
Let's create this data structure now.  Dictionaries, like tuples and elements use a unique referencing method, '{' and its evil twin '}'.
Can you write Python code for it?

my_dict = { 'tuple' : 'An immutable collection of ordered objects',
            'list' : 'A mutable collection of ordered objects',
            'dictionary' : 'A mutable collection of objects' }
my_dict


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
At first glance, tuples and lists look pretty similar.  Notice the lists use '[' and ']' instead of '(' and ')'.  But indexing and refering to the first entry as 0 and the last as -1 still works the same.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
But in all, it isn't very Pythonesque to use indexes like that (unless you have another reason in the loop) and you would opt instead for the instructor in instructors form.
More often, you are doing something with the numbers that requires them to be integers, e.g. math.
Can you write Python code for it?

sum = 0
for i in range(10):
    sum += i
print(sum)


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Conditionals can take and and or and not.  E.g.
Can you write Python code for it?

my_favorite = 'pie'

if my_favorite is 'cake' or my_favorite is 'pie':
    print(my_favorite + " : I have a recipe for that!")
else:
    print("Ew!  Who eats that?")


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Facts about tuples:
* You can't add elements to a tuple. Tuples have no append or extend method.
* You can't remove elements from a tuple. Tuples have no remove or pop method.
* You can also use the in operator to check if an element exists in the tuple.
So then, what are the use cases of tuples?
* Speed
* Write-protects data that other pieces of code should not alter
You can alter the value of a tuple variable, e.g. change the tuple it holds, but you can't modify it.
Can you write Python code for it?

my_tuple

my_tuple = ('I', 'love', 'pie')
my_tuple


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Finally, tuples can contain different types of data, not just strings.
Can you write Python code for it?

import math
my_second_tuple = (42, 'Elephants', 'ate', math.pi)
my_second_tuple


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
For
For loops are the standard loop, though while is also common.  For has the general form:
for items in list:
    do stuff
For loops and collections like tuples, lists and dictionaries are natural friends.
Can you write Python code for it?

for instructor in instructors:
    print(instructor)


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
For loops can be nested
Note: for more on formatting strings, see: https://pyformat.info
Can you write Python code for it?

for i in range(1, 4):
    for j in range(1, 4):
        print('%d * %d = %d' % (i, j, i*j))  # Note string formatting here, %d means an integer


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
If
If statements can be use to execute some lines or block of code if a particular condition is satisfied.  E.g. Let's print something based on the entries in the list.
Can you write Python code for it?

if 'Dorkus the Clown' in instructors:
    print('#fakeinstructor')


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
It is always good practice to handle all cases explicity.  Conditional fall through is a common source of bugs.
Sometimes we wish to test multiple conditions.  Use if, elif, and else.
Can you write Python code for it?

my_favorite = 'pie'

if my_favorite is 'cake':
    print("He likes cake!  I'll start making a double chocolate velvet cake right now!")
elif my_favorite is 'pie':
    print("He likes pie!  I'll start making a cherry pie right now!")
else:
    print("He likes " + my_favorite + ".  I don't know how to make that.")


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Note: what have you noticed in this session about quotes?  What is the difference between ' and "?
Another simple example:
Can you write Python code for it?

if True is False:
    print("I'm so confused")
else:
    print("Everything is right with the world")


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Notice that ordering is not preserved!
And we can add new items to the list.
Can you write Python code for it?

my_dict['cabbage'] = 'Green leafy plant in the Brassica family'
my_dict


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Notice, it doesn't work for elements.
Can you write Python code for it?

'A mutable collection of ordered objects' in my_dict


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
One of the useful things to know about a list how many elements are in it.  This can be found with the len function.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Other operators: -, /, *
Questions about tuples before we move on?
<hr>

Lists
Let's begin by creating a list called my_list that contains three elements.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Recall that Python indexes start at 0.  So the first element in a tuple is 0 and the last is array length - 1.  You can also address from the end to the front by using negative (-) indexes, e.g.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
So what happens if I decide that I really prefer pie over cake?
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Some other handy functions with lists:
* max
* min
* cmp
Sometimes you have a tuple and you need to make it a list.  You can cast the tuple to a list with list(my_tuple)
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
There are other useful methods on lists, including:
| methods  |  description  |
|---|---|
| list.append(obj)  | Appends object obj to list  |
| list.count(obj)| Returns count of how many times obj occurs in list  |
| list.extend(seq)  | Appends the contents of seq to list  |
| list.index(obj)  | Returns the lowest index in list that obj appears  |
| list.insert(index, obj)  | Inserts object obj into list at offset index  |
| list.pop(obj=list[-1])  | Removes and returns last object or obj from list  |
| list.remove(obj)  | Removes object obj from list  |
| list.reverse()  |  Reverses objects of list in place |
| list.sort([func])  | Sort objects of list, use compare func, if given  |
Try some of them now.
```
my_list.count('I')
my_list
my_list.append('I')
my_list
my_list.count('I')
my_list
my_list.index(42)
my_list.index('puppies')
my_list
my_list.insert(my_list.index('puppies'), 'furry')
my_list
```
Can you write Python code for it?

my_list.count('I')
my_list

my_list.append('I')
my_list

my_list.count('I')
my_list

#my_list.index(42)

my_list.index('puppies')
my_list

my_list.insert(my_list.index('puppies'), 'furry')
my_list

my_list.pop()
my_list

my_list.remove('puppies')
my_list

my_list.append('cabbages')
my_list


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
There is a special do nothing word: pass that skips over some arm of a conditional, e.g.
Can you write Python code for it?

if 'Joe' in instructors:
    print("Congratulations!  Joe is teaching, your class won't stink!")
else:
    pass


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
There is a unique language feature call for...else
Can you write Python code for it?

sum = 0
for i in range(10):
    sum += i
else:
    print('final i = %d, and sum = %d' % (i, sum))


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
This is a synonym for in my_dict
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
To call the function, use:
print_string("Dave is awesome!")
Note: The function has to be defined before you can call it!
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
To delete it propery, we need to pop that specific entry.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Usually we want conditional logic on both sides of a binary condition, e.g. some action when True and some when False
Can you write Python code for it?

if 'Dorkus the Clown' in instructors:
    print('There are fake names for class instructors in your list!')
else:
    print("Nothing to see here")


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
Variables have scope: global and local
In a function, new variables that you create are not saved when the function returns - these are local variables.  Variables defined outside of the function can be accessed but not changed - these are global variables, Note there is a way to do this with the global keyword.  Generally, the use of global variables is not encouraged, instead use parameters.
```
my_global_1 = 'bad idea'
my_global_2 = 'another bad one'
my_global_3 = 'better idea'
def my_function():
    print(my_global)
    my_global_2 = 'broke your global, man!'
    global my_global_3
    my_global_3 = 'still a better idea'
    return
my_function()
print(my_global_2)
print(my_global_3)
```
In general, you want to use parameters to provide data to a function and return a result with the return. E.g.
def sum(x, y):
    my_sum = x + y
    return my_sum
If you are going to return multiple objects, what data structure that we talked about can be used?  Give and example below.
Parameters have four different types:
| type | behavior |
|------|----------|
| required | positional, must be present or error, e.g. my_func(first_name, last_name) |
| keyword | position independent, e.g. my_func(first_name, last_name) can be called my_func(first_name='Dave', last_name='Beck') or my_func(last_name='Beck', first_name='Dave') |
| default | keyword params that default to a value if not provided |
Can you write Python code for it?

def print_name(first, last='the Clown'):
    print('Your name is %s %s' % (first, last))
    return


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
We can use the keys list for fun stuff, e.g. with the in operator.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
You can combine loops and conditionals:
Can you write Python code for it?

for instructor in instructors:
    if instructor.endswith('Clown'):
        print(instructor + " doesn't sound like a real instructor name!")
    else:
        print(instructor + " is so smart... all those gooey brains!")


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
You can exit loops early if a condition is met:
Can you write Python code for it?

for i in range(10):
    if i == 4:
        break
i


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
You can get a list of keys in the dictionary by using the keys method.
Can you write Python code for it?


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
You can iterate over letters in a string
Can you write Python code for it?

my_string = "DIRECT"
for c in my_string:
    print(c)


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
You can still use the in operator.
Can you write Python code for it?

'puppies' in my_list

'kittens' in my_list


I am working on the file "PreFall2018/04-Procedural_Python.ipynb".
The first task is:
You can use other objects as names, but that is a topic for another time.  You can mix and match key types, e.g.
Can you write Python code for it?

my_new_dict = {}
my_new_dict[1] = 'One'
my_new_dict['42'] = 42
my_new_dict


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
But there's a problem here: this is a catch-all exception, and will sometimes give us misleading information.  For example:
Can you write Python code for it?


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
Huh. That doesn't look like $\pi$.  Maybe we need more terms?
Can you write Python code for it?


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
Note that if your code contains even a single syntax error, none of it will run:
Can you write Python code for it?


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
Notice that the first block executes up until the point of the Runtime error.
Once the error is hit, the except block is executed.
One important note: the above clause catches any and all exceptions.  It is not
generally a good idea to catch-all.  Better is to name the precise exception you expect:
Can you write Python code for it?

def safe_divide(a, b):
    try:
        return a / b
    except:
        # print("oops, dividing by zero. Returning None.")
        return None

print(safe_divide(15, 3))
print(safe_divide(1, 0))


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
Remember this lesson, and always specify your except statements!  I once spent an entire day tracing down a bug in my code which amounted to this.
Raising Your Own Exceptions
When you write your own code, it's good practice to use the raise keyword to create your own exceptions
when the situation calls for it:
Can you write Python code for it?

import os  # the "os" module has useful operating system stuff

def read_file(filename):
    if not os.path.exists(filename):
        raise ValueError("'{0}' does not exist".format(filename))
    f = open(filename)
    result = f.read()
    f.close()
    return result


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
Semantic Errors
Semantic errors are perhaps the most insidious errors, and are by far the ones that will take most of your time.  Semantic errors occur when the code is syntactically correct, but produces the wrong result.
By way of example, imagine you want to write a simple script to approximate the value of $\pi$ according to the following formula:
$$
\pi = \sqrt{12} \sum_{k = 0}^{\infty} \frac{(-3)^{-k}}{2k + 1}
$$
You might write a function something like this, using numpy's vectorized syntax:
Can you write Python code for it?

total = 0
for k in ks:
    total += (-3.0) ** -k / (2 * k + 1)

from math import sqrt

def approx_pi(nterms=100):
    ks = np.arange(nterms)
    ans = sqrt(12) * np.sum([-3.0 ** -k / (2 * k + 1) for k in ks])
    if ans < 1:
        import pdb; pdb.set_trace()
    return ans
approx_pi(1000)


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
Unlike Syntax errors, RunTime errors occur during code execution, which means that valid code occuring before the runtime error will execute:
Can you write Python code for it?

spam = "my all-time favorite"
eggs = 1 / 0

print(spam)


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
We'll use IPython's %%file magic to quickly create a text file
Can you write Python code for it?

%%file tmp.txt
this is the contents of the file

read_file('tmp.txt')

read_file('file.which.does.not.exist')


I am working on the file "PreFall2018/07-Exceptions.ipynb".
The first task is:
When Things Go Wrong:
Exceptions and Errors
Today we'll cover perhaps one of the most important aspects of using Python: dealing with errors and bugs in code.
Three Classes of Errors
Types of bugs/errors in code, from the easiest to the most difficult to diagnose:

Syntax Errors: Errors where the code is not valid Python (generally easy to fix)
Runtime Errors: Errors where syntactically valid code fails to execute (sometimes easy to fix)
Semantic Errors: Errors in logic (often very difficult to fix)

Syntax Errors
Syntax errors are when you write code which is not valid Python.  For example:
Can you write Python code for it?

X = [1, 2, 3
a = 4

y = 4*x + 3

def f():
    return GARBAGE


I am working on the file "PreFall2018/Deep-and-Shallow-Copies.ipynb".
The first task is:
There are different ways in which an object can be copied. Typically, these are characterized as:
- Deep copy: Allocate new memory and copy the object to this memory. Modifications to this memory has no effect on the original object.
- Shallow copy: Create a pointer to the object. Modifications to the object are seen by all code with a pointer to the object.
In general, assigning one python variable to another creates a shallow copy.
Can you write Python code for it?

# Create a list
list1 = [1, 2, 3]
# Make a copy
list2 = list1
# Use "id" to see if list1 and list2 point to the same object
print(id(list1), id(list2))


I am working on the file "PreFall2018/Deep-and-Shallow-Copies.ipynb".
The first task is:
To get a deep copy, you  must use a special method. For lists use the list function. For pandas.DataFrame, use the copy method.
Can you write Python code for it?

# Making a deep copy of a list.
list1 = [1, 2, 3]
list2 = list(list1)  # Make a copy of var3 to manipulate it
print(id(list1), id(list2))
list2.append(-1)
# Changing var4 does not change var3
print(list1, list2)

# Deep and shallow copies for DataFrames
import pandas as pd
df1 = pd.DataFrame({'a':range(10), 'b': range(10)})
df1
df2 = df1
del df2['a']
df1

print(id(df1), id(df2))


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Back to strings
Armed with python "hash" and the modulus, we can finish our hash table for words.
Can you write Python code for it?

# Load the book in again

with open('leaves-of-grass.txt','r') as f:
    words = [word.lower() for line in f for word in re.findall(r"[\w']+|[.,!?;]", line)]

k = 100000
words_hashtable = [list() for i in range(k)]

for word in words:
    word_list = words_hashtable[hash(word) % k]
    word_list.append(word)

words_hashtable

def contains_hashtable(words_hashtable, k, word):
    word_list = words_hashtable[hash(word) % k]
    print contains(word_list, word)

print contains(words, 'captain')

contains_hashtable(words_hashtable, 100000, 'lincoln')


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Hash table Load Factor
$\lambda = \frac{N_{elements}}{M_{capacity}}$
Typical values are around .5 to .75 to give good performance.
The end!
Bonus: Sorted lists
Can you write Python code for it?

print "leaves" < "zebra"

print "leaves" < "abc"

words.sort()

print words[0]

print words[30000]

print words[90000]

print words[148186]

book.__contains__("captain")


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
How can we get yes/no?
Instead of a bitmap to True/False, let's store the data directly! One list per entry in the map.
Can you write Python code for it?

k = 100000

words_map = [list() for i in range(k)]

words_map


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
How do we add a number to words_map?

Use mod to determine the index
Add the number itself to the list at the index
Can you write Python code for it?

number = 3

word_list = words_map[3 % k]

word_list.append(3)

words_map

number = 100003

word_list = words_map[number % k]
word_list.append(number)

words_map


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
If we take $N % k$, the number will always be between $0$ and $k - 1$
Can you write Python code for it?

k = 100000

words_map = [False for i in range(k)]

words_map

len(words_map)


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
If we want to say the integer 2 is in our word set, we set words_map[2] = True
Can you write Python code for it?


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
In the worst case, we have to check all words.
Can you write Python code for it?


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Is this fast? What does fast mean?
Let's say that the most expensive operation for us is a single comparison, word == element
In the best case, we match on the first word.
Let's add a counter to our contains function.
Can you write Python code for it?

def contains(wordlist, word):
    if len(wordlist) == 0:
        # Wordlist is empty
        print "Checked 1 word(s)"
        return False
    for i in xrange(len(wordlist)):
        element = wordlist[i]
        if word == element:
            print "Checked", i+1, "word(s)"
            return True
    # No match in wordlist
    print "Checked", i+1, "word(s)"
    return False

print contains(words, 'leaves')

print contains(words, 'captain')


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Populate our word_map
Can you write Python code for it?

for i in words:
    words_map[i%k] = True

words_map

words_map[5]

5 % k

100005 % k


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Query, is number in word_map?
Can you write Python code for it?

def contains_modmap(word_modmap, k, word):
    word_list = word_modmap[word % k]
    print contains(word_list, word)

contains_modmap(words_map, 100000, 100003)

words_map[3 % 100000]


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Starting from our empty set, let's load our wordslist in
Can you write Python code for it?

words_map = [False for i in range(1000000)]
for i in words:
    words_map[i] = True

contains(words, 219943)

words_map[219943]


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Summary
We've figured out how to solve the query problem for a set of integer "words", but only if drawing from a fixed number of ints, e.g. 0 to 999,999
How might we apply this to the book problem?

Maybe we only care about words in the English language

~200,000 possible words, can make bitmap of size 200,000
Have some dictionary code for each word, e.g. aardvark = 3, zebra = 199,847
If you searched for a word outside the vocab, answer would be "I don't know"
New words are entering the language constantly!

We could make the bitmap big enough to store every possible word formed with lower case letters

There are 26 characters. There are $26^k$ possible strings of length $k$
Can you write Python code for it?

for i in range(2,10):
    print "Word of length %d has %d possible combinations" % (i, 26**i)


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
The modulus
We'll use the remainder of division by some number to reduce the infinite space of things to a list with length of our choosing.
Can you write Python code for it?


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
This is called a collision. In a basic hashing bitmap, we can only say with certainty is a number is NOT in the map
Can you write Python code for it?

def contains_modmap(word_modmap, k, word):
    print "Checked 1 word(s)"
    if word_modmap[word % k]:
        return "Maybe"
    else:
        return False

contains_modmap(words_map, 100000, 219943)


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
This is less than a MB of text! What if we wanted to search through every book in the library of congress?
Runtime Analysis
Determining average and worst-case runtimes can be complicated, but for a simple for loop it is easy. Since our worst case runtime is bounded by the number of words ($n$) in the list, inceasing the number of words in the list increases the runtime by a linear factor, i.e. for $2n$, the worst case is $2n$ operations.
Other algorithms will scale in different ways, for example $n^2$ or even $2^n$. There are more details, but these runtimes are referred to as "Big O", so a linear algorithm is $O(n)$.
How can we do better?
What is faster than $O(n)$?
If all we can do is access the list by index, we can't do any better given our python list. We can actually do much better if the list is sorted, but sorting a list is worst case $O(n*\log n)$, which is even slower than our look up method.
Simpler problem: list of ints
Let's say that instead of strings, our "words" are just integers randomly chosen from 0 to 1 million
Can you write Python code for it?

import random

print random.randint(0, 10**6)
print random.randint(0, 10**6)
print random.randint(0, 10**6)
print random.randint(0, 10**6)

n = len(words)
print n

words = [random.randint(0, 10**6 - 1) for i in range(n)]

words[0:10]


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Use case: fast queries
We can afford to process the data for 1 day, but queries should run in 1 second. How can we do this for arbitary lists.
Since we know that the number of integers is bounded by [0,100000) we can make a custom data structure to represent our unsorted set of words. Its form is a list, where the index is the integer in question, and the data is a boolean representing true or false (in the set, not in the set.)
This type of data structure is sometimes called a bit map. It maps the integer field to a "bit" (yes or no)
Can you write Python code for it?

# Start with the empty set of ints from 1 to 999,999
words_map = [False for i in range(1000000)]

words_map[0:10]


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
We can still use our function to check contains, because python is forgiving that way.
Can you write Python code for it?

contains(words, 1)

contains(words, 9999)

contains(words, 100000)

contains(words, 413351)


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
We set k equal to the length of our array. We can use any size of array by changing the mod k.
More about hash functions
There are many different hash functions. A good hash function will evenly distribute hashed values onto the fixed space.
Python has a built-in hash function for exactly this purpose!
Can you write Python code for it?

hash(0)

hash(1)

hash("Ryan")

hash("Ryan") % 10


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
What if each comparison took 1 second?
Can you write Python code for it?

print "Worst case is %.1f days to query 'words'" % (len(words)/60./60/24)


I am working on the file "PreFall2018/Hashtables/Data Structures and Hashing.ipynb".
The first task is:
Without 'in', how do we do this?
We have no idea how Python is doing this under the hood. Let's try this without any python keywords.
Can you write Python code for it?

def contains(wordlist, word):
    if len(wordlist) == 0:
        # Wordlist is empty
        return False
    for i in xrange(len(wordlist)):
        element = wordlist[i]
        if word == element:
            return True
    # No match in wordlist
    return False

print contains(words, 'captain')

print contains(words, 'internet')


I am working on the file "PreFall2018/Objects/Building Software With Objects.ipynb".
The first task is:
A class diagram provides a more compact representation of a class. There are three sections.
- Class name
- Attributes
- Methods
Instance methods
- functions associated with the objects constructed for a class
- provide a way to transform data in objects
- use instance attributes (references to variables beginning with "self.")
Can you write Python code for it?

class Car(object):

    def __init__(self, color, car_type, speed):
        """
        :param str color:
        :param str car_type:
        :param int speed:
        """
        self.color = color
        self.car_type = car_type
        self.speed = speed

    def start(self):
        print ("%s %s started!" % (self.color, self.car_type))

    def stop(self):
        pass

    def turn(self, direction):
        """
        :parm str direction: left or right
        """
        pass

car = Car("Blue", "Sedan", 100)
car.start()


I am working on the file "PreFall2018/Objects/Building Software With Objects.ipynb".
The first task is:
Attributes are data associated with an object (instance) or class. Object attributes (and methods) are specified by using "self". Instance attributes and methods are accessed using the dot "." operator.
Can you write Python code for it?

class Car(object):

    # The following method is called when the class
    # is created or "constructed". The variables "self.x" refers
    # to the variable "x" in a created object.
    def __init__(self, color, car_type, speed):
        self.color = color
        self.car_type = car_type
        self.speed = speed

class Car(object):

    # The following method is called when the class
    # is created or "constructed". The variables "self.x" refers
    # to the variable "x" in a created object.
    def __init__(self, color, car_type, speed):
        self.color = color
        self.car_type = car_type
        self.speed = speed

# Creating an object for a class with arguments in the __init__ method
car = Car("Blue", "HatchBack", 100)
car.color

# Creating an object for a class with arguments in the __init__ method
joe_car = Car("Blue", "Sedan", 100)
dave_car = Car("Red", "Sports", 150)
print ("Type of joe_car is %s. Type of dave_car is %s"% (type(joe_car), type(dave_car)))

# Accessed instance attributes
joe_car = Car("Blue", "Sedan", 100)
print ("Type of joe_car has (color, type, speed)=%s." % str((joe_car.color, joe_car.car_type, joe_car.speed)))


I am working on the file "PreFall2018/Objects/Building Software With Objects.ipynb".
The first task is:
EXERCISE: Implement the stop and turn methods. Run the methods.
Inheritance
Inheritance is a common way that classes reuse data and code from other classes. A child class or derived class gets attributes and methods from its parent class.
Programmatically:
- Specify inheritance in the class statement
- Constructor for derived class (class that inherits) have access to the constructor of its parent.
Inheritance is represented in diagrams as an arror from the child class to its parent class.
Can you write Python code for it?

from IPython.display import Image
Image(filename='SimpleClassHierarchy.png', width=400, height=400)

# Code for inheritance
class Sedan(Car):
    # Sedan inherits from car

    def __init__(self, color, speed):
        """
        :param str color:
        :param int speed:
        """
        super().__init__(color, "Sedan", speed)

    def play_cd(self):
        print ("Playing cd in %s sedan" % self.color)

sedan = Sedan("Yellow", 1e6)
sedan.play_cd()

sedan.car_type

joe_car = Sedan("Blue", 100)
print ("Type of joe_car has (color, type, speed)=%s." % str((joe_car.color, joe_car.car_type, joe_car.speed)))


I am working on the file "PreFall2018/Objects/Building Software With Objects.ipynb".
The first task is:
Initial concepts

An object is a container of data (attributes) and code (methods)
A class is a template for creating objects

Reuse is provided by:

reusing the same class to create many objects
"inheriting" data and code from other classes
Can you write Python code for it?

# Definiting a Car class
class Car(object):
    pass


I am working on the file "PreFall2018/Objects/Building Software With Objects.ipynb".
The first task is:
Look at Objects/ATMDiagrams.pdf for a solution.
What Else in Design?

Other diagrams: state diagrams, package diagrams, ...
Object oriented design patterns

Complex Example of Class Hierarchy
Can you write Python code for it?

from IPython.display import Image
Image(filename='SciSheetsCoreClasses.png', width=300, height=30)


I am working on the file "PreFall2018/Objects/Building Software With Objects.ipynb".
The first task is:
The diamond arrow is a "has-a" relationship. For example, the Controller has-a ATMInput. This means that a Controller object has an instance variable for an ATMInput object.
Interaction Diagram for the ATM System
An interaction diagram specifies how components interact to achieve a use case.
Interactions are from one object to another object, indicating that the first object calls a method in the second object.
Rules for drawing lines in an interaction diagram:
- The calling object must know about the called object.
- The called object must have the method invoked by the calling object.
Can you write Python code for it?

from IPython.display import Image
Image(filename='ATMAuthentication.png', width=800, height=800)


I am working on the file "PreFall2018/Python Functions.ipynb".
The first task is:
Mechanics of Writing a Function

Function definition line - How python knows that this is a function
Function body - code that does the computation of the function
Formal arguments - arguments passed to the function
Return values - value returned to the caller
Name scopes
Can you write Python code for it?

# Transforming the prime number calculation script into a function
def check_prime(num):
    upper = int(np.sqrt(num)) + 1
    for integer in range(2, upper):
        if num % integer == 0:
            return False
    return True

check_prime(-3)


I am working on the file "PreFall2018/Unit-Tests/unit-tests-completed.ipynb".
The first task is:
Code for homework or your work should use test files. In this lesson, we'll show how to write test codes in a Jupyter notebook. This is done for pedidogical reasons. It is NOT not something you should do in practice, except as an intermediate exploratory approach.
As expected, the first test passes, but the second test fails.
Exercise

Rewrite the above one-shot test for entropy using the unittest infrastructure.
Can you write Python code for it?

# Implementating a pattern test. Use functions in the test.
import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_equal_probability(self):
        def test(count):
            """
            Invokes the entropy function for a number of values equal to count
            that have the same probability.
            :param int count:
            """
            raise RuntimeError ("Not implemented.")
        #
        test(2)
        test(20)
        test(200)

#test_setup(TestEntropy)

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):
  """Write the full set of tests."""


I am working on the file "PreFall2018/Unit-Tests/unit-tests-completed.ipynb".
The first task is:
Now let's consider a pattern test. Examining the structure of the calculation of $H$, we consider a situation in which there are $n$ equal probabilities. That is, $p_i = \frac{1}{n}$.
$$
H = -\sum_{i=1}^{n} p_i \log(p_i)
= -\sum_{i=1}^{n} \frac{1}{n} \log(\frac{1}{n})
= n (-\frac{1}{n} \log(\frac{1}{n}) )
= -\log(\frac{1}{n})
$$
For example, entropy([0.5, 0.5]) should be $-log(0.5)$.
Can you write Python code for it?

# Pattern test
def test_equal_probabilities(n):
    prob = 1.0/n
    ps = np.repeat(prob , n)
    if not np.isclose(entropy(ps), -np.log(prob)):
        import pdb; pdb.set_trace()
        print ("Bad result.")
    else:
        print("Worked!")

# Run a test
test_equal_probabilities(100)


I am working on the file "PreFall2018/Unit-Tests/unit-tests-completed.ipynb".
The first task is:
Testing For Exceptions
Edge test cases often involves handling exceptions. One approach is to code this directly.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        try:
            entropy([0.1, 0.5])
            self.assertTrue(False)
        except ValueError:
            self.assertTrue(True)

#test_setup(TestEntropy)


I am working on the file "PreFall2018/Unit-Tests/unit-tests-completed.ipynb".
The first task is:
You see that there are many, many cases to test. So far, we've been writing special codes for each test case. We can do better.
Unittest Infrastructure
There are several reasons to use a test infrastructure:
- If you have many test cases (which you should!), the test infrastructure will save you from writing a lot of code.
- The infrastructure provides a uniform way to report test results, and to handle test failures.
- A test infrastructure can tell you about coverage so you know what tests to add.
We'll be using the unittest framework. This is a separate Python package. Using this infrastructure, requires the following:
1. import the unittest module
1. define a class that inherits from unittest.TestCase
1. write methods that run the code to be tested and check the outcomes.
The last item has two subparts. First, we must identify which methods in the class inheriting from unittest.TestCase are tests. You indicate that a method is to be run as a test by having the method name begin with "test".
Second, the "test methods" should communicate with the infrastructure the results of evaluating output from the code under test. This is done by using assert statements. For example, self.assertEqual takes two arguments. If these are objects for which == returns True, then the test passes. Otherwise, the test fails.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class UnitTests(unittest.TestCase):

    # Each method in the class to execute a test
    def test_success(self):
        self.assertEqual(1, 1)

    def test_success1(self):
        self.assertTrue(1 == 1)

    def test_failure(self):
        self.assertEqual(1, 2)

suite = unittest.TestLoader().loadTestsFromTestCase(UnitTests)
_ = unittest.TextTestRunner().run(suite)

# Function the handles test loading
#def test_setup(argument ?):


I am working on the file "PreFall2018/Unit-Tests/unit-tests-completed.ipynb".
The first task is:
unittest provides help with testing exceptions.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        with self.assertRaises(ValueError):
            entropy([0.1, 0.5])

suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
Code for homework or your work should use test files. In this lesson, we'll show how to write test codes in a Jupyter notebook. This is done for pedidogical reasons. It is NOT not something you should do in practice, except as an intermediate exploratory approach.
As expected, the first test passes, but the second test fails.
Exercise

Rewrite the above one-shot test for entropy using the unittest infrastructure.
Can you write Python code for it?

# Implementating a pattern test. Use functions in the test.
import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_equal_probability(self):
        def test(count):
            """
            Invokes the entropy function for a number of values equal to count
            that have the same probability.
            :param int count:
            """
            raise RuntimeError ("Not implemented.")
        #
        test(2)
        test(20)
        test(200)

 suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
Now let's consider a pattern test. Examining the structure of the calculation of $H$, we consider a situation in which there are $n$ equal probabilities. That is, $p_i = \frac{1}{n}$.
$$
H = -\sum_{i=1}^{n} p_i \log(p_i)
= -\sum_{i=1}^{n} \frac{1}{n} \log(\frac{1}{n})
= n (-\frac{1}{n} \log(\frac{1}{n}) )
= -\log(\frac{1}{n})
$$
For example, entropy([0.5, 0.5]) should be $-log(0.5)$.
Can you write Python code for it?

# Pattern test
print (entropy([0.5, 0.5]), entropy([1/3, 1/3, 1/3]), entropy(np.repeat(1/20, 20)))


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
Suppose that all of the probability of a distribution is at one point. An example of this is a coin with two heads. Whenever you flip it, you always get heads. That is, the probability of a head is 1.
What is the entropy of such a distribution? From the calculation above, we see that the entropy should be $log(1)$, which is 0. This means that we have a test case where we know the result!
Can you write Python code for it?

# One-shot test. Need to know the correct answer.
SMALL_VALUE = 1e-5
entropy([SMALL_VALUE, 1-SMALL_VALUE])


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
Test Files
Although I presented the elements of unittest in a notebook. your tests should be in a file. If the name of module with the code under test is foo.py, then the name of the test file should be test_foo.py.
The structure of the test file will be very similar to cells above. You will import unittest. You must also import the module with the code under test. Take a look at test_prime.py in this directory to see an example.
Discussion
Question: What tests would you write for a plotting function?
Test Driven Development
Start by writing the tests. Then write the code.
We illustrate this by considering a function geomean that takes a list of numbers as input and produces the geometric mean on output.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntryopy(unittest.TestCase):

    def test_oneshot(self):
        self.assertEqual(geomean([1,1]), 1)

    def test_oneshot2(self):
        self.assertEqual(geomean([3, 3, 3]), 3)


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
Testing For Exceptions
Edge test cases often involves handling exceptions. One approach is to code this directly.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        try:
            entropy([0.1, -0.5])
            self.assertTrue(False)
        except ValueError:
            self.assertTrue(True)

suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
Unit Tests
Overview and Principles
Testing is the process by which you exercise your code to determine if it performs as expected. The code you are testing is referred to as the code under test.
There are two parts to writing tests.
1. invoking the code under test so that it is exercised in a particular way;
1. evaluating the results of executing code under test to determine if it behaved as expected.
The collection of tests performed are referred to as the test cases. The fraction of the code under test that is executed as a result of running the test cases is referred to as test coverage.
For dynamical languages such as Python, it's extremely important to have a high test coverage. In fact, you should try to get 100% coverage. This is because little checking is done when the source code is read by the Python interpreter. For example, the code under test might contain a line that has a function that is undefined. This would not be detected until that line of code is executed.
Test cases can be of several types. Below are listed some common classifications of test cases.
- Smoke test. This is an invocation of the code under test to see if there is an unexpected exception. It's useful as a starting point, but this doesn't tell you anything about the correctness of the results of a computation.
- One-shot test. In this case, you call the code under test with arguments for which you know the expected result.
- Edge test. The code under test is invoked with arguments that should cause an exception, and you evaluate if the expected exception occurrs.
- Pattern test - Based on your knowledge of the calculation (not implementation) of the code under test, you construct a suite of test cases for which the results are known or there are known patterns in these results that are used to evaluate the results returned.
Another principle of testing is to limit what is done in a single test case. Generally, a test case should focus on one use of one function. Sometimes, this is a challenge since the function being tested...
Can you write Python code for it?

import numpy as np
# Code Under Test
def entropy(ps):
    items = ps * np.log(ps)
    if any(np.isnan(items)):
        raise ValueError("Cannot compute log of ps!")
    return -np.sum(items)

np.isnan([.1, .9])

# Smoke test
entropy([0.5, 0.5])


I am working on the file "PreFall2018/Unit-Tests/unit-tests.ipynb".
The first task is:
You see that there are many, many cases to test. So far, we've been writing special codes for each test case. We can do better.
Unittest Infrastructure
There are several reasons to use a test infrastructure:
- If you have many test cases (which you should!), the test infrastructure will save you from writing a lot of code.
- The infrastructure provides a uniform way to report test results, and to handle test failures.
- A test infrastructure can tell you about coverage so you know what tests to add.
We'll be using the unittest framework. This is a separate Python package. Using this infrastructure, requires the following:
1. import the unittest module
1. define a class that inherits from unittest.TestCase
1. write methods that run the code to be tested and check the outcomes.
The last item has two subparts. First, we must identify which methods in the class inheriting from unittest.TestCase are tests. You indicate that a method is to be run as a test by having the method name begin with "test".
Second, the "test methods" should communicate with the infrastructure the results of evaluating output from the code under test. This is done by using assert statements. For example, self.assertEqual takes two arguments. If these are objects for which == returns True, then the test passes. Otherwise, the test fails.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class UnitTests(unittest.TestCase):

    # Each method in the class to execute a test
    def test_success(self):
        self.assertEqual(1, 1)

    def test_success1(self):
        self.assertTrue(1 == 1)

    def test_failure(self):
        self.assertEqual(1, 1)

suite = unittest.TestLoader().loadTestsFromTestCase(UnitTests)
_ = unittest.TextTestRunner().run(suite)

import unittest

# Define a class in which the tests will run
class UnitTests(unittest.TestCase):

    # Each method in the class to execute a test
    def test_success(self):
        self.assertEqual(1, 1)

    def test_success1(self):
        self.assertTrue(1 == 1)

    def test_failure(self):
        self.assertEqual(1, 1)

suite = unittest.TestLoader().loadTestsFromTestCase(UnitTests)
_ = unittest.TextTestRunner().run(suite)

# Function the handles test loading
#def test_setup(argument ?):


I am working on the file "PreFall2018/Visualization-in-Depth/Visualization-in-depth.ipynb".
The first task is:
Bokeh tools
Tools can be specified and positioned when the Figure is created. The interaction workflow is (a) select a tool (identified by vertical blue line), (b) perform gesture for tool.
Can you write Python code for it?

TOOLS = 'box_zoom,box_select,resize,reset'
plot = figure(plot_width=200, plot_height=200, title=None, tools=TOOLS)
plot.scatter(range(10), range(10))
show(plot)

from bokeh.models import HoverTool, BoxSelectTool
TOOLS = [HoverTool(), BoxSelectTool()]
plot = figure(plot_width=200, plot_height=200, title=None, tools=TOOLS)
show(plot)


I am working on the file "PreFall2018/Visualization-in-Depth/Visualization-in-depth.ipynb".
The first task is:
Exercise: Plot the biological data with colors and a hover that shows the evolutionary line.
Bokeh Widgets
See widget.py and my_app.py
Bokeh Server
Can you write Python code for it?


I am working on the file "PreFall2018/Visualization-in-Depth/Visualization-in-depth.ipynb".
The first task is:
Now add ad-hoc data
Can you write Python code for it?

from bokeh.plotting import figure, output_file, show, ColumnDataSource
from bokeh.models import HoverTool

output_file("toolbar.html")

hover = HoverTool(
        tooltips=[
            ("index", "$index"),
            ("(x,y)", "(@x, @y)"),
            ("desc", "@desc"),
        ]
    )

p = figure(plot_width=400, plot_height=400, tools=[hover],
           title="Mouse over the dots")

source = ColumnDataSource(
        data={
            'x': [1, 2, 3, 4, 5],
            'y': [2, 5, 8, 2, 7],
            'desc': ['A', 'b', 'C', 'd', 'E'],
        }
    )

p.circle('x', 'y', size=20, source=source)

show(p)


I am working on the file "PreFall2018/Visualization-in-Depth/Visualization-in-depth.ipynb".
The first task is:
Objectives

Describe how to create interactive visualizations using bokeh
Running example and visualization goals
How to approach a new system
Hover
Widgets
Study bokeh as a system
Function specification - what it provides programmers
Design (which has client and server parts)

How to Learn a New System
Steps
- Find an example close to what you want
- Create an environment that runs the example
- Abstract the key concepts of how it works
- Transform the example into what you want
Running Example - Biological Data
Can you write Python code for it?

from IPython.display import Image
Image(filename='biological_data.png')

df_bio = pd.read_csv("biological_data.csv")
df_bio.head()


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Getting More Control Over Plots
Let's take a more detailed approach to plotting so we can better control what gets rendered.
In this section, we show how to control various elements of plots to produce a desired visualization. We'll use the package matplotlib, a python package that is modelled after MATLAB style plotting.
Make a dataframe out of the count data.
Can you write Python code for it?

df_counts = pd.DataFrame({'From': from_counts.sort_index(), 'To': to_counts.sort_index()})


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Let's start with the values for starttime. What type are these?
Can you write Python code for it?

print (df.starttime[0])
print (type(df.starttime[0]))


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Need to align the counts by the station. Do we do this?
Can you write Python code for it?

df_counts.head()

"""
Basic bar chart using matplotlib
"""
n_groups = len(df_counts.index)
index = np.arange(n_groups)  # The "raw" x-axis of the bar plot

fig = plt.figure(figsize=(12, 8))  # Controls global properties of the bar plot
rects1 = plt.bar(index, df_counts.From)
plt.xlabel('Station')
plt.ylabel('Counts')
plt.xticks(index, df_counts.index)  # Convert "raw" x-axis into labels
_, labels = plt.xticks()  # Get the new labels of the plot
plt.setp(labels, rotation=90)  # Rotate labels to make them readable
plt.title('Station Counts')
plt.show()


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Now we need to compute the average value and its standard deviation across the days for each station.
The groupby produced a MultiIndex. So, further operations on the result must take this into account.
Can you write Python code for it?

h_index = groupby_day_from.index
h_index.levshape  # Size of the components of the MultiIndex

from_means = groupby_day_from.groupby(level=[0]).mean()  # Computes the mean of counts by day
from_stds = groupby_day_from.groupby(level=[0]).std()   # Computes the standard deviation

groupby_day_to = df.groupby(['to_station_id', 'startday']).size()
to_means = groupby_day_to.groupby(level=[0]).mean()  # Computes the mean of counts by day
to_stds = groupby_day_to.groupby(level=[0]).std()   # Computes the standard deviation

df_day_counts = pd.DataFrame({'from_mean': from_means, 'from_std': from_stds, 'to_mean': to_means, 'to_std': to_stds})
df_day_counts.head()


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Plotting with Error Bars
Can you write Python code for it?

"""
Plotting two variables as a bar chart with error bars
"""
n_groups = len(df_day_counts.index)
index = np.arange(n_groups)  # The "raw" x-axis of the bar plot
fig = plt.figure(figsize=(12, 8))  # Controls global properties of the bar plot
bar_width = 0.35  # Width of the bars
opacity = 0.6  # How transparent the bars are

#VVVV Changed to do two plots with error bars
error_config = {'ecolor': '0.3'}
rects1 = plt.bar(index, df_day_counts.from_mean, bar_width,
                 alpha=opacity,
                 color='b',
                 yerr=df_day_counts.from_std,
                 error_kw=error_config,
                 label='From')
rects2 = plt.bar(index + bar_width, df_day_counts.to_mean, bar_width,
                 alpha=opacity,
                 color='r',
                 yerr=df_day_counts.to_std,
                 error_kw=error_config,
                 label='to')
#^^^^ Changed to do two plots with error bars

plt.xticks(index + bar_width / 2, df_counts.index)
_, labels = plt.xticks()  # Get the new labels of the plot
plt.setp(labels, rotation=90)  # Rotate labels to make them readable
plt.legend()

plt.xlabel('Station')
plt.ylabel('Counts')
plt.title('Station Counts')
plt.show()


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Question: How we would get the same information using groupby?
Simple Plots for Series
Let's address the question "Which stations have the biggest difference between the in-flow and out-flow of bicycles?"
What kind of objects are returned from pd.value_counts? Are these plottable? How do we figure this out?
Can you write Python code for it?


I am working on the file "PreFall2018/Visualization-in-Python/Visualization in Python.ipynb".
The first task is:
Some issues:
- Bogus value 'Pronto shop'
- Difficult to read the labels on the x-axis
- The x and y axis aren't labelled
- Lost information about "from" and "to"
Writing a Data Cleansing Function
We want to get rid of the row 'Pronto shop' in both from_counts and to_counts.
Can you write Python code for it?

# Selecting a row
from_counts[from_counts.index == 'Pronto shop']

# Deleting a row
new_from_counts = from_counts[from_counts.index != 'Pronto shop']
new_from_counts.plot.bar()

def simple_clean_rows(df):
    """
    Removes from df all rows with the specified indexes
    :param pd.DataFrame or pd.Series df:
    :return pd.DataFrame or pd.Series:
    """

    df = df[df.index != 'Pronto Shop']
    return df

def clean_rows(df, indexes):
    """
    Removes from df all rows with the specified indexes
    :param pd.DataFrame or pd.Series df:
    :param list-of-str indexes
    :return pd.DataFrame or pd.Series:
    """
    for idx in indexes:
        df = df[df.index != idx]
    return df

dff = clean_rows(to_counts, ['Pronto Shop', 'CBD-13'])
dff.plot.bar()


I am working on the file "PreProcessing/Adjust_Betas.ipynb".
The first task is:
EPIC
Can you write Python code for it?

betas = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'betas')
betas = betas['s3']
betas = betas.groupby(axis=0, level=0).first()

cell_counts = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'cell_counts')
cell_counts = cell_counts.groupby(level=0).first()
cell_counts = cell_counts.ix[betas.columns]

flow_sorted_data = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5','flow_sorted_data')
flow_sorted_data = flow_sorted_data.groupby(level=0).first()
flow_sorted_data = flow_sorted_data.ix[betas.index]
cell_type = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5', 'label_map')

n2 = flow_sorted_data.groupby(cell_type, axis=1).mean()
avg = n2[cell_counts.columns].dot(cell_counts.T)

cc = avg.mean(1)
adj = (betas - avg).add(cc, axis=0)
adj = adj.dropna(how='all', axis=1)

store.append('EPIC_adj', adj)
store.create_table_index('EPIC_adj', optlevel=9, kind='full')


I am working on the file "PreProcessing/Adjust_Betas.ipynb".
The first task is:
Hannum
Read in quantile-normalized beta values and cell counts from the MINFI pipeline.
Can you write Python code for it?

betas = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'betas')
betas = betas['s1']
betas = betas.groupby(axis=0, level=0).first()

cell_counts = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'cell_counts')
cell_counts = cell_counts.groupby(level=0).first()
cell_counts = cell_counts.ix[betas.columns]


I am working on the file "PreProcessing/Adjust_Betas.ipynb".
The first task is:
Read in cell compositions from probe annotations.
Can you write Python code for it?

flow_sorted_data = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5','flow_sorted_data')
flow_sorted_data = flow_sorted_data.groupby(level=0).first()
flow_sorted_data = flow_sorted_data.ix[betas.index]
cell_type = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5', 'label_map')

n2 = flow_sorted_data.groupby(cell_type, axis=1).mean()
avg = n2[cell_counts.columns].dot(cell_counts.T)


I am working on the file "PreProcessing/BMIQ_Horvath.ipynb".
The first task is:
Normalization Step
Can you write Python code for it?

gold_standard = pd.read_csv('/cellar/users/agross/Data/MethylationAge/Horvath/probeAnnotation21kdatMethUsed.csv', index_col=0)
horvath = pd.read_table('/cellar/users/agross/TCGA_Code/Methlation/data/Horvath_Model.csv', index_col=0, skiprows=[0,1])
intercept = horvath.CoefficientTraining['(Intercept)']
horvath = horvath.iloc[1:]

df = df.ix[gold_standard.index]
df = df.T.fillna(gold_standard.goldstandard2).T

df_r = robjects.r.t(convert_to_r_dataframe(df))
gs = list(gold_standard.goldstandard2.ix[df.index])
gs_r = robjects.FloatVector(gs)

del df

data_n = robjects.r.BMIQcalibration(df_r, gs_r)
data_n = convert_robj(data_n).T
clear_output()


I am working on the file "PreProcessing/BMIQ_Horvath.ipynb".
The first task is:
Read in Betas
Can you write Python code for it?

path = '/cellar/users/agross/TCGA_Code/Methlation/data/'
f = path + 'all_betas_raw.csv'
df = pd.read_csv(f, low_memory=True, header=0, index_col=0)

labels = pd.read_csv(path + 'all_betas_raw_pdata.csv',
                     index_col=0)


I am working on the file "PreProcessing/BMIQ_Normalization.ipynb".
The first task is:
Adjust data to account for differences in cellular composition.
Can you write Python code for it?

study = betas.columns
study = pd.Series(study.get_level_values(0), study.get_level_values(1))
betas.columns = betas.columns.get_level_values(1)

cc = avg.mean(1)
adj = (betas - avg).add(cc, axis=0)
adj = adj.dropna(how='all', axis=1)


I am working on the file "PreProcessing/BMIQ_Normalization.ipynb".
The first task is:
Here s1 is the Hannum data, which we used to train our model.  We are normalizeing all of our data with the median value of the Hannum cohort as our reference for each probe.
Can you write Python code for it?

gold_standard_ah = adj.ix[:, ti(study=='s1')].median(1)

del betas
del avg

df = adj
df = df.ix[gold_standard_ah.index]
df = df.T.fillna(gold_standard_ah).T

df_r = robjects.r.t(convert_to_r_dataframe(df))
gs = list(gold_standard_ah.ix[df.index])
gs_r = robjects.FloatVector(gs)

del df

data_n = robjects.r.BMIQcalibration(df_r, gs_r)
data_n = convert_robj(data_n).T
clear_output()

data_n.columns = data_n.columns.map(lambda s: s.replace('.','-'))
data_n.columns = data_n.columns.map(lambda s: s[1:] if s.startswith('X') else s)

store = pd.HDFStore(HDFS_DIR + 'methylation_norm.h5')
store.append('quant_BMIQ_adj', data_n)
store.create_table_index('quant_BMIQ_adj', optlevel=9, kind='full')


I am working on the file "PreProcessing/BMIQ_Normalization.ipynb".
The first task is:
Load Horvath normalization source into R namespace.
Can you write Python code for it?


I am working on the file "PreProcessing/BMIQ_Normalization.ipynb".
The first task is:
Read in quantile-normalized beta values and cell counts from the MINFI pipeline.
Can you write Python code for it?

betas = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'betas')
cell_counts = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'cell_counts')


I am working on the file "PreProcessing/BMIQ_Normalization_script.ipynb".
The first task is:
BMIQ Normalization
Here we are running BMIQ normalization on all of our quantile-normalized data together.  I am using the implementation provided by Steve Horvath along with his recent methylation-age paper.  We are doing this as a result of this paper's recomendation to run both quantile normalization and BMIQ in series on the same datasets.
Can you write Python code for it?

import pandas as pd

import rpy2.robjects as robjects
from pandas.rpy.common import convert_to_r_dataframe
from pandas.rpy.common import convert_robj

from IPython.display import clear_output


I am working on the file "PreProcessing/BMIQ_Normalization_script.ipynb".
The first task is:
Load Horvath normalization source into R namespace.
Read in cell composition adjusted, quantile-normalized beta values and cell counts from the MINFI pipeline.
Can you write Python code for it?

c = [u'3001', u'3002', u'3003', u'3004', u'3005',
     u'3006', u'3007', u'3008', u'3009', u'3010']

betas = pd.read_hdf(HDFS_DIR + 'methylation_norm.h5', 'betas_adj', columns=c)

gold_standard_ah = pd.read_hdf(HDFS_DIR + 'methylation_norm.h5', 'Hannum_gold_standard')

betas = betas.ix[gold_standard_ah.index]

if betas.isnull().sum().sum() > 0:
    betas = betas.T.fillna(gold_standard_ah).T

robjects.r.library('WGCNA');
robjects.r.source("/cellar/users/agross/Data/MethylationAge/Horvath/NORMALIZATION.R");
clear_output()

df_r = robjects.r.t(convert_to_r_dataframe(betas))
gs = list(gold_standard_ah.ix[betas.index])
gs_r = robjects.FloatVector(gs)

del betas

data_n = robjects.r.BMIQcalibration(df_r, gs_r)
data_n = convert_robj(data_n).T
clear_output()

data_n.columns = data_n.columns.map(lambda s: s.replace('.','-'))
data_n.columns = data_n.columns.map(lambda s: s[1:] if s.startswith('X') else s)

store = pd.HDFStore(HDFS_DIR + 'methylation_norm_tmp.h5')

#store = pd.HDFStore('/data_ssd/methylation_norm.h5')
#store.append('quant_BMIQ_adj', data_n)
#store.create_table_index('quant_BMIQ_adj', optlevel=9, kind='full')


I am working on the file "PreProcessing/Pan_Study_Data.ipynb".
The first task is:
Combine Clinical Data
Can you write Python code for it?

age = pd.concat([hiv_clinical.age, age_hannum, age_italy])
age.name = 'age'

gender = pd.concat([hiv_gender, gender_hannum, gender_italy])
gender.name = 'gender'

study = {}
for i in betas.columns:
    if i[0] == 's2':
        study[i[1]] = duration.ix[i[1]]
    else:
        study[i[1]] = i[0]
study = pd.Series(study)

#Do not import
o = ['s1','s3','HIV Control','HIV Short', 'HIV Long']
violin_plot_pandas(study, age, order=o)

hiv = study.isin(['HIV Short','HIV Long'])


I am working on the file "PreProcessing/Pan_Study_Data.ipynb".
The first task is:
Hannum Cohort

Genome-wide methylation profiles reveal quantitative views of human aging rates.
All relatively normal samples (from Glaucoma GWAS?)
Half are female, about 1/3 are Hispanic
Can you write Python code for it?

f = '/cellar/users/agross/Data/Hannum_Methylation/idats/Full Illumina List Nov 9.txt'
clin = pd.read_table(f, index_col=0)
clin.index = clin.index.map(str)
gender_hannum = clin.sex
age_hannum = clin.Age
ethnicity_hannum = clin.ethnicity
hannum_clinical = pd.concat([age_hannum, gender_hannum, ethnicity_hannum], axis=1,
                            keys=['age', 'gender', 'ethnicity'])

pd.crosstab(hannum_clinical.gender, hannum_clinical.ethnicity).T


I am working on the file "PreProcessing/Pan_Study_Data.ipynb".
The first task is:
Looks like some outliers in one of the controls... would be best to drop them.
Can you write Python code for it?

#Do not import
violin_plot_pandas(study, cell_counts.Bcell, order=o)

study = study.ix[ti(cell_counts.Bcell < .25)].dropna()
keepers = study.index

betas = betas.astype(float)

store = pd.HDFStore(HDFS_DIR + 'dx_methylation.h5')

store.append('betas', betas)
store.create_table_index('betas', optlevel=9, kind='full')

store['age'] = age
store['study'] = study
store['gender'] = gender

store.append('cell_counts', cell_counts)
store.create_table_index('cell_counts', optlevel=9, kind='full')

store.close()


I am working on the file "PreProcessing/Pan_Study_Data.ipynb".
The first task is:
Read in Data Normalized Together

I had a few issues getting the dataset out of R, so I had to export the data and the sample labels seperately.
Here I combine these two files together
Can you write Python code for it?

ann = pd.read_csv(path + 'data/beta_combined_quantile_labels.csv',
                  index_col=0)
idx = ann.set_index(['studyIndex','sampleNames']).index

df = pd.read_csv(path + 'data/beta_combined_quantile.csv', index_col=0,
                 low_memory=True, engine='c')
df.columns = idx
betas = df

fix_col = lambda s: s if s[0] != 's3' else (s[0], '_'.join(s[1].split('_')[1:]))
betas.columns = pd.MultiIndex.from_tuples(betas.columns.map(fix_col))


I am working on the file "PreProcessing/Pan_Study_Data.ipynb".
The first task is:
Read in HIV methylation data annotations
Can you write Python code for it?

c1 = pd.read_excel(ucsd_path + 'DESIGN_Fox_v2_Samples-ChipLAyout-Clinical UNMC-UCSD methylomestudy.xlsx',
                   'HIV- samples from OldStudy', index_col=0)
c2 = pd.read_excel(ucsd_path + 'DESIGN_Fox_v2_Samples-ChipLAyout-Clinical UNMC-UCSD methylomestudy.xlsx',
                   'HIV+ samples', index_col=0)
clinical = c1.append(c2)


I am working on the file "PreProcessing/save_detection_p_values.ipynb".
The first task is:
UCSD
Can you write Python code for it?

ucsd = pd.read_csv(PATH + 'data/UCSD_Methylation/detectionP.csv',
                index_col=0)
p = pd.read_csv(PATH + 'data/UCSD_Methylation/pData.csv',
                index_col=0)
ucsd.columns = p.Sample_Name
ucsd = ucsd.replace(0, nan)
ucsd = ucsd.stack()

detection_p = pd.concat([ucsd, hannum, epic])

detection_p = detection_p.reset_index()

detection_p.to_hdf(HDFS_DIR + 'dx_methylation.h5', 'detection_p')


I am working on the file "Precio de Casas.ipynb".
The first task is:
Aplicar el modelo en 3 casas
Can you write Python code for it?

house1 = sales[sales['id']=='5309101200']

sales

house3 = sales[sales['id'] == '6414100192']
house3

print house3['price']

print sqft_model.predict(house3)

house2 = sales[sales['id']=='3793500160']

house2['price']

house2

print sqft_model.predict(house2)

house3 = sales[sales['id'] == '1321400060']

print house3['price']

print sqft_model.predict(house3)


I am working on the file "Precio de Casas.ipynb".
The first task is:
Construir nuestro modelo de regresion
Can you write Python code for it?

sqft_model = graphlab.linear_regression.create(train_data, target='price', features=['sqft_living'],validation_set=None)


I am working on the file "Precio de Casas.ipynb".
The first task is:
Crear un simple modelo de regresión basado en sqft_living
Crear nuestro training data y nuestro test data
Can you write Python code for it?


I am working on the file "Precio de Casas.ipynb".
The first task is:
Evaluar el modelo
El promedio de las ventas
Can you write Python code for it?


I am working on the file "Precio de Casas.ipynb".
The first task is:
Explorar los datos de las casas
Una correlacion con el sqft de una casa con su precio
Can you write Python code for it?

graphlab.canvas.set_target('ipynb')
sales.show(view="Scatter Plot", x="sqft_living", y="price")


I am working on the file "Precio de Casas.ipynb".
The first task is:
Los puntos azules son nuestros datos y la linea verde es nuestra regresion lineal que usaremos para predecir los precios de las casas.
Abajo estan los coeficientes de la función.
Can you write Python code for it?


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
How does the maximum return period compare to the Weibull calculations above?
Just to take this one step further, we'll fit a linear regression to the data plotted on the log plot above using the np.polyfit command. This fits a polynomial of a specified degree to the x and y data that you supply and returns the coefficients. You can then use np.polyval to evaluate the fitted polynomial at each point in a list of x-values. Let's take a look...
Can you write Python code for it?

fitted_coeffs = np.polyfit(np.log10(W_T), ranked, 1) #Returns the slope and
#intercept of the linear regression. Note that we had to first take the
#log-base-10 of the return period (since the relationship is linear in log space)

#Generate a list of years to apply the fitted polynomial to, from the minimum
#return period to the maximum return period (log-transformed, of course)
yrs = np.log10(np.arange(np.min(W_T), np.max(W_T)+1))
yval = np.polyval(fitted_coeffs, yrs) #Solve the linear equation at each return period of interest

#Now let's remake the plot:
#Now let's plot the data as a frequency/magnitude curve.
plt.semilogx(W_T, ranked, 'ro', 10**yrs, yval, 'k-')
#Note that the semilogx command takes the log-base-10 of the x-variable.
#We ALREADY took the log of the return period in computing 'yrs', so
#to avoid taking the log of it again, we raise it to the power of 10 before
#feeding it to the 'semilogx' function.

plt.xlabel('Return period, years')
plt.ylabel('Daily rainfall, mm')
plt.title('Weibull method with fitted regression line')


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
Kind of messy! Well, it's real data. Let's see what the Gumbel and Weibull methods give us for the probability of exceeding the maximum daily rain event in this record.
Gumbel distribution
First we need to calculate alpha and beta...
Can you write Python code for it?

alpha = np.sqrt(6)/np.pi*np.std(pMD) #From the formula
beta = np.mean(pMD)-0.5772*alpha
G_exceedP = 1-np.exp(-np.exp(-(pMD-beta)/alpha)) #exceedence probability
#This is the probability that you will get a storm that delivers this amount
#of rain or more in any given year.
G_T = 1/G_exceedP #Return interval, in years

#Now let's plot the data as a frequency/magnitude curve.
plt.plot(G_exceedP, pMD, 'o')
plt.xlabel('Exceedence probability')
plt.ylabel('Daily rainfall, mm')
plt.title('Gumbel method')

#To answer the original question, we can also query the exceedence probability of the maximum.
print(G_exceedP[pMD==np.max(pMD)])
#This literally means "print the entry of G_exceedP from the same row as where
#pMD equals its maximum value."

#And again in terms of return period...
plt.plot(G_T, pMD, 'ro') #ooh, 'r' makes the dots red!
plt.xlabel('Return period, years')
plt.ylabel('Daily rainfall, mm')
plt.title('Gumbel method')


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
Let's break down what we just did. Recall that np is how we refer to a command that comes from the numpy library (whose commands are actually very similar to Matlab commands, in case you are familiar with those!). np.isnan queries each element in an array and assigns a "true" to those elements that are "NaN" and a "false" to elements that are not "NaN." It returns an array that is the same length as the argument in the command. Insert a cell below to print out the length of np.isnan(pday). Do you remember how to look at the first few entries of this array?
Next, the ~ symbol: This inverts the outcome of a logical operation. In other words, it will cause a "true" statemetn to become "false," and vice-versa.
Last, when you index a variable with a logical array, the "output" will be just those elements of the array (rows) that evaluate as "true."
Putting this all together, we are creating an array, pday_no_nan, which is equal to all of the rows of pday that are NOT NaNs. In other words, just the actual rows of daily data! We should see that the length is 1/4 that of p6, which is indeed the case!
It's generally good practice not to overwrite a variable with a variable of the same name unless you are absolutely sure that you have coded the formula correctly. In this case, our check has shown us that we have, so we can now feel free to rename pday_no_nan as pday for convenience.
Can you write Python code for it?


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
Manipulating the precipitation data
Hyetograms
Let's create a hyetogram of one of the storms that is present in the 6-hourly data. (If you'd like to practice on your own, do this for a different storm!) Browsing the input CSV file in the spreadsheet program of your choice, you see that rows 3801-3816 on the spreadsheet (remember that in python this correspondes to indices 3799-3814 (you subtract 2 because recall that the first row is the column headings, and Python indexes starting with 0) constitutes a nice storm. Let's make a hyetogram of it!
Can you write Python code for it?

mystorm = p6[3799:3814] #Grab only the data from that storm
print(mystorm) #Always a good idea to check and make sure you got the right thing.
#We will plot this using a bar plot from matplotlib.

#Bar locations
x = np.arange(len(mystorm)) #Generates a sequential range of numbers up to the length of mystorm.

#Let's label each bar with the median of the time increment that it corresponds to.
bar_labels = range(3, len(mystorm)*6-3, 6)#This command generates a sequence of numbers,
#starting at 3, advancing in increments of 6, all the way up to the length of 'mystorm'-3.
#Print it to verify!

#Set the bar widths
bar_width = 1 #This ensures that the bars are depicted side-by-side, without
#any space between them.

#Now let's generate the plot!
plt.bar(x, mystorm, width=bar_width, align='center')
plt.grid(axis='x') #Generates a grid parallel to the x-axis.
plt.xticks(x, bar_labels)
plt.ylabel('Rainfall amount, mm')
plt.xlabel('Hour')
plt.title('Hyetogram for Niger storm')


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
Now let's generate the histogram of the annual data.
Can you write Python code for it?

plt.hist(pAnn, normed=True)
plt.xlabel('Rainfall, mm')
plt.ylabel('Probability density')
plt.title('Annual rainfall')


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
Still looks pretty extreme!
Now let's aggregate this data into annual data. There is much less data (23 years), but it is still an interesting exercise. To do this, we will first "reshape" the daily array into a matrix in which the rows represent day of the year and the columns represent years. Thank god somebody has already taken out the "leap days" every 4 years, or this operation would be a lot trickier!
Once we reshape the array into a matrix, we will simply sum all of the rows in each column to get an annual total, which will leave us with an array that has the total rainfall for each year (1 x 23 in size).
Can you write Python code for it?

p_reshaped = np.reshape(pday, [365, 23])
pAnn = np.sum(p_reshaped,0) #The '0' indicates that we sum over the rows.
print(len(pAnn))
#What will be the output of this box if instead you sum over the columns (1)?


I am working on the file "PrecipFrequencyAnalyses.ipynb".
The first task is:
We can see that the pd.read_csv command reads in the maximum number of rows on the spreadsheet. Although the data in the column for daily precipitation do not extend for as many rows as the data in the column for 6-hourly precipitation, the length of both arrays ends up being the same, with the "extra" rows in the daily column being occupied by "nan" (Not A Number--coding parlance for blanks here).
So, let's get rid of those blanks.
Can you write Python code for it?

pday_no_nan = pday[~np.isnan(pday)]
print(len(pday_no_nan))


I am working on the file "Predict_Next_Review_Five_Star_round10.ipynb".
The first task is:
3.2 Join DataFrames to Populate the Features
After generatING the featureS within each dataframe, we join them together into one big dataframe.
Each join is a many-to-one join.
After the joins, each row represents one review, including features from the review itself, the user who’s written the review, and the business the review is for.
Can you write Python code for it?

# The `user_df` DataFrame is already indexed by the join key (`user_id`). Make sure it's on the right side of join.
review_join_user = review_df.join(user_df, on='user_id', lsuffix='_review', rsuffix='_user')

review_join_user_join_biz = review_join_user.join(biz_df, on='business_id', rsuffix='_biz')

drop_columns(df=review_join_user_join_biz, columns=['user_id', 'business_id'])


I am working on the file "Predict_Next_Review_Five_Star_round10.ipynb".
The first task is:
4.2 Split Training and Testing Set

Training set: used for an machine learning algorithm to train from.
Testing set: used to to estimate / evaluate how well the model has been trained.

Split them so that we don’t evaluate on the same dataset we train from
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

# Split the data into a training set and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y)

print 'training data shape', X_train.shape
print 'test data shape', X_test.shape
print 'converted label data shape', y_train.shape
print 'features', features