I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Betweenness Centrality
A path is a sequence of nodes between a star node and an end node where no node appears twice on the path, and is measured by the number of edges included (also called hops). The most interesting path to compute for two given nodes is the shortest path, e.g. the minimum number of edges required to reach another node, this is also called the node distance. Note that paths can be of length 0, the distance from a node to itself.
Can you write Python code for it?

# centrality = nx.betweenness_centrality(G)
# normalized = nx.betweenness_centrality(G, normalized=True)
# weighted   = nx.betweenness_centrality(G, weight="weight")

degrees = nbest_centrality(G, nx.betweenness_centrality, n=15)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Closeness Centrality
Another centrality measure, closeness takes a statistical look at the outgoing paths fora  particular node, v. That is, what is the average number of hops it takes to reach any other node in the network from v? This is simply computed as the reciprocal of the mean distance to all other nodes in the graph, which can be normalized to n-1 / size(G)-1 if all nodes in the graph are connected. The reciprocal ensures that nodes that are closer (e.g. fewer hops) score "better" e.g. closer to one as in other centrality scores.
Can you write Python code for it?

# centrality = nx.closeness_centrality(graph)
# normalied  = nx.closeness_centrality(graph, normalized=True)
# weighted   = nx.closeness_centrality(graph, distance="weight")

degrees = nbest_centrality(G, nx.closeness_centrality, n=15)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Clustering and Cohesion
In this next section, we're going to characterize our social network as a whole, rather than from the perspective of individual actors. This task is usually secondary to getting a feel for the most important nodes; but it is a chicken and an egg problem- determining the techniques to analyze and split the whole graph can be informed by key player analyses, and vice versa.
The density of a network is the ratio of the number of edges in the network to the total number of possible edges in the network. The possible number of edges for a graph of n vertices is n(n-1)/2 for an undirected graph (remove the division for a directed graph). Perfectly connected networks (every node shares an edge with every other node) have a density of 1, and are often called cliques.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Computing Key Players
In the previous graph, we began exploring ego networks and strong ties between individuals in our social network. We started to see that actors with strong ties to other actors created clusters that centered around themselves. This leads to the obvious question: who are the key figures in the graph, and what kind of pull do they have? We'll look at a couple measures of "centrality" to try to discover this: degree centrality, betweeness centrality, closeness centrality, and eigenvector centrality.
Degree Centrality
The most common and perhaps simplest technique for finding the key actors of a graph is to measure the degree of each vertex. Degree is a signal that determines how connected a node is, which could be a metaphor for influence or popularity. At the very least, the most connected nodes are the ones that spread information the fastest, or have the greatest effect on their community. Measures of degree tend to suffer from dillution, and benefit from statistical techniques to normalize data sets.
Can you write Python code for it?

def nbest_centrality(graph, metric, n=10, attribute="centrality", **kwargs):
    centrality = metric(graph, **kwargs)
    nx.set_node_attributes(graph, attribute, centrality)
    degrees = sorted(centrality.items(), key=itemgetter(1), reverse=True)

    for idx, item in enumerate(degrees[0:n]):
        item = (idx+1,) + item
        print "%i. %s: %0.4f" % item

    return degrees

degrees = nbest_centrality(G, nx.degree_centrality, n=15)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Eigenvector Centrality
The eigenvector centrality of a node, v is proportional to the sum of the centrality scores of it's neighbors. E.g. the more important people you are connected to, the more important you are. This centrality measure is very interesting, because an actor with a small number of hugely influential contacts may outrank ones with many more mediocre contacts. For our social network, hopefully it will allow us to get underneath the celebrity structure of heroic teams and see who actually is holding the social graph together.
Can you write Python code for it?

# centrality = nx.eigenvector_centality(graph)
# centrality = nx.eigenvector_centrality_numpy(graph)

degrees = nbest_centrality(G, nx.eigenvector_centrality_numpy, n=15)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
For testing and diagnostics it's useful to generate a random Graph. NetworkX comes with several graph models including:

Complete Graph G=nx.complete_graph(100)
Star Graph G=nx.star_graph(100)
Erdős-Rényi graph, binomial graph G=nx.erdos_renyi_graph(100, 0.20)
Watts-Strogatz small-world graph G=nx.watts_strogatz_graph(100, 0.20)
Holme and Kim power law G=nx.powerlaw_cluster_graph(100, 0.20)

But there are so many more, see Graph generators for more information on all the types of graph generators NetworkX provides. These, however are the best ones for doing research on social networks.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Graphs can also be analyzed in terms of distance (the shortest path between two nodes). The longest distance in a graph is called the diameter of the social graph, and represents the longest information flow along the graph. Typically less dense (sparse) social networks will have a larger diameter than more dense networks. Additionally, the average distance is an interesting metric as it can give you information about how close nodes are to each other.
Can you write Python code for it?

for subgraph in nx.connected_component_subgraphs(G):
    print nx.diameter(subgraph)
    print nx.average_shortest_path_length(subgraph)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Let's actually get into some clustering. The python-louvain library uses NetworkX to perform community detection with the louvain method. Here is a simple example of cluster partitioning on a small, built-in social network.
Can you write Python code for it?

partition = community.best_partition(G)
print "%i partitions" % len(set(partition.values()))
nx.set_node_attributes(G, 'partition', partition)

pos = nx.spring_layout(G)
plt.figure(figsize=(12,12))
plt.axis('off')

nx.draw_networkx_nodes(G, pos, node_size=200, cmap=plt.cm.RdYlBu, node_color=partition.values())
nx.draw_networkx_edges(G,pos, alpha=0.5)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
NetworkX has a ton of Graph serialization methods, and most have methods in the following format for serialization format, format:

Read Graph from disk: read_format
Write Graph to disk: write_format
Parse a Graph string: parse_format
Generate a random Graph in format: generate_format

The list of formats is pretty impressive:

Adjacency List
Multiline Adjacency List
Edge List
GEXF
GML
Pickle
GraphML
JSON
LEDA
YAML
SparseGraph6
Pajek
GIS Shapefile

The JSON and GraphmL are most noteworthy (for use in D3 and Gephi/Neo4j)
Initial Analysis of Email Network
We can do some initial analyses on our network using built in NetworkX methods.
Can you write Python code for it?

# Generate a list of connected components
# See also nx.strongly_connected_components
for component in nx.connected_components(G):
    print len(component)

len([c for c in nx.connected_components(G)])

# Get a list of the degree frequencies
dist = FreqDist(nx.degree(G).values())
dist.plot()

# Compute Power log sequence
degree_sequence=sorted(nx.degree(G).values(),reverse=True) # degree sequence

plt.loglog(degree_sequence,'b-',marker='.')
plt.title("Degree rank plot")
plt.ylabel("degree")
plt.xlabel("rank")

# Graph Properties
print "Order: %i" % G.number_of_nodes()
print "Size: %i" % G.number_of_edges()

print "Clustering: %0.5f" % nx.average_clustering(G)

print "Transitivity: %0.5f" % nx.transitivity(G)

hairball = nx.subgraph(G, [x for x in nx.connected_components(G)][0])
print "Average shortest path: %0.4f" % nx.average_shortest_path_length(hairball)

# Node Properties
node = 'benjamin@bengfort.com' # Change to an email in your graph
print "Degree of node: %i" % nx.degree(G, node)
print "Local clustering: %0.4f" % nx.clustering(G, node)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Serialization of Graphs
Most Graphs won't be constructed in memory, but rather saved to disk. Serialize and deserialize Graphs as follows:
Can you write Python code for it?

G = nx.read_graphml(GRAPHML) # opposite of nx.write_graphml

print nx.info(G)


I am working on the file ".ipynb_checkpoints/Introduction to Networkx-checkpoint.ipynb".
The first task is:
Visualizing Graphs
NetworkX wraps matplotlib or graphviz to draw simple graphs using the same charting library we saw in the previous chapter. This is effective for smaller size graphs, but with larger graphs memory can quickly be consumed.  To draw a graph, simply use the networkx.draw function, and then use pyplot.show to display it.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Basic Data Types: Strings, Lists, Tuples
Try the same thing on your own to learn about the following python objects: strings, lists, and tuples.
Can you write Python code for it?

some_string1 = 'apples'
some_string2 = 'and'
some_string3 = 'bananas'
print some_string1, some_string2, some_string3
print some_string1 + some_string2 + some_string3

mutable_list = ["apple", "apple", "banana", "kiwi", "bear", "strawberry", "strawberry"]
immutable_tuple = ("apple", "apple", "banana", "kiwi", "bear", "strawberry", "strawberry")

print len(some_string1)
print len(mutable_list)
print len(immutable_tuple)

print some_string1[0:5]
print mutable_list[0:4]
print immutable_tuple[5:6]

try:
    some_string1[5] = 'd'
except TypeError as e:
    print e

mutable_list[5] = 'mango'
try:
    immutable_tuple[5] = 'not going to work'
except TypeError as e:
    print e

a = [3 for i in range(10)]
print a


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Exercise: early data exploration and pandas syntax
Practice the following steps with these 4 data sets located at http://teachingdatascience.github.io/Rdatasets/datasets.html.
Copy the csv link and use that with read_csv to import it.
DATA SETS

chickwts
infert
msleep
Cars93

INSTRUCTIONS

import the data into python using pd.read_csv()
call the dtypes variable to identify the columns and their data types.
call the describe function to collect statistics about numerical data types.
for each other column (object columns):use
dataframe['column'].unique()  and
dataframe.groupby('column')['column'].count()

Write a short markdown that describes the data set and its columns.

Data Munging basics with pandas
What interesting data point do we learn when we run the following code?
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
In the Signed_In 0 group, we notice both Age and Gender are also always 0. We know this by:

In the Signed_In 0 group, there is only one Gender group as well in the DataFrame Index (0)
the min/max age for Signed_in group 0 is also 0.

This intuitively makes sense when working with marketing data: if a user is not logged in, you likely do not know their age or gender either.
Data does not always come in forms that we expect, so it is generally a requirement for us to work through a process called "data munging," which is the process of extracting and cleaning up a data set. Given that, we will explore the basics of data munging and aggregation:

Filter
Sort
Select and Unique
Mutate and Transform
Pivot and Aggregate

Much of this fits into the Split-Apply-Combine strategy of data analysis, popularized by Hadley Wickham's R package plyr (and later, dplyr). In fact, many of these concepts are shared between R and pandas, the primary difference being the syntax.
While we go through each of these examples, take good notes, comment through the code, and write questions that we can come back to.
Filter
Say we want to only focus on our marketing analysis on users we have data for--after all, it'd be easier to target users to convert on ads if we have a target demographic that performs well. Earlier we used a filtering command that finds rows that return True.
Can you write Python code for it?

nyt_signedin_only = nyt[nyt.Signed_In == 1]
nyt_signedin_only = nyt[nyt['Signed_In'] == 1]
nyt_signedin_only.groupby('Gender').describe()


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Libraries for Data Science¶
Data Scientists use a wide variety of libraries in Python that make working with data significantly easier. Those libraries primarily consist of:
1.numpy
2.scipy
3.pandas
4.matplotlib
5.statsmodels
6.scikit-learn
7.nltk

Though there are countless others available.
For today, we'll primarily focus ourselves around the library that is 99% of our work: pandas
pandas and object oriented programming¶
pandas is a library built on top of numpy, which allows us to use excel-like matrices in the python programming space. These special matrices are called DataFrames, the primary object in pandas.
Earlier we loaded up the csv file from our computer; pandas can also parse from a URL.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Like with everything else in python, the DataFrame is an object. We'll use a function called type to identify the object name:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Mutate, Transform (apply)
There are three most common techniques in mutating or transforming the data.
One approach would be to directly use values from other columns to derive new columns. The other would be to apply functions to columns using a named or lambda function.
You can generate the new columns by defining their name as a key.
Can you write Python code for it?

nyt['column_of_ones'] = 1

nyt['saw_ad_many_times'] = nyt['Impressions'].apply(lambda x: 1 if x > 5 else 0)

def saw_add_func(x):
    if x > 5:
        return 1
    else:
        return 0

nyt['saw_ad_many_times'] = nyt['Impressions'].apply(saw_add_func)

# practice on one column:
# a common approach to missing data, let's either set the values to null, or -1:
# import numpy as np
# np.nan

# by default apply works with a single column, however you can use it to iterate by row as well (though it is slower)
nyt['not_signed_in'] = nyt.apply(lambda row: 0 if row['Signed_In'] else 1, axis=1)


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Pivot and Aggregate
Earlier we saw use of the function groupby. A pandas groupby object has other functionality, such as returning statistics about a column, or all columns, or aggregating only what's defined.
Similarly, pivot tables can be used to explore groups side by side.
Can you write Python code for it?

# import numpy as np
nyt['column_of_ones'] = 1
nyt['Click_Thru'] = nyt['Clicks'] / nyt['Impressions']
nyt['Click_Thru'] = nyt['Click_Thru'].apply(lambda x: 0 if x == np.inf else x)
nyt_signedin_only = nyt[nyt.Signed_In == 1]
nyt_group = nyt.groupby('Age')

print nyt_group.mean()

print nyt_group['Click_Thru'].mean()

print nyt_group.agg({
    'Click_Thru': ['mean', 'max'],
    'Gender': 'mean',
})

print nyt[nyt.Signed_In == 1].pivot_table(
    values='column_of_ones',
    index='Gender',
    columns='Age',
    aggfunc='count'
)


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Practice filtering for the following:

rows where age is less than 30
rows where gender is 0 but signed_in is 1
rows where age is less than 30 or more than 60.

Selecting Rows, Columns, and Finding Uniques
Earlier we selected columns (or a list of columns) using key notation (like with dictionaries). Pandas also accepts dot notation (akin to an object mapper, or javascript object notation).
To select rows based on the index, we can use .ix[], which accepts either keys (when the index is a string) or a range (when the index is integer based). (pandas does allow more specific typing, loc and iloc)
Otherwise, passing a range in will do fine.
Can you write Python code for it?

# Finding the first ten rows
print nyt[0:10]

# This will still return the first ten rows
print nyt.set_index('Age')[0:10]

# Since the index is positional, this will also return the first 10 rows.
print nyt.ix[0:10]

# This first one will work fine, as it returns back all rows where Age == 40.
# The second will fail, as .ix does not treat the index as positional, when the index does not represent the row order.
print nyt.set_index('Age').ix[40]
try:
    print nyt.set_index('Age').ix[40:45]
except Exception as e:
    print e.message

# Using dot notation and returning uniques of that column
print nyt.Age.head()
print nyt.Age.unique()


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Practice:

Aggregate the mean and standard deviation (std) of Age when Signed_In equals 1
Find the min values of Age for each value of Impressions
Find the max values of Clicks for each value of Impressions
Determine: Are Signed In users more or less likely to click thru on an ad?

Plotting
matplotlib's core functionality serves as a plotting tool within python. While calling .describe() on DataFrames is useful to get a rough idea of what your data looks like, plots allow you to visualize what your data really looks like. For today, we'll use matplotlib within the context of pandas.
Consider the following data set and code:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
This translates to:

Found in the pandas library...
... in the core module...
... in the frame module...
...as a class, DataFrame

Objects in python are filled with variables and functions, and we use dot notation to access them.
# object.variable
nyt.dtypes
# object.function()
nyt.describe()

One great advantage with iPython is it has tab completion which means we can type nyt., press tab, and it'll show us what variables and functions exist.
Practice this with the following code cell with the following steps:

wrap the code with the type function: what data type does it return?
if it returns an "instancemethod" object, try the wrapper with a () to excute the function. What data type does the function return?
Can you write Python code for it?

# example:
print nyt.dtypes
nyt.dtypes?
print type(nyt.dtypes)

# Try wrapping type with the following:
print nyt.describe
print nyt.describe()
print nyt.shape
print nyt.index
print nyt.columns
print nyt.groupby
print nyt.groupby('Age')
print nyt.Age


I am working on the file ".ipynb_checkpoints/Introduction to Python for Data Analysis-checkpoint.ipynb".
The first task is:
Use these new tools in order to visualize some of this New York Times ad performance data.
Can you write Python code for it?

nyt[nyt.Signed_In == 1].Age.hist()

print nyt[nyt.Signed_In == 1].Clicks.hist()

nyt.describe()

nyt['Click_Thru'] = nyt['Clicks'] / nyt['Impressions']
nyt[nyt.Signed_In == 1].plot(kind='scatter', x='Age', y='Click_Thru')


I am working on the file ".ipynb_checkpoints/JsItBad-checkpoint.ipynb".
The first task is:
Features
Can you write Python code for it?

def length(code):
    return len(code)

def nb_lines(code):
    return len(code.split('\n'))

def avg_char_per_line(code):
    return length(code)/nb_lines(code)

def nb_strings(code):
    '''Ugly approximation, no simple way out of this short of actually parsing the JS.'''
    return len(code.split("'"))+len(code.split('"'))

def nb_non_printable(code):
    '''\cite{likarish2009obfuscated} use unicode symbol, but we are more general'''
    return len([x for x in code if not x in string.printable])

hex_octal_re = re.compile('([^A-F0-9]0[0-7]+|0x[A-F0-9]+)')
def hex_or_octal(code):
    '''Ugly as hell, but we dont want to parse'''
    return len(list(hex_octal_re.finditer(code)))

def max_nesting_level(code):
    l = 0
    max_l = 0
    for c in code:
        if c in '({[':
            l+=1
            max_l = l if l > max_l else max_l
        elif c in ')}]':
            l-=1
    return max_l

features = [length, nb_lines, avg_char_per_line, nb_strings, nb_non_printable, hex_or_octal, max_nesting_level]

X = np.array([[f(x['code']) for f in features] for x in data])
X[:30]

#http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html#example-manifold-plot-compare-methods-py
from sklearn import manifold
%matplotlib inline
import matplotlib.pylab as plt

n_neighbors = 10
n_components = 2
#Y = manifold.Isomap(n_neighbors, n_components).fit_transform(X)
#Y = manifold.LocallyLinearEmbedding(n_neighbors, n_components,
#                                        eigen_solver='auto').fit_transform(X)
Y = manifold.MDS(n_components, max_iter=100, n_init=1).fit_transform(X)
#Y = manifold.SpectralEmbedding(n_components=n_components,
#                                n_neighbors=n_neighbors).fit_transform(X)
#Y = manifold.TSNE(n_components=n_components, init='pca', random_state=0).fit_transform(X)
plt.scatter(Y[:, 0], Y[:, 1], c=[x['color'] for x in data], alpha=0.2)
for label, x, y in zip([x['name'] for x in data], Y[:, 0], Y[:, 1]):
    if '.js' in label and not ('min.' in lab2el):
       ...


I am working on the file ".ipynb_checkpoints/JudicialRulings-checkpoint.ipynb".
The first task is:
Next block caches metadata for retrieving Supreme Court transcripts into a csv
Can you write Python code for it?

import csv
with open("judicialMetadata.csv", "w+") as metadata:
    header = allRecentRecords[0].keys()
    writer = csv.DictWriter(metadata, fieldnames=header)
    writer.writerows(allRecentRecords)


I am working on the file ".ipynb_checkpoints/JudicialRulings-checkpoint.ipynb".
The first task is:
Should put a csv reading block here for faster processing
Can you write Python code for it?

import pdfquery
import requests

def getPDFTree(url, tempURL):
    pdf = requests.get(url)
    with open(tempURL, "wb+", buffering=0) as fp:
        fp.write(pdf.content)
        pdfTree = pdfquery.PDFQuery(fp)
        pdfTree.load()
        return pdfTree

url = "https://www.supremecourt.gov/oral_arguments/argument_transcripts/2000/00-6374.pdf"
tempURL = "temp/argument_transcripts/2004/04-603.x,l"
pdfTree = getPDFTree(url, tempURL)

pdfTree.tree.write("temp/argument_transcripts/2006/05-85.xml", pretty_print=True, encoding="utf-8")

TOContents = pdfTree.pq('LTTextLineHorizontal:contains("C O N T E N T S ")')
assert TOContents != None, "Table of contents is formatted differently"

import re
with open("temp/argument_transcripts/2004/04-603.xml") as fp:
    text = fp.read()
    tagRegex = r"\</?[^\<\>/]*\>"
    text = re.sub(tagRegex, "", text)
    text = re.sub(r'\n\s*[0-9]+', "\n", text)
    text = text.replace('FOURTEENTH STREET, N.W.WASHINGTON, D.C. 20005(800) FOR DEPO  ALDERSON REPORTING COMPANY, INC.  1111 FOURTEENTH STREET, N.W.  SUITE 400  WASHINGTON, D.C. 20005  (202)289-2260  (800) FOR DEPO', '')
    tocLength = text.find("C O N T E N T S ")
    print(text[tocLength  : ])


I am working on the file ".ipynb_checkpoints/Jupyter_notebook_introduction-checkpoint.ipynb".
The first task is:
Display of graphics
When displaying graphics, you should have them inline, meaning that they are displayed directly in the Jupyter notebook and not in a separate window.  You can specify that, as in the cell below, using the %matplotlib inline magic function.
Generally, I prefer presenting graphics as scalable vector graphics (SVG).  Vector graphics are infinitely zoom-able; i.e., the graphics are represented as points, lines, curves, etc., in space, not as a set of pixel values as is the case with raster graphics (such as PNG).  By default, graphics are displayed as PNGs, but you can specify SVG as I have at the top of this document in the first code cell.
%config InlineBackend.figure_formats = {'svg',}

If SVG graphics aren't working in your browser PNG graphics at a high resolution can be used instead
%config InlineBackend.figure_formats = {'png', 'retina'}
Can you write Python code for it?

%matplotlib inline

%config InlineBackend.figure_formats = {'svg',}
#%config InlineBackend.figure_formats = {'png', 'retina'}


I am working on the file ".ipynb_checkpoints/Jupyter_notebook_introduction-checkpoint.ipynb".
The first task is:
Example plot
We can generate some data to plot using the np.linspace function and then feeding that data into the np.exp function.
Can you write Python code for it?

x = np.linspace(0, 2 * np.pi, 200)
y = np.exp(np.sin(np.sin(x)))


I am working on the file ".ipynb_checkpoints/Jupyter_notebook_introduction-checkpoint.ipynb".
The first task is:
Import packages for scientific computing
One of the things that makes Python so powerful for science are the plethora of packages for scientific computing. However, the need to import these packages and understand what they are is also confusing to new users.
Usually at the top of a notebook, you should put in a code cell that imports all the modules we'll need. For this notebook, we will import numpy so that we can use its numerical structures, the scipy.integrate module and matplotlib.pyplot for plotting.
Can you write Python code for it?

import numpy as np
import scipy.integrate
import matplotlib.pyplot as plt


I am working on the file ".ipynb_checkpoints/Jupyter_notebook_introduction-checkpoint.ipynb".
The first task is:
Note, however, if the last line does not return a value, such as if we assigned a variable, there is no visible output from the code cell.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Jupyter_notebook_introduction-checkpoint.ipynb".
The first task is:
These data can then be plotted as below with axes then being labeled.
Can you write Python code for it?

plt.plot(x, y)
plt.xlim((0, 2 * np.pi))
plt.xlabel('$x$')
plt.ylabel('$\mathrm{e}^{\sin{x}}$')
plt.title('Example plot')
plt.show()


I am working on the file ".ipynb_checkpoints/Jupyter_notebook_introduction-checkpoint.ipynb".
The first task is:
With this function in hand, we just have to pick our initial conditions and time points, run the numerical integration, and then plot the result.
Can you write Python code for it?

# Parameters to use
p = np.array([10.0, 28.0, 8.0 / 3.0])

# Initial condition
r0 = np.array([0.1, 0.0, 0.0])

# Time points to sample
t = np.linspace(0.0, 80.0, 10000)

# Use scipy.integrate.odeint to integrate Lorentz attractor
r = scipy.integrate.odeint(lorenz_attractor, r0, t, args=(p,))

# Unpack results into x, y, z.
x, y, z = r.transpose()

# Plot the result
plt.plot(x, z, '-', linewidth=0.5)
plt.xlabel(r'$x(t)$', fontsize=18)
plt.ylabel(r'$z(t)$', fontsize=18)
plt.title(r'$x$-$z$ proj. of Lorenz attractor traj.')
plt.show()


I am working on the file ".ipynb_checkpoints/KL rambling notes on Python-checkpoint.ipynb".
The first task is:
/...this is where I learned to not use pip install with scikit-learn...
To upgrade scikit-learn:
conda update scikit-learn
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/KL rambling notes on Python-checkpoint.ipynb".
The first task is:
OK...can I get a simple scatter plot?
Can you write Python code for it?

plt.scatter(range(0,len(silAverage)), silAverage)
plt.grid() #put on a grid

plt.xlim(-1,20)

#get list of column names in pandas data frame
list(my_dataframe.columns.values)

for i in range(0,len(ut)):
    if i == 10:
        break
    p = ut.iloc[i,:]
    n = p.name
    if n[0] == 'R':
        #do the plotting,
        #print 'yes'
        CO = p.KEGG
        kos = CO_withKO[CO]['Related KO']
        cos = CO_withKO[CO]['Related CO']
        #Tracer()()
        for k in kos:
            if k in KO_RawData.index:
                kData=KO_RawData.loc[kos].dropna()
                kData=(kData.T/kData.sum(axis=1)).T
                #? why RawData, the output from the K-means will have the normalized data, use that for CO
                #bc easier since that is the file I am working with right now.
                #cData=CO_RawData.loc[cos].dropna()
                #cData=(cData.T/cData.sum(axis=1)).T
                cData = pd.DataFrame(p[dayList]).T

                #go back and check, but I think this next step is already done
                #cData=(cData.T/cData.sum(axis=1)).T

                fig, ax=plt.subplots(1)
                kData.T.plot(color='r', ax=ax)
                cData.T.plot(color='k', ax=ax)

    else:
        #skip over the KO plotting, so effectively doing nothing
        #print 'no'


I am working on the file ".ipynb_checkpoints/Kitchen Sink-Bass-checkpoint.ipynb".
The first task is:
Need help?
Try checking the docstring of a function you are struggling with.
moving_statistics?
help(moving_statistics)
Can you write Python code for it?

moving_statistics?

import pyeeg

pyeeg.samp_entropy?


I am working on the file ".ipynb_checkpoints/Kitchen Sink-Bass-checkpoint.ipynb".
The first task is:
WARNING All strings should be raw, especially if in Windows.
r'String!'
Can you write Python code for it?

#initialize new file
Data = {}
Settings = {}
Results ={}

############################################################################################
#manual Setting block
Settings['folder']= r"/Users/abigaildobyns/Desktop"
Settings['Label'] = r'rat34_ECG.txt'
Settings['Output Folder'] = r"/Users/abigaildobyns/Desktop/demo"

#transformation Settings
Settings['Absolute Value'] = True #Must be True if Savitzky-Golay is being used

Settings['Bandpass Highcut'] = r'none' #in Hz
Settings['Bandpass Lowcut'] = r'none' #in Hz
Settings['Bandpass Polynomial'] = r'none' #integer

Settings['Linear Fit'] = False #between 0 and 1 on the whole time series
Settings['Linear Fit-Rolling R'] = 0.75 #between 0 and 1
Settings['Linear Fit-Rolling Window'] = 1000 #window for rolling mean for fit, unit is index not time
Settings['Relative Baseline'] = 0 #default 0, unless data is normalized, then 1.0. Can be any float

Settings['Savitzky-Golay Polynomial'] = 4 #integer
Settings['Savitzky-Golay Window Size'] = 251 #must be odd. units are index not time

#Baseline Settings
Settings['Baseline Type'] = r'static' #'linear', 'rolling', or 'static'
#For Linear
Settings['Baseline Start'] = 0.0 #start time in seconds
Settings['Baseline Stop'] = 1.0 #end time in seconds
#For Rolling
Settings['Rolling Baseline Window'] = r'none' #leave as 'none' if linear or static

#Peaks
Settings['Delta'] = 0.25
Settings['Peak Minimum'] = -1 #amplitude value
Settings['Peak Maximum'] = 1 #amplitude value

#Bursts
Settings['Burst Area'] = False #calculate burst area
Settings['Exclude Edges'] = True #False to keep edges, True to discard them
Settings['Inter-event interval minimum (seconds)'] = 0.0100 #only for bursts, not for peaks
Settings['Maximum Burst Duration (s)'] = 10
Settings['Minimum Burst Duration (s)'] = 0
Settings['Minimum Peak Number'] = 1 #minimum number of peaks/burst, integer
Settings['Threshold']= 0.15 #linear: proportion of baseline.
                           #static: literal value.
                           #rolling, linear ammount...


I am working on the file ".ipynb_checkpoints/Laav_Lane_Lines_Detection-checkpoint.ipynb".
The first task is:
Build a Lane Finding Pipeline
Build the pipeline and run your solution on all test_images. Make copies into the test_images_output directory, and you can use the images in your writeup report.
Try tuning the various parameters, especially the low and high Canny thresholds as well as the Hough lines parameters.
Can you write Python code for it?

# TODO: Build your pipeline that will draw lane lines on the test_images
# then save them to the test_images directory.

for im in os.listdir("test_images/"):

    if im=='output':
        continue

    image = mpimg.imread('test_images/' + im)

    #Copy the image to prevent unecessary changes
    image_cp = np.copy(image)

    # Convert image to grey scale
    gray = grayscale(image_cp)

    # Define a kernel size and apply Gaussian smoothing
    kernel_size = 5
    blur_gray = gaussian_blur(gray, kernel_size)

    # Define our parameters for Canny and apply
    low_threshold = 50
    high_threshold = 150
    edges = canny(blur_gray, low_threshold, high_threshold)

    # Define a polygon region of interest
    # Keep in mind the origin (x=0, y=0) is in the upper left in image processing

    # Coordinates of vertices
    left_upper = [int(xsize/2)-50, lane_limit_height]
    right_upper = [int(xsize/2)+50, lane_limit_height]
    right_bottom = [xsize-50, ysize]
    left_bottom = [50, ysize]

    vertices = np.array( [[ left_upper,right_upper,right_bottom,left_bottom]], dtype=np.int32 )

    region_canny_image = region_of_interest(edges, vertices)

    # ````````````````` Test region of interest```````````````````````````````
    # mpimg.imsave('test_images/output/' + 'test_region_area.png', region_image)
    # plt.imshow(region_image)

    # Define the Hough transform parameters
    # Make a blank the same size as our image to draw on
    rho = 2
    theta = np.pi/180
    threshold = 32
    min_line_len = 10
    max_line_gap = 200
    # As you increase above longer the line connecting the left lane lines

    # Run Hough on edge detected image
    line_image = hough_lines(region_canny_image, rho, theta, threshold, min_line_len, max_line_gap)
    # plt.imshow(line_image)

    # Combine the processed image after selecting the required region with the original img
    comb_hough_orig = weighted_img(line_image, image)
    # plt.imshow(comb_hough_orig)

    #saving the resultant image
  ...


I am working on the file ".ipynb_checkpoints/Laav_Lane_Lines_Detection-checkpoint.ipynb".
The first task is:
Ideas for Lane Detection Pipeline
Some OpenCV functions (beyond those introduced in the lesson) that might be useful for this project are:
cv2.inRange() for color selection
cv2.fillPoly() for regions selection
cv2.line() to draw lines on an image given endpoints
cv2.addWeighted() to coadd / overlay two images
cv2.cvtColor() to grayscale or change color
cv2.imwrite() to output images to file
cv2.bitwise_and() to apply a mask to an image
Check out the OpenCV documentation to learn about these and discover even more awesome functionality!
Helper Functions
Below are some helper functions to help get you started. They should look familiar from the lesson!
Can you write Python code for it?

import math

# im,port sys for computing max, min int
import sys

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    (assuming your grayscaled image is called 'gray')
    you should call plt.imshow(gray, cmap='gray')"""
    return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    # Or use BGR2GRAY if you read an image with cv2.imread()
    # return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

def canny(img, low_threshold, high_threshold):
    """Applies the Canny transform"""
    return cv2.Canny(img, low_threshold, high_threshold)

def gaussian_blur(img, kernel_size):
    """Applies a Gaussian Noise kernel"""
    return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)

def region_of_interest(img, vertices):
    """
    Applies an image mask.

    Only keeps the region of the image defined by the polygon
    formed from `vertices`. The rest of the image is set to black.
    """
    #defining a blank mask to start with
    mask = np.zeros_like(img)

    #defining a 3 channel or 1 channel color to fill the mask with depending on the input image
    if len(img.shape) > 2:
        channel_count = img.shape[2]  # i.e. 3 or 4 depending on your image
        ignore_mask_color = (255,) * channel_count
    else:
        ignore_mask_color = 255

    #filling pixels inside the polygon defined by "vertices" with the fill color
    cv2.fillPoly(mask, vertices, ignore_mask_color)

    #returning the image only where mask pixels are nonzero
    masked_image = cv2.bitwise_and(img, mask)
    return masked_image

def draw_lines(img, lines, color=[255, 0, 0], thickness=2):
    """
    NOTE: this is the function you might want to use as a starting point once you want to
    average/extrapolate the line segments you detect to map out the full
    extent of the lane (going from the result shown in raw-lines-example.mp4
    to that shown in P1_example.mp4).

    Think about things...


I am working on the file ".ipynb_checkpoints/Laav_Lane_Lines_Detection-checkpoint.ipynb".
The first task is:
Improve the draw_lines() function
At this point, if you were successful with making the pipeline and tuning parameters, you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform. As mentioned previously, try to average and/or extrapolate the line segments you've detected to map out the full extent of the lane lines. You can see an example of the result you're going for in the video "P1_example.mp4".
Go back and modify your draw_lines function accordingly and try re-running your pipeline. The new output should draw a single, solid line over the left lane line and a single, solid line over the right lane line. The lines should start from the bottom of the image and extend out to the top of the region of interest.
Now for the one with the solid yellow lane on the left. This one's more tricky!
Can you write Python code for it?

yellow_output = 'test_videos_output/solidYellowLeft.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4').subclip(0,5)
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))


I am working on the file ".ipynb_checkpoints/Laav_Lane_Lines_Detection-checkpoint.ipynb".
The first task is:
Let's try the one with the solid white lane on the right first ...
Can you write Python code for it?

white_output = 'test_videos_output/solidWhiteRight.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4").subclip(0,5)
clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)


I am working on the file ".ipynb_checkpoints/Laav_Lane_Lines_Detection-checkpoint.ipynb".
The first task is:
Test Images
Build your pipeline to work on the images in the directory "test_images"
You should make sure your pipeline works well on these images before you try the videos.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Laav_Lane_Lines_Detection-checkpoint.ipynb".
The first task is:
Writeup and Submission
If you're satisfied with your video outputs, it's time to make the report writeup in a pdf or markdown file. Once you have this Ipython notebook ready along with the writeup, it's time to submit for review! Here is a link to the writeup template file.
Optional Challenge
Try your lane finding pipeline on the video below.  Does it still work?  Can you figure out a way to make it more robust?  If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!
Can you write Python code for it?

challenge_output = 'test_videos_output/challenge.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip3 = VideoFileClip('test_videos/challenge.mp4').subclip(0,5)
clip3 = VideoFileClip('test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(process_image)
%time challenge_clip.write_videofile(challenge_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))


I am working on the file ".ipynb_checkpoints/Lab 3 - Cenni di Programmazione Funzionale - Parte seconda-checkpoint.ipynb".
The first task is:
ESERCIZIO 1: Scrivere un'espressione che calcoli il quadrato delle differenze elemento per elemento di due "vettori" dati, ovvero:
$$\sum_{i = 1,..,n} (x_i-y_i)^2, \quad x,y \in \mathbb{R}^n$$
Usare nell'espressione le due liste $[1,2,3,4,5,6,7,8,9]$  e $[9,8,7,6,5,4,3,2,1]$.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lab 3 - Cenni di Programmazione Funzionale - Parte seconda-checkpoint.ipynb".
The first task is:
ESERCIZIO 2: Calcolare il quadrato dei numeri dispari di una lista data. Suggerimento: utilizzare sia map che filter.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lab 3 - Cenni di Programmazione Funzionale - Parte seconda-checkpoint.ipynb".
The first task is:
ESERCIZIO 3: Scrivere una funzione che calcoli la norma di un vettore $\sqrt{\sum_{i=1,..,n} x_i^2}$. Suggerimento: la funzione sqrt è una funzione della libreria math e deve essere importata con ìl comando from math import sqrt.
Can you write Python code for it?

from math import sqrt

# SVOLGERE ESERCIZIO


I am working on the file ".ipynb_checkpoints/Lab 3 - Cenni di Programmazione Funzionale - Parte seconda-checkpoint.ipynb".
The first task is:
List comprehensions
Usare la notazione di list comprehensions può rendere talvolta il codice più compatto e può spostare il focus di chi lo legge dal come si stan facendo i conti al cosa si sta calcolando. Si consideri l'esempio seguente:
collection = list()
for datum in data_set:
    if condition(datum):
        collection.append(datum)
    else:
        new = modify(datum)
        collection.append(new)
che usando la notazione di list comprehensions si può riscrivere come:
collection = [d if condition(d) else modify(d) for d in data_set]
Funzione reduce
La funzione reduce letteralmente "riduce" una sequenza di elementi ad uno scalare. In programmazione funzionale viene chiamata anche fold. La funzione reduce prende in input una funzione, chiamata "combinante", una sequenza iterabile di elementi e un valore iniziale (facoltativo). In output viene restituito un valore che risulta dall'applicare in sequenza agli elementi della lista la funzione data:

`reduce(function, sequence[, initial]) -> value`

Per esempio, se viene passata in input la funzione f(x,y), la lista [1,2,3,4] e il valore iniziale 0, la funzione reduce calcola il valore:
$$v = f(f(f(f(0,1), 2), 3), 4)$$
Se la funzione data è la somma, questo risulta equivalente a calcolare:
$$((((0+1)+2)+3)+4) = 10$$
La funzione reduce non è una builtin di Python e deve essere importata con il comando:
from functools import reduce
ESEMPIO 4: Scrivere il codice per l'esempio precedente
Can you write Python code for it?

from functools import reduce

reduce(lambda x,y: x+y, [1,2,3,4])


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
Create an English stemmer
http://www.nltk.org/howto/stem.html for general intro.
http://www.nltk.org/api/nltk.stem.html for more details (including languages covered).
Can you write Python code for it?

s = nltk.stem.SnowballStemmer('english')
s.stem("cats"), s.stem("ran"), s.stem("jumped")


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
Creating a CountVectorizer object
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
train_counts = vectorizer.fit_transform(twenty_train.data)


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
If we modify build_analyzer() to apply the NLTK stemmer to the output of default build_analyzer(), we get a version that does stemming as well:
Can you write Python code for it?

import nltk.stem
english_stemmer = nltk.stem.SnowballStemmer('english')
class StemmedCountVectorizer(CountVectorizer):
    def build_analyzer(self):
        analyzer=super(StemmedCountVectorizer, self).build_analyzer()
        return lambda doc:(english_stemmer.stem(w) for w in analyzer(doc))


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
Integrating NLTK with SciKit's vectorizer
NLTK Stemmer
The stemmer can be used to stem documents before feeding into SciKit's vectorizer, thus obtaining a more compact index.
One way to do this is to define a new class StemmedCountVectorizer extending CountVectorizer by redifining the method build_analyzer() that handles preprocessing and tokenization.
http://scikitlearn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
build_analyzer() takes a string as input and outputs a list of tokens.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words="english")
analyze = vectorizer.build_analyzer()
analyze("John bought carrots and potatoes")


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
More preprocessing
For stemming and more advanced preprocessing, supplement SciKit Learn with another Python library, NLTK. Up next.
More advanced preprocessing with NLTK
NLTK is described in detail in a book by Bird, Klein and Loper available online:
http://www.nltk.org/book_1ed/ for version 2.7 of python
About NLTK

It is not the best
It is very easy to use

You should read the book linked above to get familiar with the package and with text preprocessing.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
NLTK for text analytics

NERs
Sentiment analysis
Extracting information from social media.
Can you write Python code for it?

from nltk.tokenize import word_tokenize
text = word_tokenize("And now for something completely different")

nltk.pos_tag(text)


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
Number of times word "hard" occurs
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
So now we can create an instance of this class:
Can you write Python code for it?

stem_vectorizer = StemmedCountVectorizer(min_df=1,
                                        stop_words='english')
stem_analyze = stem_vectorizer.build_analyzer()
Y = stem_analyze("John bought carrots and potatoes")

[tok for tok in Y]


I am working on the file ".ipynb_checkpoints/Lab1 Text processing with python-checkpoint.ipynb".
The first task is:
We can now see how frequently the word algorithm occurs in the subset of the 20Newgroups collection we are considering.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/LeagueRank_notebook-checkpoint.ipynb".
The first task is:
2. Fetch most frequently played champions
Since we already had a dictionary of all user ids mapping to all categories of ranks, we can now use those user ids to get the stats data of their most frequently used champions. We will use the raw RESTful APIs of Riot with python here. And here are all the libraries needed in this process.
Can you write Python code for it?

import csv
import json
import os
import urllib2


I am working on the file ".ipynb_checkpoints/LeagueRank_notebook-checkpoint.ipynb".
The first task is:
To use the Riot api, one more important thing to do is to get your own API key. API key can be obtained from here. Note that normal developr API key has a narrow request limit, whereas production API key for commercial use has a looser requirement of request limit. For now, we are just gonna use the normal API key for demonstration.
After getting your own API key, put it in the config dictionary below:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/LeagueRank_notebook-checkpoint.ipynb".
The first task is:
With the above class, now we can start crawling the stats data of all champions saving them to csv files by the following code. Notice that this process is pretty slow since we added the sleep methods in our code. Riot APIs have a limitation on the API calls rate. You cannot send more than 500 requests per 10 minutes. So everytime we send a request here, we sleep for 1 second to prevent error responses.
Can you write Python code for it?

def main():
    import time
    tiers = get_tier()
    for tier, rank_dict in tiers.iteritems():
        print 'starting tier: {}'.format(tier)
        for summoner_id in rank_dict:
            print 'tier: {}, summoner id: {}'.format(tier, summoner_id)
            top_champion = TopChampion(config['key'], summoner_id, tier, 3)
            top_champion.get_top_champions()
            top_champion.save_top_champions()
            time.sleep(1)
        print 'end tier: {}'.format(tier)


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
3. Variable Resolution

Python first looks at the function's namespace first before looking at the global namespace.
Can you write Python code for it?

aString = 'Global var'
def foo():
    print aString

foo()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
4. Function Arguments: args and kwargs

Python allows us to pass function arguments (duh..)
There arguments are local to the function. For instance:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
7. Functions and Class are Objects

Functions and objects are like anything else in python.
All objects inherit from a base class in python.
For instance,
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
8. Closures
Remember this example?
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
9. Decorators!

Decorators are callables that take a function as argument, and return a replacement function (with additional functionalities)
Can you write Python code for it?

def outer(func):
    def inner(*args):
        "Inner"
        print 'Decorating...'
        ret = func()
        ret += 1
        return ret
    return inner

def foo():
    "I'm foo"
    return 1

print foo()

decorated_foo = outer(foo)

print decorated_foo()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
9.1 Logging and timing a function

Decorators can be classes, they can take input arguments/keyword args.

Lets build a decorator that logs and times another function
Can you write Python code for it?

import time
from pdb import set_trace
def logger(func):
    def inner(*args, **kwargs):
        print "Arguments were: %s, %s"%(args, kwargs)
        return func(*args, **kwargs)
    return inner

def timer(func):
    def inner(*args, **kwargs):
        t=time.clock()
        set_trace()
        func(*args, **kwargs)
        print "Time taken: %0.2f sec"%(time.clock()-t)
        return func(*args, **kwargs)
    return inner

@logger
@timer
def foo(a=5, b=2):
    time.sleep(2)
    return a+b

@logger
@timer
def bar(a=10, b=1):
    time.sleep(1)
    return a-b

foo(2,3)
bar(5,7)


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
A common practice is to replace the original function with the decorated function
Can you write Python code for it?

def outer(func):
    def inner():
        "Inner"
        print 'Decorating...'
        ret = func()
        ret += 1
        return ret
    return inner

def foo():
    "I'm foo"
    return 1

print foo()
foo = outer(foo)

print foo()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
All arg and kwarg conventions apply here

6.1 Overriding class methods
Lets try:
Can you write Python code for it?

class foo():
    def __init__(i, num):
        i.num = num

d = foo(2)
d()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Arguments in functions can be classified as:
Args
kwargs (keyword args)

When calling a function, args are mandatory. kwargs are optional.
Can you write Python code for it?

"Args"
def foo(x,y):
    print x+y

"kwargs"
def bar(x=5, y=8):
    print x-y

"Both"
def foobar(x,y=100):
    print x*y

"Calling with args"
foo(5,12)

"Calling with kwargs"
bar()

"Calling both"
foobar(10)


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Aside: __repr__, __call__,__getitem__,...  are all awesome.
Can you write Python code for it?

class foo():
    "Me is foo"
    def __init__(i, num):
        i.num = num
    def __add__(i, new):
        i.num += new
        return i
    def __sub__(i, new):
        i.num -= new
        return i
    def __repr__(i):
        return i.__doc__
    def __getitem__(i, num):
        print "Nothing @ %d"%(num)

FOO = foo(4)
FOO[2]


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Declare global every time the global x needs changing

6. Classes

Define classes with the class keyword
Here's a simple class
Can you write Python code for it?

class foo():
    def __init__(i, arg1): # self can br replaced by anything.
        i.arg1 = arg1
    def bar(i, arg2): # Always use self as the first argument
        print i.arg1, arg2

FOO = foo(7)

FOO.bar(5)

print FOO.arg1


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
It follows that the variable a here is a class.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Let's rewrite this:
Can you write Python code for it?

class foo():
    def __init__(i, num):
        i.num = num
    def __add__(i, new):
        i.num += new
        return i
    def __sub__(i, new):
        i.num -= new
        return i

FOO = foo(5)
FOO += 1
print FOO.num
FOO -= 4
print FOO.num


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Lets look at memory locations of the functions.
Can you write Python code for it?

def outer(func):
    def inner(*args):
        "Inner"
        print 'Decorating...'
        ret = func()
        ret += 1
        return ret
    print inner.__doc__, inner
    return inner

def foo():
    "I'm foo"
    return 1

print foo.__name__, foo

decorated_foo = outer(foo)

print decorated_foo.__name__, decorated_foo


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Nested lambda is permitted (idk why you'd use them, still, worth a mention)
Can you write Python code for it?

foo = lambda x: lambda y: x+y
print foo(3)(5)


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Never call args after kwargs

5. Nesting functions

You can nest functions.
Class nesting is somewhat uncommon, but can be done.
Can you write Python code for it?

def outer():
    x=1
    def inner():
        print x
    inner()

outer()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Obviously, this fails. Why? As per variable lifetime rules (see 2.1), foo() has ceased execution, x is destroyed.
So how about this?
Can you write Python code for it?

def foo():
    x='Outer String'
    def bar():
        print x
    return bar

test = foo()

test()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Python uses @ to represent foo = outer(foo). The above code can be retwritten as follows:
Can you write Python code for it?

def outer(func):
    def inner():
        "Inner"
        print 'Decorating...'
        ret = func()
        ret += 1
        return ret
    return inner

@outer
def foo():
    "I'm foo"
    return 1

print foo()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Q. What happens if there is no return?
2. Scope

In python functions have their own scope (namespace).
Python first looks at the function's namespace first before looking at the global namespace.
Let's use locals() and globals() to see what happens:
Can you write Python code for it?

aString = 'Global var'
def foo():
    a = 'Local var'
    print locals()

foo()
print globals()


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
Q. will these two work?
Can you write Python code for it?

"Args"
def foo(x,y):
    print x+y

"kwargs"
def bar(x=5, y=8):
    print x-y

"Both"
def foobar(x,y=100):
    print x*y

bar(x=9, 7) #1
foo(x=5, 6) #2


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
There are many such redefinitions permitted by python. See Python Docs

6.2 Emulating numeric types

A very useful feature in python is the ability to emulate numeric types.

Would this work?
Can you write Python code for it?

class foo():
    def __init__(i, num):
        i.num = num
FOO = foo(5)
FOO += 1


I am working on the file ".ipynb_checkpoints/Lecture4-checkpoint.ipynb".
The first task is:
This means:
Functions and Classes can be passed as arguments.
Functions can return other functions/classes.
Can you write Python code for it?

from pdb import set_trace # pdb is quite useful

def add(x,y): return x+y
def sub(x,y): return x-y
def foo(x,y,func=add):
    set_trace()
    return func(x,y)

foo(7,4,sub)


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
3. モデリング対象の設定
対象の明確化
このコンペティションの最終的なゴールは、各ユーザが将来どのクーポンを購入するかをより正確に予測することです。
モデリング(≒予測モデルの構築)は
- ユーザの属性データ(性別、年代、地域など)
- クーポンの属性データ(ジャンル、価格、地域など)
- いつどのユーザがどのクーポンを見たか、購買したか等のログデータ
の3つのデータを主に利用します（これは、クーポンに限らず他の多くの問題で共通しています)。
これらのデータを利用して将来ユーザがどのクーポンを購買するのかをモデリングするわけですが、
特に、このコンペティションでは、モデリングの良し悪しを評価する指標が与えられており、
参加者はこの指標のスコアを競っている形になっています。
https://www.kaggle.com/c/coupon-purchase-prediction/details/evaluation によれば、
このコンペティションでは、Mean Average Precision@10(MAP@10)という指標を最大化するような予測モデルを構築することが最終的なゴールです。
Map@10は、下記の数式で定義されています。
$$
MAP@10=\frac{1}{|U|}\sum_{u=1}^{|U|}\frac{1}{min(m, 10)} \sum_{k=1}^{min(n, 10)}P(k)
$$
$|U|$：ユーザ数
$P(k)$：k番目まで着目した時のPrecision(購買すると予測したクーポン中の実際に購買されたクーポンの割合)
$n$：購買すると予測されたクーポンの数
$m$：実際に購買されたクーポンの数(m=0の時は、Precision=0)
この指標を直接最大化するようにモデリングする方がコンペティションの最終的な評価は高いと考えられますが、
ここでは、簡単のため、下記のように問題を少し変更し別の指標で代替することにします。
- 全てのユーザは検証用の期間で必ずクーポンをちょうど1つ購入する。
- その1つのクーポンを訓練用の期間のデータから予測する。
このように問題を簡単にし、検証用の期間で購買される確率が最も高いクーポンを選択する問題にすることで、
正解率(Accuracy)でモデルの良し悪しを近似的に評価することが出来ます。
このように、近似した問題に対してモデルを構築することとして、以下は進めていきます。
モデル構築に於ける3つのデータ：training data, validation data, test data
データはまず大きく分けて、訓練用のデータとテスト用のデータ(test data)に大別されます。
特に、Kaggleのコンペティションのデータセットは明示的にそれらが分けられているものもあります。
通常、訓練用のデータで予測モデルを構築し、テスト用のデータで構築した予測モデルの精度を検証します。
<img src="files/data_split.png" width="600px;"/>
このチュートリアルでは、機械学習のアルゴリズムに基づいた予測モデルを構築しますが、
機械学習のアルゴリズムには一般にハイパーパラメタというものが存在します。
機械学習アルゴリズムには、内部パラメタとハイパーパラメタの2つのパラメタがあり、
内部パラメタは自動で学習しますが、ハイパーパラメタは予め設定する必要があります。
良い精度を出すためにはハイパーパラメタをこのうまく調整する必要があります。
実際に予測モデルを構築する際には、訓練用のデータをさらにtraining dataとvalidation dataに分けて、
あるハイパーパラメタでtraining dataから予測モデルを構築しvalidation dataで精度を検証し、
別のハイパーパラメタで予測モデルを構築しvalidation dataで精度を検証し、、、
ということを繰り返します。
最終的な精度評価に際しては、
validation dataへの検証実験で良い結果となったハイパーパラメタで構築された予測モデルを今度はtest dataに適用し、
実際に未知のデータに対して、どれくらいの精度で予測できるのかということ検証します。
ここでは、便宜的に訓練データの最後の1週間の期間(2012-06-17 00:00:00から2012-06-23 23:59:59)をvalidation dataとし、
残りの約1年の期間をtraining dataとして、データを分割します。
Can you write Python code for it?

validation_start = datetime.strptime('2012-06-17 00:00:00', '%Y-%m-%d %H:%M:%S')
validation_end = validation_start + timedelta(days=7)
dbcur.execute(''' DROP TABLE IF EXISTS coupon_visit_train_training;''')  # チュートリアルの便宜上一回削除します。
dbcur.execute(''' CREATE TABLE IF NOT EXISTS coupon_visit_train_training LIKE coupon_visit_train;''')
dbcur.execute('''
    INSERT INTO coupon_visit_train_training
        SELECT *
        FROM coupon_visit_train
        WHERE i_date >= "2011-07-01 00:00:00" AND i_date < %s
    ;
''', (validation_start, ))

dbcur.execute(''' DROP TABLE IF EXISTS coupon_visit_train_validation;''')  # チュートリアルの便宜上一回削除します。
dbcur.execute(''' CREATE TABLE IF NOT EXISTS coupon_visit_train_validation LIKE coupon_visit_train;''')
dbcur.execute('''
    INSERT INTO coupon_visit_train_validation
        SELECT *
        FROM coupon_visit_train
        WHERE i_date >= %s
    ;
''', (validation_start, ))


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
3. 実際に購買したクーポンと照らし合わせ、MAP@10を算出する。
Can you write Python code for it?

#  validation期間に購買したクーポンリストを抽出。
dbcur.execute('''
    SELECT
        user_id_hash, view_coupon_id_hash
        FROM coupon_visit_train_validation
        WHERE purchase_flg = 1
    ;
''')
user_pcoupon_true = {}
for row in dbcur.fetchall():
    if row[0] not in user_pcoupon_true:
        user_pcoupon_true[row[0]] = []
    user_pcoupon_true[row[0]].append(row[1])

#  ap10を算出する関数を定義。
def get_ap10(y_pred, y_true):
    ap10 = 0.
    y_true = set(y_true)
    for i in range(len(y_pred)):
        if y_pred[i] in y_true:
            c = set(y_pred[:i + 1])
            ap10 += len(y_true & c) / float(i + 1)
    ap10 /= min(len(y_true), 10)
    return ap10

map10 = 0.
n_purchased_user = 0.
for user_id in user_pcoupon_pred:
    if user_id not in user_pcoupon_true:
        # 当該ユーザがvalidation期間にcouponを買わなかった場合、
        # ap@10は0
        continue
    n_purchased_user += 1
    y_true = user_pcoupon_true[user_id]
    y_pred = user_pcoupon_pred[user_id]

    map10 += get_ap10(y_pred, y_true)

max_map10 = n_purchased_user / len(user_pcoupon_pred)
map10 /= len(user_pcoupon_pred)

print 'max_map@10: %.5f, map@10: %.5f' % (max_map10, map10)


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
Excercise
test dataに対して、同様の手法で予測・csvに結果を出力して、
コンペティションにサブミットしてください。
※1日に最大5回までしかサブミットできないので、注意してください。
4. 機械学習アルゴリズムによる予測モデルの構築
先の章では、モデリング対象や評価指標への理解を深めました。
具体的に先に挙げた、
- ユーザの属性データ(性別、年代、地域など)
- クーポンの属性データ(ジャンル、価格、地域など)
- いつどのユーザがどのクーポンを見たか、購買したか等のログデータ
のそれぞれの要素がどれぐらい将来の購買行動と関連があるのか、
を考える前に、まず簡単に機械学習アルゴリズムによる予測モデルを構築して、
アルゴリズムの内部がどうなっているのか、
アルゴリズムのinputとoutputがどうなっているのか、
見ていきましょう。
ここでは、下記の設定、
対象アルゴリズム：ロジスティック回帰
対象データ：ユーザの属性データとクーポンの属性データ
将来購買されるクーポンを予測するということをやってみましょう。
機械学習アルゴリズム
ロジスティック回帰は非常にシンプルなアルゴリズムで
下記の数式のように定義されています。
$$
v=Wx+b
$$
$$
y \simeq \tilde{y}=\frac{exp(v)}{\sum_{k=1}^{N}exp(v_k)}
$$
$x$：素性ベクトル(ユーザとクーポンの特徴を表現したベクトル)
$W$：素性ベクトルの各列をどれだけ評価するかのウェイトマトリックス(パラメタ)
$b$：バイアス項(パラメタ)
$N$：クラスの数(この場合、買われたか否か)
$v$：N次元ベクトル
$\tilde{y}$：各クラスが発生する確率
$y$：正解ベクトル
ロジスティック回帰は与えられた素性ベクトル$x$の各次元の重みを評価して、各クラスに所属する確率$\tilde{y}$を算出します。
モデルの学習の際には、$\tilde{y}$と$y$の誤差が最小となるように内部のパラメタ$W$と$b$を学習し、
予測の際には、入力$x$を計算式に当てはめ、出力結果$\tilde{y}$を見て、例えば、所属する確率が最大のクラスを予測結果とします。
つまり、学習の際は入力は$x$と$y$で、予測の際は入力は$x$のみです。
これは、他の多くの機械学習アルゴリズムに共通です。
機械学習ライブラリの1つであるscikit-learnは、
多くの機械学習アルゴリズムに対して共通のフレームワークで提供され内部がブラックボックス化されており、
中身が分からなくても簡単に利用できるものとなっています。
ここでは、scikit-learnで提供されているロジスティック回帰を利用していきます。
データ加工(素性/ラベル作成)
簡単のためログデータを用いずに、
ユーザの属性データとクーポンの属性データのみを用いて素性$x$を作成します。
特徴ベクトルを作成する際には、
1. 各次元のスケールを合わせる
2. 連続値で表現すべきでないところは、フラグにする
を意識しましょう。
ユーザの特徴ベクトル
sequel proでuser_listテーブルを見てみると、
素性に利用しやすそうな属性として、性別、年齢、場所があります。
性別と場所は連続値でないため、フラグで表現します。
年齢のような連続値は、場合によってはフラグとした方が良い時もありますが、
ここでは、連続値として扱います。
sequel proで年齢のカラムをソートしてみると、
最小値が15最大値が80なので素性を作る時はスケールを合わせるために、
年齢値から15引いて65で割ります。
このように、[0, 1]の値での表現のように最小値と最大値を整えるスケーリングをmini max scaling等と呼びます。
他には平均0分散1の標準正規分布に変換するstandard scalingがあります。
※いずれもscikit-learnのpreprocessingモジュール(http://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing )に実装されています。
Can you write Python code for it?

# ユニークな都道府県リストの取得
dbcur.execute(''' SELECT pref_name FROM prefecture_locations ORDER BY pref_name ; ''')
pref_data = []
for row in dbcur.fetchall():
    pref_data.append(row)

# ユーザの素性を作成。(ユーザの素性はtraining、validation, testで共通)
dbcur.execute('''
    SELECT
        t1.user_id_hash,
        IF(t1.sex_id = 'm', 1, 0),
        (t1.age-15)/65,
        ''' + ', '.join([u'IF(t1.pref_name = "' + p[0] + u'", 1, 0)' for i, p in enumerate(pref_data)]) + '''
    FROM user_list AS t1
''')

user_feature = {}  # ユーザの素性ベクトル
for row in dbcur.fetchall():
    user_feature[row[0]] = row[1:]


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
coupon_list_train.csv, coupon_list_test.csv
2つともWarningが出ますが、
日時の値が正しくないために発生しているだけなので、無視します。
下記のクエリの
SQL
SET validperiod=IF(@validperiod = 'NA', Null, @validperiod)
を外すと、おそらくNULLが入ってほしい箇所に0が入ってしまうため、ここでは、NULLがはいるように変換します。
Can you write Python code for it?

### coupon_list_train
csv_path = DATA_DIR + '/coupon_list_train.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_list_train;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_list_train (
    capsule_text VARCHAR(20),
    genre_name VARCHAR(50),
    price_rate INT,
    catalog_price INT,
    discount_price INT,
    dispfrom DATETIME,
    dispend DATETIME,
    dispperiod INT,
    validfrom DATE,
    validend DATE,
    validperiod INT,
    usable_date_mon VARCHAR(7),
    usable_date_tue VARCHAR(7),
    usable_date_wed VARCHAR(7),
    usable_date_thu VARCHAR(7),
    usable_date_fri VARCHAR(7),
    usable_date_sat VARCHAR(7),
    usable_date_sun VARCHAR(7),
    usable_date_holiday VARCHAR(7),
    usable_date_before_holiday VARCHAR(7),
    large_area_name VARCHAR(30),
    ken_name VARCHAR(8),
    small_area_name VARCHAR(30),
    coupon_id_hash VARCHAR(32),
    PRIMARY KEY(coupon_id_hash),
    INDEX(ken_name),
    INDEX(genre_name)

) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_list_train
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(capsule_text,genre_name,price_rate,catalog_price,discount_price,dispfrom,dispend,dispperiod,validfrom,validend,@validperiod,usable_date_mon,usable_date_tue,usable_date_wed,usable_date_thu,usable_date_fri,usable_date_sat,usable_date_sun,usable_date_holiday,usable_date_before_holiday,large_area_name,ken_name,small_area_name,coupon_id_hash)
SET validperiod=IF(@validperiod = 'NA', Null, @validperiod)
;
''')

### coupon_list_test
csv_path = DATA_DIR + '/coupon_list_test.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_list_test;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_list_test (
    capsule_text VARCHAR(20),
    genre_name VARCHAR(50),
    price_rate INT,
    catalog_price INT,
    discount_price INT,
    dispfrom DATETIME,
    dispend DATETIME,
    dispperiod INT,
    validfrom DATE,
    validend DATE,
    validperiod INT,
    usable_date_mon VARCHAR(7),
   ...


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
クーポンの特徴ベクトル
アイテムは、比較的用意に利用できそうなフラグのカラムが多いです。
ここでは、簡単のためフラグのみを、
学習に利用するクーポンはvalidation期間の前の7日で購買可能なデータ
を利用して特徴ベクトルを作っていきます。
※実際には、より多くのデータを使った方が精度は出る傾向にあります。
Can you write Python code for it?

training_start = validation_start - timedelta(days=7)  # 訓練開始日時を算出。

# カテゴリリストの取得
dbcur.execute(''' SELECT DISTINCT(capsule_text) FROM coupon_list_train ORDER BY capsule_text;''')
capsule_data = []
for row in dbcur.fetchall():
    capsule_data.append(row)

# ジャンルリストの取得
dbcur.execute(''' SELECT DISTINCT(genre_name) FROM coupon_list_train ORDER BY genre_name;''')
genre_data = []
for row in dbcur.fetchall():
    genre_data.append(row)

# 大エリアリストの取得
dbcur.execute(''' SELECT DISTINCT(large_area_name) FROM coupon_list_train ORDER BY large_area_name;''')
larea_data = []
for row in dbcur.fetchall():
    larea_data.append(row)

# 都道府県リストの取得
dbcur.execute(''' SELECT DISTINCT(ken_name) FROM coupon_list_train ORDER BY ken_name;''')
pref_data = []
for row in dbcur.fetchall():
    pref_data.append(row)

# 小エリアリストの取得
dbcur.execute(''' SELECT DISTINCT(small_area_name) FROM coupon_list_train ORDER BY small_area_name;''')
sarea_data = []
for row in dbcur.fetchall():
    sarea_data.append(row)

def get_item_feature(f_date, t_date):
    #  クーポンの素性を作成する関数。
    # @f_date：対象期間の開始日時
    # @t_date：対象期間の終了日時
    # テーブルが訓練用のテーブルとなっている為、training とvalidationのデータを作成する際にしか利用できない。
    dbcur.execute('''
        SELECT
            coupon_id_hash,
            ''' + ', '.join([u'IF(capsule_text = "' + p[0] + u'", 1, 0)' for i, p in enumerate(capsule_data)]) + ''',
            ''' + ', '.join([u'IF(genre_name = "' + p[0] + u'", 1, 0)' for i, p in enumerate(genre_data)]) + ''',
            COALESCE(CAST(usable_date_mon AS SIGNED), 0),
            COALESCE(CAST(usable_date_tue AS SIGNED), 0),
            COALESCE(CAST(usable_date_wed AS SIGNED), 0),
            COALESCE(CAST(usable_date_thu AS SIGNED), 0),
            COALESCE(CAST(usable_date_fri AS SIGNED), 0),
            COALESCE(CAST(usable_date_sat AS SIGNED), 0),
            COALESCE(CAST(usable_date_sun AS SIGNED), 0),
            COALESCE(CAST(usable_date_holiday AS SIGNED), 0),
            COALESCE(CAST(usable_date_before_holiday AS SIGNED), 0),
            ''' + ',...


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
テーブルの作成に利用したCREATE TABLE文には、
テーブルの型の定義ではなく、インデックスと呼ばれるものの定義も含まれています。
インデックスとはデータの検索を高速化するものです。
PRIMARY KEY
テーブル内でuniqueで、かつ、検索するカラムに付与する。
例えば、user_listテーブルのuser_id_hashは当該テーブルで、ユニークであり，かつ、ユーザの検索によく用いるため、PRIMARY KEYを付与しておいた方が良い。
INDEX
テーブル内でuniqueではないが、検索するカラムに付与する。例えば、ユーザを性別や年齢に応じて検索・集計して、割合を見たい場合には、sex_idやageなどのカラムに付与しておいた方が良い。
TODO
MYSQL関数などの説明の加筆。
Exercise
下記の他のファイルについても同様にテーブルを作成し、データをインサートしてください。
- prefecture_locations.csv
- coupon_area_train.csv, coupon_area_test.csv
- coupon_detail_train.csv
- coupon_visit_train.csv
- coupon_list_train.csv, coupon_list_test.csv
実装例

prefecture_locations.csv
Can you write Python code for it?

### prefecture_locations
csv_path = DATA_DIR + '/prefecture_locations.csv'
dbcur.execute('''DROP TABLE IF EXISTS prefecture_locations;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS prefecture_locations (
    pref_name VARCHAR(15),
    PRIMARY KEY(pref_name),
    prefectual_office VARCHAR(15),
    latitude DOUBLE,
    longitude DOUBLE
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE prefecture_locations
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(pref_name, prefectual_office, latitude, longitude)
;
''')


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
モジュールのimportや変数の初期化
次に、このチュートリアルで利用するモジュールのimportや一部の変数の初期化を行います。
python
%matplotlib inline
は ipython notebookに特有のマジックコマンドというものです。
pythonの文法と異なりますが、matplotlibという画像を描画するライブラリの出力結果がブラウザ上に表示されるように設定するものです。
(ここでは、おまじない程度に考えてください。)
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import MySQLdb
import numpy

from sklearn.utils import shuffle
from sklearn.cross_validation import train_test_split
from sklearn.metrics import f1_score, accuracy_score
from sklearn.linear_model import LogisticRegression
from datetime import datetime, timedelta
from itertools import product

# Random Seed
rng = numpy.random.RandomState(1234)

dbcon = MySQLdb.connect(**MYSQL)
dbcur = dbcon.cursor()


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
ランダム推定・MAP@10の評価
先にも触れましたが、本コンペティションではMAP@10というスコアを競い合っています。
提出のフォーマットは、https://www.kaggle.com/c/coupon-purchase-prediction/details/evaluation に
記載されています。
訓練データをtraining dataと validation dataに分割したので、
試しに、各ユーザが購買するクーポンをランダムに推定して、
・MAP@10の性質
・ランダム推定の精度
・サブミッションの形式
をざっくり把握していきましょう。

validation期間に購買されうるクーポンを抽出する。
抽出したクーポン群から各ユーザが購買するクーポンをランダムに10個選び、予測結果とする。
実際に購買したクーポンと照らし合わせ、MAP@10を算出する。
サブミッション用のcsvファイルを作成する。

の4ステップで見ていきます。
1. validation期間に購買されうるクーポンを抽出する。
Can you write Python code for it?

# validation 期間に購買されうるクーポンの抽出
dbcur.execute('''
    SELECT
        coupon_id_hash
        FROM coupon_list_train
        WHERE
            NOT (dispend <= %s OR dispfrom > %s)
    ;
''', (validation_start, validation_end))
coupon_ids = []
for row in dbcur.fetchall():
    coupon_ids.append(row[0])


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
先ほどの、ランダム予測よりだいぶ上がったようです。
今回の例では、ユーザの特徴とクーポンの特徴を単純に結合しましたが、
モデルの入力である素性は無数の形式が存在します。
機械学習アルゴリズムを利用する際、この素性を如何に設計するかは非常に重要です。
他に予測精度をあげるポイントとして、
・機械学習アルゴリズムを選ぶ。
・機械学習アルゴリズムのハイパーパラメタを調整する。
・前処理をちゃんとやる。
・目的関数を正しく選ぶ。
などがあります。
興味がある方は、適用を検討してみてください。
Excercise

同様の方法を test dataに適用、各ユーザの購買クーポンを予測し、submissionしてみてください。
学習に用いるデータ期間を7日から増やして、精度の変化を見てみてください。

5. データの概観把握・予測モデルの改善
TODO: もっと多くの実例を取り上げる。
このコンペティションの最終的なゴールは、各ユーザが将来どのクーポンを購入するかを予測することです。
先のように、ランダムに選ぶよりは、ユーザやクーポンの特徴をモデリングした方が将来の購買するクーポンを予測しやすいことが分かりました。
では、将来ユーザがどのクーポンを購入するかは、何と関係があるでしょうか。
その関係性を捉えることで、モデリングの際によりよい結果が得られることが多いです。
下記の2ステップで、購買確率の予測に寄与する指標が何かを探していきます。
1. 重要のデータの量やラベルの種類を確認するために、集計&可視化する。
2. 関係性についての仮説をたてる。
3. 仮説を検証するために、データを集計&可視化する。
重要のデータの量やラベルの種類の確認
ログデータが少ないユーザの数の確認。
TODO：coldstart問題について加筆。
TODO：GROUP BYの説明とこの集計の仕組みの説明の加筆。
Can you write Python code for it?

dbcur.execute('''
    SELECT
        COUNT(*),
        SUM(purchase_flg),
        COUNT(DISTINCT(view_coupon_id_hash))
    FROM
        coupon_visit_train
    GROUP BY user_id_hash
    ;
''')
n_view = []
n_purchase = []
n_view_u = []

for row in dbcur.fetchall():
    n_view.append(int(row[0]))
    n_purchase.append(int(row[1]))
    n_view_u.append(int(row[2]))

n_view = numpy.asarray(n_view)
n_purchase = numpy.asarray(n_purchase)
n_view_u = numpy.asarray(n_view_u)

### user-coldstartがどういった状況か見る為に、最初の20件だけ見る。
span = 20
fig = plt.figure(figsize=(18, 8))
ax = fig.add_subplot(2, 3, 1)
ax.hist(n_view, bins=numpy.arange(0, span), cumulative=True)
ax.set_title('page view count distribution')

ax = fig.add_subplot(2, 3, 2)
ax.hist(n_purchase, bins=numpy.arange(0, span), cumulative=True)
ax.set_title('purchase count distribution')

ax = fig.add_subplot(2, 3, 3)
ax.hist(n_view_u, bins=numpy.arange(0, span), cumulative=True)
ax.set_title('unique page view count distribution')

ax = fig.add_subplot(2, 3, 4)
ax.plot(n_view, n_purchase, 'x')
ax.set_title('X=page view count, Y=purchase count')

ax = fig.add_subplot(2, 3, 5)
ax.plot(n_view_u, n_purchase, 'x')
ax.set_title('X=unique page view count, Y=purchase count')

ax = fig.add_subplot(2, 3, 6)
ax.plot(n_view, n_view_u, 'x')
ax.set_title('X=page view count, Y=unique page view count')

plt.show()

## 3Dにしても良く分からないことが多いので，辞めましょう。
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(5, 5))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(n_view, n_view_u, n_purchase, marker='x')
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.show()


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
実行すると、それぞれのレコードでWarningが発生しますが、
データベースに展開されたレコードのlongitudeを確認すると正しく展開されているため、ここではWarningは無視します。
(確認しておりませんが、行末の改行コードがWarningの原因かもしれません、、、)
- coupon_area_train.csv, coupon_area_test.csv
Can you write Python code for it?

### coupon_area_train
csv_path = DATA_DIR + '/coupon_area_train.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_area_train;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_area_train (
    small_area_name  VARCHAR(32),
    pref_name VARCHAR(15),
    coupon_id_hash VARCHAR(32),
    INDEX(coupon_id_hash),
     INDEX(pref_name)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_area_train
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(small_area_name,pref_name,coupon_id_hash)
;
''')

### coupon_area_test
csv_path = DATA_DIR + '/coupon_area_test.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_area_test;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_area_test (
    small_area_name  VARCHAR(32),
    pref_name VARCHAR(15),
    coupon_id_hash VARCHAR(32),
    INDEX(coupon_id_hash),
    INDEX(pref_name)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_area_test
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(small_area_name,pref_name,coupon_id_hash)
;
''')


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
最終的な精度評価に用いるテストデータに含まれる各クーポンに対して、どれくらいviewやpurchaseのデータが存在するか、の確認。
ほとんどのクーポンでviewデータがなく、また、全てのクーポンでpurchaseのデータが存在しないitem-cold start状態なので、
協調フィルタリングやその派生の手法は不向きであると考えられます。
Can you write Python code for it?

dbcur.execute('''
    SELECT
        t1.coupon_id_hash, COUNT(t2.view_coupon_id_hash), COALESCE(SUM(t2.purchase_flg), 0)
    FROM coupon_list_test AS t1
    LEFT JOIN coupon_visit_train AS t2 ON t1.coupon_id_hash = t2.view_coupon_id_hash
    GROUP BY t1.coupon_id_hash
    ORDER BY SUM(t2.purchase_flg)
    ;
''')
view_count = []
purchase_count = []
for row in dbcur.fetchall():
    view_count.append(int(row[1]))
    purchase_count.append(int(row[2]))

view_count = numpy.asarray(view_count)
purchase_count = numpy.asarray(purchase_count)
plt.figure()
plt.plot(purchase_count, view_count, '.')
plt.show()


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
次に、データのインサートです。
csvファイルなど、dumpされたファイルからMySQLにインサートする場合にはLOAD DATA INFILE構文を利用します。
LOAD DATA INFILE構文については、 http://dev.mysql.com/doc/refman/5.6/ja/load-data.html を参照ください。
Can you write Python code for it?

csv_path = DATA_DIR + '/user_list.csv'
query = '''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE user_list
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(reg_date, sex_id, age,@withdraw_date, pref_name, user_id_hash)
SET
    withdraw_date = IF(CHAR_LENGTH(@withdraw_date) != 19 , '9999-12-31 23:59:59', STR_TO_DATE(@withdraw_date, "%Y-%m-%d %H:%i:%s"))
;
'''
dbcur.execute(query)


I am working on the file ".ipynb_checkpoints/Lecture_01-checkpoint.ipynb".
The first task is:
関係性についての仮説をたてる
どういったユーザがどういったクーポンを購買するかについての仮説を立てます。
今回のデータセットでは、例えば、下記のような仮説が立てられます。
1. ユーザは自分の地域と同じ地域を対象とするクーポンを購買する可能性が高い。
2. ユーザは女性の方が男性より割引率の高いクーポンを購買する可能性が高い。
3. ユーザは最後に購買したクーポンと同じジャンルのクーポンを、将来購買する可能性が高い。
4. (過去に購買したクーポンがない)ユーザは最後に閲覧したクーポンと同じジャンルのクーポンを、将来購買する可能性が高い。
仮説を立てる際に意識して頂きたいことは、
立てた仮説がどういったユーザがどういったクーポンを購買するかを具体的に表現していて、
そこから、どのようにデータを集計し、検証を行うかが具体的に分かるレベルで考えるということです。
たとえば、
「ユーザは過去の行動に応じて、将来に購買するクーポンを変える可能性が高い。」
という仮説をたてたとしましょう。しかし、この抽象度では具体的に何をすべきか分かりません。
従って、正しいかもしれないが、次のアクションに繋がらないという点で、役に立たない(≒価値がない)仮説だと言えます。
また、別の角度から見ると、立てた仮説はグレーであればあるほど、白黒はっきりさせた(検証した)あとの有用性が大きいことが多いです。
実際に、「データを分析する」際には、この「仮説を立てて、重要なところから取りかかる」ということが非常に重要です。
スキルは所詮スキルでしかなく、使いどころを誤るといたずらに時間を溶かす結果になりますので、是非意識してもらえればと思います。
仮説を検証するために、データを集計&可視化する
具体例として挙げた仮説を一つずつ検証いていきましょう。
1. ユーザは自分の地域と同じ地域を対象とするクーポンを購買する可能性が高い。
この仮説は、時間軸とは関係のなく、いわゆる分析的な集計である為、ここでは訓練データ全体で集計します。
Can you write Python code for it?

dbcur.execute('''
    SELECT
        AVG(same_pref_purchase_cnt),
        AVG(same_pref_view_cnt),
        AVG(same_pref_purchase_cnt / same_pref_view_cnt),
        AVG(diff_pref_purchase_cnt),
        AVG(diff_pref_view_cnt),
        AVG(diff_pref_purchase_cnt / diff_pref_view_cnt)
    FROM (
        SELECT
            t1.user_id_hash,
            SUM(t1.pref_name = t3.ken_name AND purchase_flg = 1) AS same_pref_purchase_cnt,
            SUM(t1.pref_name = t3.ken_name) AS same_pref_view_cnt,
            SUM(t1.pref_name != t3.ken_name AND purchase_flg = 1) AS diff_pref_purchase_cnt,
            SUM(t1.pref_name != t3.ken_name) AS diff_pref_view_cnt
        FROM user_list AS t1
        LEFT JOIN coupon_visit_train AS t2 ON t1.user_id_hash = t2.user_id_hash
        LEFT JOIN coupon_list_train AS t3 ON t2.view_coupon_id_hash = t3.coupon_id_hash
        WHERE t1.pref_name != ""
        GROUP BY t1.user_id_hash
    ) AS t1

    ;
''')
data = None
for row in dbcur.fetchall():
    data = row
print 'same_purchase: %.2f, same_view: %.2f, same_rate: %.2f, diff_purchase: %.2f, diff_view: %.2f, diff_rate: %.2f' % (data)


I am working on the file ".ipynb_checkpoints/Lesson 2A-L1-checkpoint.ipynb".
The first task is:
Crop Image
Can you write Python code for it?

cropped = img[1700:,300:,:]

show_img(cropped)

# Size of Cropped Image
cropped.shape


I am working on the file ".ipynb_checkpoints/Lesson 2A-L1-checkpoint.ipynb".
The first task is:
Inspect Image Values
My image has three channels: Red, Green, and Blue. That's why when selecting the 50th row and 100th column, three values are returned.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Lesson 2A-L1-checkpoint.ipynb".
The first task is:
Load and Display an Image
Can you write Python code for it?

img_path = os.path.join(IMG, 'sea.jpeg')
img = cv2.imread(img_path)
# What is the size of the Image?
print img.shape
# What is the class of the image?
print img.dtype
show_img(img)


I am working on the file ".ipynb_checkpoints/Lesson 2A-L1-checkpoint.ipynb".
The first task is:
Some Helper Functions
Can you write Python code for it?

def show_img(img):
    """
    Function takes an image, and shows the image using pyplot.
    The image is shown in RGB
    """
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    plt.imshow(img)


I am working on the file ".ipynb_checkpoints/Lesson 2A-L1-checkpoint.ipynb".
The first task is:
TODO: Extract a 2D slice between rows 101 to 103 and columns 201 to 203 (inclusive)
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Linear and Quadratic Discriminant Analysis-checkpoint.ipynb".
The first task is:
Confusion matrix:
Can you write Python code for it?

predictions = [model.predict(h) for h in Hemocrit.hemocrit]
truth = Hemocrit.status.values
confusion = pd.DataFrame(np.array([predictions, truth]).T, columns=('Prediction', 'Truth'))

confusion.groupby('Prediction').Truth.value_counts()


I am working on the file ".ipynb_checkpoints/Linear and Quadratic Discriminant Analysis-checkpoint.ipynb".
The first task is:
Iris Example
Can you write Python code for it?

iris = pd.read_csv('data/iris.csv')

iris_training = pd.concat([iris[iris.Species == 'setosa'].sample(25, random_state=8675309),
                           iris[iris.Species == 'versicolor'].sample(25, random_state=8675309),
                           iris[iris.Species == 'virginica'].sample(25, random_state=8675309)])

iris_test = iris.loc[iris.index.difference(iris_training.index)]

iris_training.groupby('Species')['Sepal.Length'].hist()
plt.show()

model = lda(iris_training.Species, iris_training['Sepal.Length'])

predictions = np.array([model.predict(x) for x in iris_test['Sepal.Length']])
truth = iris_test['Species'].values

results = pd.DataFrame(np.array([predictions, truth]).T,
                       columns=['Prediction', 'Truth'])
vcounts = results.groupby('Prediction').Truth.value_counts()
vcounts_dense = np.zeros((3,3))
for i in xrange(model.K):
    k_i = model.K_labels[i]
    for j in xrange(model.K):
        k_j = model.K_labels[j]
        try:
            vcounts_dense[i,j] = vcounts[k_i][k_j]
        except KeyError:
            pass
comparison = pd.DataFrame(vcounts_dense, columns=model.K_labels)
comparison['Truth'] = model.K_labels
comparison

x = stats.norm.rvs(loc=4, scale=1.3, size=200)

def qda(K_x, X):
    K = set(K_x)
    X_grouped = {k:[] for k in list(K)}
    for k, x in zip(K_x, X):
        X_grouped[k].append(x)

    # Maximize f to find mu and sigma
    params_k = {}
    for k, x in X_grouped.iteritems():
        guess = (np.mean(x), np.std(x))

        # Variance must be greater than 0.
        constraints = {'type': 'eq', 'fun': lambda params: params[1] > 0}
        f = lambda params: np.sum(((-1.*(x - params[0])**2)/(2.*params[1]**2)) - np.log(params[1]*np.sqrt(2.*np.pi)))
        params_k[k] = optimize.minimize(lambda params: -1.*f(params), guess, constraints=constraints).x

    K_ordered = np.array(params_k.keys())[np.argsort(np.array(zip(*params_k.values())[0]))]
    X_bound = []
    for i in xrange(1, len(K_ordered)):
        k_0, k_1 = K_ordered[i-1],...


I am working on the file ".ipynb_checkpoints/Linear and Quadratic Discriminant Analysis-checkpoint.ipynb".
The first task is:
LDA assumes that the variance in each group is the same, and that the predictor(s) are normally distributed for each group. In other words, different $\mu_k$, one shared $\sigma$.
Can you write Python code for it?

X = [np.linspace(0, 7, 50),
     np.linspace(2, 10, 50),
     np.linspace(7, 16, 50)]

plt.figure(figsize=(10, 4))
k = 1
for x in X:
    mu_k = x.mean()
    plt.plot(x, stats.norm.pdf(x, loc=mu_k))
    plt.plot([mu_k, mu_k], [0, 0.5], c='k')
    plt.text(mu_k + 0.2, 0.5, "$\mu_%i$" % k, size=18)
    k += 1
plt.ylim(0, 0.75)

plt.xlabel('Predictor', size=18)
plt.ylabel('Probability', size=18)
plt.show()


I am working on the file ".ipynb_checkpoints/Linear and Quadratic Discriminant Analysis-checkpoint.ipynb".
The first task is:
LDA is like inverted ANOVA: ANOVA looks for differences in a continuous response among categories, whereas LDA infers categories using a continuous predictor.
Can you write Python code for it?

bycategory = [ [Y[i] for i in xrange(len(Y)) if K[i] == k] for k in categories ]

plt.figure(figsize=(10, 5))
plt.subplot(121)
plt.boxplot(bycategory)
plt.ylim(0, 8)
plt.title('ANOVA')

plt.subplot(122)
plt.boxplot(bycategory, 0, 'rs', 0)
plt.title('LDA')
plt.xlim(0, 8)

plt.show()


I am working on the file ".ipynb_checkpoints/Linear and Quadratic Discriminant Analysis-checkpoint.ipynb".
The first task is:
Receiver Operator Characteristic (ROC) curve
Provides a visual summary of the confusion matrix over a range of criteria. Given a confusion matrix, $N=TN+FP$, $P=TP+FN$.
Can you write Python code for it?

plt.figure(figsize=(5, 5))
plt.text(0.25, 0.75, 'TN', size=18)
plt.text(0.75, 0.75, 'FP', size=18)
plt.text(0.25, 0.25, 'FN', size=18)
plt.text(0.75, 0.25, 'TN', size=18)

plt.xticks([0.25, 0.75], ['Neg', 'Pos'], size=20)
plt.yticks([0.25, 0.75], ['Pos', 'Neg'], size=20)
plt.ylabel('Truth', size=24)
plt.xlabel('Prediction', size=24)
plt.title('Confusion Matrix', size=26)
plt.show()


I am working on the file ".ipynb_checkpoints/Linear and Quadratic Discriminant Analysis-checkpoint.ipynb".
The first task is:
The true positive rate, or Power (or Sensitivity) is $\frac{TP}{P}$ and the Type 1 error is $\frac{FP}{N}$. The ROC curve shows Power vs. Type 1 error. Ideally, we can achieve a high true positive rate at a very low false positive rate:
Can you write Python code for it?

plt.figure()
X = np.linspace(0., 0.5, 200)
f = lambda x: 0.001 if x < 0.01 else 0.8
plt.plot(X, map(f, X))
plt.ylabel('True positive rate (power)')
plt.xlabel('False positive rate (type 1 error)')
plt.show()


I am working on the file ".ipynb_checkpoints/LogisticRegScikitlearn-checkpoint.ipynb".
The first task is:
Exploring the data

Let us get a feel for the parameters.
We see that age is a tailed distribution.
Certainly not Gaussian! We don't see much of a correlation between many of the features, with the exception of Age and Age2.
Hours worked has some interesting behaviour. How would one describe this distribution?
Can you write Python code for it?

import seaborn as seaborn
g = seaborn.pairplot(data)

logreg = linear_model.LogisticRegression(C=1e5)

age2 = np.square(data['age'])
data = data[['age', 'educ', 'hours']]
data['age2'] = age2
data['income'] = income
X = data[['age', 'age2', 'educ', 'hours']]
Y = data['income']
logreg.fit(X, Y)

# check the accuracy on the training set
logreg.score(X, Y)

Y.mean()


I am working on the file ".ipynb_checkpoints/Lxmls_Day5-checkpoint.ipynb".
The first task is:
5.4.4 Some final reflections on Backpropagation
If you are new to the neural network topic, this is about the most important piece of theory you should learn
about deep learning. Here are some reflections that you should keep in mind.
* Thanks to the multi-layer structure and the chain rule, Backpropagation allows models that compose
linear and non-linear functions with any depth (in principle5
).
* The formulas are also valid for other cost functions and output layer non-linearities with minor modifi-
cations. It is only necessary to compute the equivalent of Eq. 5.13.
* The formulas are also valid for hidden non-linearities other than the sigmoid. Element-wise non-linear
transformations still allow the simplification in Eq. 5.21. With little effort it is also possible to deal with
other cases.
* However, there is an important limitation: unlike the log-linear models, the optimization problem is non
convex. This removes some formal guarantees, most importantly we can get trapped in local minima
during training
5.5 Deriving gradients and GPU code with Theano
5.5.1 An Introduction to Theano
As you may have observed, the speed of SGD training for MLPs slows down considerably when we increase the number of layers. One reason for this is that the code that we use here is not very optimized. It is thought for you to learn the basic principles. Even if the code was more optimized, it would still be very slow for reasonable network sizes. The cost of computing each linear layer is proportional to the dimensionality of the previous and current layers, which in most cases will be rather large.
For this reason most deep learning applications use Graphics Processing Units (GPU) in their computations. This specialized hardware is normally used to accelerate computer graphics, but can also be used for general computation intensive tasks. However, we need to deal with specific interfaces and operations in order to use a GPU. This is where Theano comes in. Theano is a multidimensional symbolic expression python module
with...
Can you write Python code for it?

# Numpy code
x = test_x # Test set
W1, b1 = mlp.params[:2] # Weights and bias of fist layer
z1 = np.dot(W1, x) + b1 # Linear transformation
tilde_z1 = 1/(1+np.exp(-z1)) # Non-linear transformation


I am working on the file ".ipynb_checkpoints/Lxmls_Day5-checkpoint.ipynb".
The first task is:
5.5.2 Symbolic Forward Pass
Exercise 5.3 Complete the method forward() inside of the lxmls/deep learning/mlp.py:class TheanoMLP. Note that this is called only once at the initialization of the class. To debug your implementation put a breakpoint at the init function call. Hint: Note that this is very similar to NumpyMLP.forward(). You just need to keep track of the symbolic variable representing the output of the network after each layer is applied and compile the function at the end. After you are finished instantiate a Theano class and check that Numpy and Theano forward pass are the same.
Can you write Python code for it?

def _forward(self, x, all_outputs=False):
    """
    Symbolic forward pass

    all_outputs = True  return symbolic input and intermediate activations
    """

    # This will store activations at each layer and the input. This is
    # needed to compute backpropagation
    if all_outputs:
        activations = [x]

        # Input
    tilde_z = x

    # ----------
    # Solution to Exercise 5.3
    for n in range(self.n_layers):

        # Get weigths and bias (always in even and odd positions)
        W = self.params[2*n]
        b = self.params[2*n+1]

        z = T.dot(W, tilde_z) + b # Linear transformation

        # see e.g. theano.printing.debugprint(tilde_z)
        z.name = 'z%d' % (n+1)

        # Non-linear transformation
        if self.actvfunc[n] == "sigmoid":
            tilde_z = T.nnet.sigmoid( z )
        elif self.actvfunc[n] == "softmax":
            tilde_z = T.nnet.softmax( z.T ).T

        tilde_z.name = 'tilde_z%d' % (n+1) # Name variable

        if all_outputs:
            activations.append(tilde_z)
    # End of solution to Exercise 5.3
    # ----------

    if all_outputs:
        tilde_z = activations

    return tilde_z

mlp_a = dl.NumpyMLP(geometry, actvfunc)
mlp_b = dl.TheanoMLP(geometry, actvfunc)


I am working on the file ".ipynb_checkpoints/Lxmls_Day5-checkpoint.ipynb".
The first task is:
5.5.3 Symbolic Differentiation
Exercise 5.4 We first see an example that does not use any of the code in TheanoMLP but rather continues from what
you wrote in Ex. 5.2. In this exercise you completed a sigmoid layer with Theano. To get some values for the weights
we used the first layer of the network you trained in 5.2. Now we are going to use the second layer as well. This is thus assuming that your network in 5.2 has only two layers e.g. the recommended geometry (I, 20, 2). Make sure this is the case before starting this exercise.
For the sake of clarity, lets write here the part of Ex. 5.2 that we had completed
Can you write Python code for it?

# Get the values from our MLP
W1, b1 = mlp.params[:2] # Weights and bias of fist layer
# First layer symbolic variables
_x = T.matrix('x')
_W1 = theano.shared(value=W1, name='W1', borrow=True)
_b1 = theano.shared(value=b1, name='b1', borrow=True, broadcastable=(False, True))
# First layer symbolic expressions
_z1 = T.dot(_W1, _x) + _b1
_tilde_z1 = T.nnet.sigmoid(_z1)


I am working on the file ".ipynb_checkpoints/Lxmls_Day5-checkpoint.ipynb".
The first task is:
5.5.4 Symbolic mini-batch update
Exercise 5.5 Define the updates list. This is a list where each element is a tuple of a parameter and the update rule to be
applied that parameter. In this case we are defining the SGD update rule, but take into account that using more complex
update rules like e.g. momentum or adam implies just replacing the last line of the following snippet.
Can you write Python code for it?

W2, b2 = mlp_a.params[2:4]

# Second layer symbolic variables
_W2 = theano.shared(value=W2, name='W2', borrow=True)
_b2 = theano.shared(value=b2, name='b2', borrow=True,
                    broadcastable=(False, True))
_z2 = T.dot(_W2, _tilde_z1) + _b2
_tilde_z2 = T.nnet.softmax(_z2.T).T

# Ground truth
_y = T.ivector('y')

# Cost
_F = -T.mean(T.log(_tilde_z2[_y, T.arange(_y.shape[0])]))

# Gradient
_nabla_F = T.grad(_F, _W1)
nabla_F = theano.function([_x, _y], _nabla_F)

# Print computation graph
print "\nThis is my softmax classification cost\n"
theano.printing.debugprint(_F)


I am working on the file ".ipynb_checkpoints/Lxmls_Day5-checkpoint.ipynb".
The first task is:
Exercise 5.6
Can you write Python code for it?

import time

# Understanding the mini-batch function and givens/updates parameters

# Numpy
geometry = [train_x.shape[0], 20, 2]
actvfunc = ['sigmoid', 'softmax']
mlp_a = dl.NumpyMLP(geometry, actvfunc)
#
init_t = time.clock()
sgd.SGD_train(mlp_a, n_iter, bsize=bsize, lrate=lrate, train_set=(train_x, train_y))
print "\nNumpy version took %2.2f sec" % (time.clock() - init_t)
acc_train = sgd.class_acc(mlp_a.forward(train_x), train_y)[0]
acc_test = sgd.class_acc(mlp_a.forward(test_x), test_y)[0]
print "Amazon Sentiment Accuracy train: %f test: %f\n" % (acc_train, acc_test)

# Theano grads
mlp_b = dl.TheanoMLP(geometry, actvfunc)
init_t = time.clock()
sgd.SGD_train(mlp_b, n_iter, bsize=bsize, lrate=lrate, train_set=(train_x, train_y))
print "\nCompiled gradient version took %2.2f sec" % (time.clock() - init_t)
acc_train = sgd.class_acc(mlp_b.forward(train_x), train_y)[0]
acc_test = sgd.class_acc(mlp_b.forward(test_x), test_y)[0]
print "Amazon Sentiment Accuracy train: %f test: %f\n" % (acc_train, acc_test)

# Theano compiled batch

# Cast data into the types and shapes used in the theano graph
# IMPORTANT: This is the main source of errors when beginning with theano
train_x = train_x.astype(theano.config.floatX)
train_y = train_y.astype('int32')

# Model
mlp_c = dl.TheanoMLP(geometry, actvfunc)

# Define givens variables to be used in the batch update
# Get symbolic variables returning a mini-batch of data

# Define updates variable. This is a list of gradient descent updates
# The output is a list following theano.function updates parameter. This
# consists on a list of tuples with each parameter and update rule
_x = T.matrix('x')
_y = T.ivector('y')
_F = mlp_c._cost(_x, _y)
updates = [(par, par - lrate*T.grad(_F, par)) for par in mlp_c.params]

#
# Define the batch update function. This will return the cost of each batch
# and update the MLP parameters at the same time using updates
batch_up = theano.function([_x, _y], _F, updates=updates)
n_batch = int(np.ceil(float(train_x.shape[1])/bsize))
#

init_t =...