I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
But that's ugly. There's a thing called ggplot for R that looks nice. We want to look nice. We want to look like ggplot.
Can you write Python code for it?

# Import matplotlib
# What's available?

# Use ggplot

# Make a histogram

# Try some other styles


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
Drawing pictures
Okay okay enough code and enough stupid numbers. I'm visual. I want graphics. Okay????? Okay.
Can you write Python code for it?

!pip install matplotlib

import matplotlib.pyplot as plt
%matplotlib inline
# This will scream we don't have matplotlib.
df['Ht (Ft.)'].hist()


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
I want more graphics! Do tall people make more money?!?!
Can you write Python code for it?

# How does experience relate with the amount of money they're making?

# At least we can assume height and weight are related

# At least we can assume height and weight are related
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html

# We can also use plt separately
# It's SIMILAR but TOTALLY DIFFERENT


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
I want to know how many people are in each position. Luckily, pandas can tell me!
Can you write Python code for it?

# Grab the POS column, and count the different values in it.
df['POS'].value_counts() # counts the number of values that match each position

df['Race'].value_counts() # race of players


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
If we scroll we can see all of it. But maybe we don't want to see all of it. Maybe we hate scrolling?
Can you write Python code for it?

# Look at the first few rows

df.head() # shows header + first 5 rows!


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
NOTE: That was not df['Name', 'Age'], it was df[['Name', 'Age']]. You'll definitely type it wrong all of the time. When things break with pandas it's probably because you forgot to put in a million brackets.
Describing your data
A powerful tool of pandas is being able to select a portion of your data, because who ordered all that data anyway.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
Now that was a little weird, yes - we used df['POS'] instead of df[['POS']] when viewing the data's details.
But now I'm curious about numbers: how old is everyone? Maybe we could, I don't know, get some statistics about age? Some statistics to describe age?
Can you write Python code for it?

# Summary statistics for Age
df['Age'].value_counts() # statistics about age

df['Age'].describe() #statistics about NBA players and their ages

# That's pretty good. Does it work for everything? How about the money?
df.describe() # shows info for all of the numerical data

# EEEK minum weight = 20 lbs -- seems incorrect

df['Ht (In.)'].describe()

#df.columns # look at column names again

df['2013 $'].describe() # this column is string as opposed to int -- therefore it didn't work :()


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
So yes, head and tail work kind of like the terminal commands. That's nice, I guess.
But maybe we're incredibly demanding (which we are) and we want, say, the 6th through the 8th row (which we do). Don't worry (which I know you were), we can do that, too.
Can you write Python code for it?

# Show the 6th through the 8th rows

df[6:9]


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
That might look better with a little more customization. So let's customize it.
Can you write Python code for it?

# Pass in all sorts of stuff!
# Most from http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html
# .range() is a matplotlib thing


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
That's stupid, though, what's an inch even look like? What's 80 inches? I don't have a clue. If only there were some wa to manipulate our data.
Manipulating data
Oh wait there is, HA HA HA.
Can you write Python code for it?

# Take another look at our inches, but only the first few

df['Ht (In.)'].head()

# Divide those inches by 12
df['Ht (In.)'].head()/12 #divides every single value by 12

# Let's divide ALL of them by 12
df['Ht (In.)']/12

# Can we get statistics on those?
height_in_feet = df['Ht (In.)']/12
height_in_feet.describe()

# Let's look at our original data again
df.head()


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
The average basketball player makes 3.8 million dollars and is a little over six and a half feet tall.
But who cares about those guys? I don't care about those guys. They're boring. I want the real rich guys!
Sorting and sub-selecting
Can you write Python code for it?

# This is just the first few guys in the dataset. Can we order it?

# Let's try to sort them


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
Unfortunately because that has dollar signs and commas it's thought of as a string. We'll fix it in a second, but let's try describing one more thing.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/07 - Introduction to Pandas-checkpoint.ipynb".
The first task is:
When you import pandas, you use import pandas as pd. That means instead of typing pandas in your code you'll type pd.
You don't have to, but every other person on the planet will be doing it, so you might as well.
Now we're going to read in a file. Our file is called NBA-Census-10.14.2013.csv because we're sports moguls. pandas can read_ different types of files, so try to figure it out by typing pd.read_ and hitting tab for autocomplete.
Can you write Python code for it?

# We're going to call this df, which means "data frame"
# It isn't in UTF-8 (I saved it from my mac!) so we need to set the encoding
#saved on mac, therefore the encoding needs to be mac_roman!
#it will not open if the encoding is not set :()
df = pd.read_csv('NBA-Census-10.14.2013.csv', encoding='mac_roman')

# encoding, the most common are: mac_roman if saved on a mac, latin-1 if saved on PC or UTF-8
 # 'pd.read_csv?' will give you more info about how read_csv works


I am working on the file ".ipynb_checkpoints/0_obtain_comic_data_j-checkpoint.ipynb".
The first task is:
Web APIでは，パラメータidで「ユニークID」「雑誌巻号ID」あるいは「雑誌コード」を，pageで検索結果の取得ページ番号（1ページあたり100件，デフォルトは1）を指定することができます．ここで，週刊少年ジャンプは「雑誌巻号ID」にJUMPrglを含むため，id=JUMPrglを指定します．また，週刊少年ジャンプの検索結果は合計24ページ（2320件）あるため，pageに1から24を順次指定する必要があります．
詳細はWebAPI仕様をご参照ください．
なお，2017年3月31日より文化庁メディア芸術データベースのURLが変更されたため，WebAPI仕様に記載のリクエストURL（https://mediaarts-db.jp/mg/api/v1/results_magazines）ではなく，新しいURL（https://mediaarts-db.bunka.go.jp/mg/api/v1/results_magazines）を使用する必要があることにご注意ください．
Can you write Python code for it?

magazines = search_magazine()

len(magazines)

magazines[0]

magazines[-1]


I am working on the file ".ipynb_checkpoints/0_obtain_comic_data_j-checkpoint.ipynb".
The first task is:
タイムアウトに柔軟に対応するため，目次情報を一括処理せず，無理やり逐次処理にしています．また，サーバに負荷をかけないよう，sleep()で一時停止していることにご注意ください．
なお，2017年3月31日より文化庁メディア芸術データベースのURLが変更されたため，WebAPI仕様に記載のリクエストURL（https://mediaarts-db.jp/mg/api/v1/magazine）ではなく，新しいURL（https://mediaarts-db.bunka.go.jp/mg/api/v1/magazine）を使用する必要があることにご注意ください．
以下を実行し，wj-api.jsonにデータを保存します．
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/0_obtain_comic_data_j-checkpoint.ipynb".
The first task is:
雑誌巻号情報の取得
以下の関数extract_data()で必要な目次情報を抽出し，save_data()で目次情報を保存します．
Can you write Python code for it?

def extract_data(content):
    """
    contentに含まれる目次情報を取得する関数です．
    - year: 発行年
    - no: 号数
    - title: 作品名
    - author: 著者
    - color: カラーか否か
    - pages: 掲載ページ数
    - start_page: 作品のスタートページ
    - best: 巻頭から数えた掲載順
    - worst: 巻末から数えた掲載順
    """

    # マンガ作品のみ抽出します．
    comics = [comic for comic in content['contents']
             if comic['category']=='マンガ作品']
    data = []
    year = int(content['basics']['date_indication'][:4])

    # 号数が記載されていない場合があるので，例外処理が必要です．
    try:
        no = int(content['basics']['number_indication'])
    except ValueError:
        no = content['basics']['number_indication']

    for comic in comics:
        title= comic['work']
        if not title:
            continue

        # ページ数が記載されていない作品があるので，例外処理が必要です．
        # 特に理由はないですが，無記載の作品は10ページとして処理を進めます．
        try:
            pages = int(comic['work_pages'])
        except ValueError:
            pages = 10

        # 「いぬまるだしっ」等，1週に複数話掲載されている作品に対応するため
        # data中にすでにtitleが含まれる場合は，新規datumとして登録せずに，
        # 既存のdatumのページ数のみ加算します．
        if len(data) > 0 and title in [datum['title'] for datum in data]:
            data[[datum['title'] for datum in
                  data].index(title)]['pages'] += pages
        else:
            data.append({
                'year': year,
                'no': no,
                'title': comic['work'],
                'author': comic['author'],
                'subtitle': comic['subtitle'],
                'color': comic['note'].count('カラー'),
                'pages': int(comic['work_pages']),
                'start_pages': int(comic['start_page'])
            })

    # 企画物のミニマンガを除外するため，合計5ページ以下のdatumはリストから除外します．
    filterd_data = [datum for datum in data if datum['pages'] > 5]
    for n, datum in enumerate(filterd_data):
        datum['best'] = n + 1
        datum['worst'] = len(filterd_data) - n

    return filterd_data


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Derive some statistics about them
Can you write Python code for it?

proportion_women_survived = np.sum(women_onboard) / np.size(women_onboard)
proportion_men_survived   = np.sum(men_onboard)   / np.size(men_onboard)

print 'Proportion of women who survived is %s' % proportion_women_survived
print 'Proportion of men   who survived is %s' % proportion_men_survived


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Filter the whole data, to find statistics for just women, by just placing women_only_stats as a "mask" on my full data -- Use it in place of the '0::' part of the array index.
You can test it by placing it there, and requesting column index [4], and the output should all read 'female' e.g. try typing this:   data[women_only_stats,4]
Can you write Python code for it?

women_onboard = data[women_only_stats, 1].astype(np.float)
men_onboard   = data[men_only_stats,   1].astype(np.float)

print 'Women onboard: %s' % women_onboard.size
print 'Men   onboard: %s' % men_onboard.size
print 'Total onboard: %s' % (women_onboard.size+men_onboard.size)


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Load training data
Can you write Python code for it?

csv_file_object = csv.reader(open('./data/train.csv', 'rb')) 	# Load in the csv file
header          = csv_file_object.next() 						# Skip the fist line as it is a header
data            =[] 											# Create a variable to hold the data

for row in csv_file_object: 							# Skip through each row in the csv file,
    data.append(row[0:]) 								# adding each row to the data variable
data = np.array(data) 									# Then convert from a list to an array.


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Model algorithm: Females survive, Males die
Loop through each passenger in the train file:
 - if passenger is female, set as survived
 - if passenger is male,   set as died
Can you write Python code for it?

for row in test_file_object:									# For each row in test file,
    if row[3] == 'female':										# is it a female, if yes then
        predictions_file_object.writerow([row[0], "1"])			# write the PassengerId, and predict 1
    else:														# or else if male,
        predictions_file_object.writerow([row[0], "0"])			# write the PassengerId, and predict 0.

test_file.close()												# Close out the files.
predictions_file.close()


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Now that I have my indicator that women were much more likely to survive, I am done with the training set.
First Basic Model: Women Survive
Let's read in the test file and write out a simple prediction:
- if female, then model that she survived (1)
- if male, then model that he did not survive (0)
Can you write Python code for it?

test_file        = open('./data/test.csv', 'rb')         # First, read in test.csv
test_file_object = csv.reader(test_file)
header           = test_file_object.next()

header


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Playing w/ the data
I have an array of 12 columns and 891 rows.
I can access any element I want, so the entire first column would be data[0::,0].astype(np.float)
-- This means all of the rows (from start to end), in column 0 I have to add the .astype() command, because when appending the rows, python thought it was a string - so needed to convert
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Proportion of Survivors
Can you write Python code for it?

number_passengers    = np.size(data[0::,1].astype(np.float))
number_survived      = np.sum(data[0::,1].astype(np.float))
proportion_survivors = number_survived / number_passengers
print 'Proportion of Survivors: %s/%s = %s' % (number_passengers \
                                               , number_survived \
                                               , proportion_survivors)


I am working on the file ".ipynb_checkpoints/1-Basic Gender Model-checkpoint.ipynb".
The first task is:
Take a look at the training data
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Analysis-checkpoint.ipynb".
The first task is:
Different features have different ditributions.
Some are subject to Gussain distribution.
b. t-SNE
use TSNE to see the linear separability of the data.
Can you write Python code for it?

model=TSNE(n_components=2,random_state=0) # reduct the dimention to 2 for visualization
np.set_printoptions(suppress=True)
Y=model.fit_transform(lld.feature,lld.label) # the reducted data

plt.scatter(Y[:, 0], Y[:, 1],c=lld.label[:,0],cmap=plt.cm.Spectral)
plt.title('training data')
plt.axis('tight')

print Y.shape


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
'date' should now show up in the available indices...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
...and edges() for edges:
Can you write Python code for it?

coauthor_graph.edges(data=True)[:10]    # [:10] just shows the first ten.
                                        # data=True tells edges() to return details about each edge.


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
...and we can now look up all of the Papers published in 1985:
Can you write Python code for it?

for paper in corpus[('date', 1985)]:
    print paper.date, paper.title


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
All of the Papers in the Corpus are stored by wosid in the indexed_papers attribute. The code cell below shows the first ten Papers with their indexing keys.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Corpus objects
A Corpus is a collection of Papers with superpowers. Each Paper represents one bibliographic record. Most importantly, the Corpus provides a consistent way of indexing bibliographic records. Indexing is important, because it sets the stage for all of the subsequent analyses that we may wish to do with our bibliographic data.
A Corpus behaves like a list of Papers. We can selecte a single Paper like this:
Can you write Python code for it?

corpus[500].__dict__  # [500] gets the 501st Paper, and __dict__ generates a
                      #  key-value representation of the data in the Paper.


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Each Paper should also have a date, journal, and wosid (WoS accession ID). Many will also have dois. Note that we can access the attributes of each Paper using . notation:
Can you write Python code for it?

# corpus[500] gets a Paper, and ``.date`` gets the date attribute.
print 'Date:'.ljust(20), corpus[500].date
print 'Journal:'.ljust(20), corpus[500].journal
print 'WoS accession ID:'.ljust(20), corpus[500].wosid
print 'DOI:'.ljust(20), corpus[500].doi


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Each Paper will also have authors. Tethne represents author names as "tuples" of the form (last, first). Depending on the record, first might be first and middle initials, or first and middle names.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
For networks with anything more than a few nodes, it's hard to visualize what's going on in the iPython environment. So we'll expore the coauthor_graph and visualize it in a network analysis package called Cytoscape.
Cytoscape understands several network file formats. GraphML ((link here)) is probably the most versatile, so we'll use it to export our coauthor graph.
The tethne.writers.graph module has several functions for writing graphs to disk. We'll use to_graphml():
Can you write Python code for it?

from tethne.writers.graph import to_graphml


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Indexing
The most important functionality of the Corpus is indexing. Indexing provides a way of looking up Papers by specific attributes, e.g. by the year in which they were published, or by author.
Each Corpus has a single "primary" index. For WoS data, the wosid field (WoS accession ID) is used by default, since every WoS record has one. You can see which field was used as the primary index by accessing the .index_by attribute of the Corpus.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Introduction to Tethne: Loading Data, part 1
In this notebook we will take our first steps with the Tethne Python package. We'll parse some bibliographic records from the ISI Web of Science, and take a look at the Corpus class and its various features. We'll then use some of the functions in tethne.networks to generate some simple networks from our bibliographic dataset.
Methods in Digital & Computational Humanities
This notebook is part of a cluster of learning resources developed by the Laubichler Lab and the Digital Innovation Group at Arizona State University as part of an initiative for digital and computational humanities (d+cH). For more information, see our evolving online methods course at https://diging.atlassian.net/wiki/display/DCH.
Getting Help
Development of the Tethne project is led by Erick Peirson. To get help, first check our issue tracking system on GitHub. There, you can search for questions and problems reported by other users, or ask a question of your own. You can also reach Erick via e-mail at erick.peirson@asu.edu.
Documentation & Tutorials
Additional documentation and tutorials for the Tethne Python package are available at ....
Using this notebook
This is an interactive Python notebook. Most of the content is just marked-down text, like this paragraph, that provides expository on some aspect of the Tethne package. Some of the cells are "code" cells, which look like this:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Simple Networks
One of the core features of Tethne is a set of functions for building networks from bibliographic datasets. These functions are located in the tethne.networks subpackage. In this section, we'll build a coauthorship network.
The first step is to import the networks subpackage.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Tethne uses a package called NetworkX to build networks. All of the network-building functions return NetworkX Graph objects. We can see how large our network is using the order() and size() methods:
Can you write Python code for it?

print coauthor_graph.order()    # Number of nodes.
print coauthor_graph.size()     # Number of edges.


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
There are several things to notice in the output above. First, each Paper should (generally) have a title:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
To parse data from a WoS dataset, use the read method. Each module in the tethne.readers subpackage should have a read method.
read can parse a single data file, or a directory full of data files, and returns a Corpus object. Just pass it a string containing the path to your data. First, try parsing a single WoS field-tagged data file.
Can you write Python code for it?

corpus = wos.read('/Users/erickpeirson/Dropbox/HSS ThatCamp Workshop/sample_data/wos/savedrecs.txt')


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
Unlike other bibliographic datasets, WoS data contain the cited references of each Paper. Each cited reference is represented as a Paper:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
We should have quite a few more records this time:
Can you write Python code for it?

print 'Loaded %i records!' % len(corpus)


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
You can execute the code in a code cell by clicking on it and pressing Shift-Enter on your keyboard, or by clicking the right-arrow "Run" button in the toolbar at the top of the page. The cell below will automatically be selected, so you can run many cells in quick succession by repeatedly pressing Shift-Enter (or the "Run" button). It's a good idea to run all of the code cells in order, from the top of the tutorial, since many commands later in the tutorial will depend on earlier ones.
Play!!
As we work through the notebook, you'll need to modify certain values depending on where your data is located. You should also experiment! Try changing the parameters in the functions demonstrated below, and re-run the code-cell to see the result. That's what's great about iPython notebooks: you can play around with specific chunks of code without having to re-run the entire script.
Getting Bibliographic Data from the ISI Web of Science
The ISI Web of Science is a proprietary database owned by Thompson Reuters. It is one of the oldest and most comprehensive scientific bibliographic databases in existance. If you are affiliated with an academic institution, you may have access to this database via an institutional license.
For the purpose of this tutorial, you can download a practice dataset from (insert link to dataset). Move the downloaded zip to a place where you can find it, and uncompress its contents. You'll need the full path to the uncompressed dataset.
Perform a search for literature of interest using the interface provided.
Your search criteria will be informed by the objectives of your research
project. If you are attempting to characterize the development of a research
field, for example, you should choose terms that pick out that field as uniquely
as possible (consider using the Publication Name search field). You can also
pick out literatures originating from particular institutions, by using the
Organization-Enhanced search field.

Note also that you can restrict your research to one of three indexes in...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/1. Working with data from the Web of Science-checkpoint.ipynb".
The first task is:
to_graphml() accepts two arguments: the graph itself, and a string with the path to the output file (that will be created). In the example below, I just put the graph on my desktop.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/10_introduction_to_artificial_neural_networks-checkpoint.ipynb".
The first task is:
Activation functions
Can you write Python code for it?

def logit(z):
    return 1 / (1 + np.exp(-z))

def relu(z):
    return np.maximum(0, z)

def derivative(f, z, eps=0.000001):
    return (f(z + eps) - f(z - eps))/(2 * eps)

z = np.linspace(-5, 5, 200)

plt.figure(figsize=(11,4))

plt.subplot(121)
plt.plot(z, np.sign(z), "r-", linewidth=2, label="Step")
plt.plot(z, logit(z), "g--", linewidth=2, label="Logit")
plt.plot(z, np.tanh(z), "b-", linewidth=2, label="Tanh")
plt.plot(z, relu(z), "m-.", linewidth=2, label="ReLU")
plt.grid(True)
plt.legend(loc="center right", fontsize=14)
plt.title("Activation functions", fontsize=14)
plt.axis([-5, 5, -1.2, 1.2])

plt.subplot(122)
plt.plot(z, derivative(np.sign, z), "r-", linewidth=2, label="Step")
plt.plot(0, 0, "ro", markersize=5)
plt.plot(0, 0, "rx", markersize=10)
plt.plot(z, derivative(logit, z), "g--", linewidth=2, label="Logit")
plt.plot(z, derivative(np.tanh, z), "b-", linewidth=2, label="Tanh")
plt.plot(z, derivative(relu, z), "m-.", linewidth=2, label="ReLU")
plt.grid(True)
#plt.legend(loc="center right", fontsize=14)
plt.title("Derivatives", fontsize=14)
plt.axis([-5, 5, -0.2, 1.2])

save_fig("activation_functions_plot")
plt.show()

def heaviside(z):
    return (z >= 0).astype(z.dtype)

def sigmoid(z):
    return 1/(1+np.exp(-z))

def mlp_xor(x1, x2, activation=heaviside):
    return activation(-activation(x1 + x2 - 1.5) + activation(x1 + x2 - 0.5) - 0.5)

x1s = np.linspace(-0.2, 1.2, 100)
x2s = np.linspace(-0.2, 1.2, 100)
x1, x2 = np.meshgrid(x1s, x2s)

z1 = mlp_xor(x1, x2, activation=heaviside)
z2 = mlp_xor(x1, x2, activation=sigmoid)

plt.figure(figsize=(10,4))

plt.subplot(121)
plt.contourf(x1, x2, z1)
plt.plot([0, 1], [0, 1], "gs", markersize=20)
plt.plot([0, 1], [1, 0], "y^", markersize=20)
plt.title("Activation function: heaviside", fontsize=14)
plt.grid(True)

plt.subplot(122)
plt.contourf(x1, x2, z2)
plt.plot([0, 1], [0, 1], "gs", markersize=20)
plt.plot([0, 1], [1, 0], "y^", markersize=20)
plt.title("Activation function: sigmoid", fontsize=14)
plt.grid(True)


I am working on the file ".ipynb_checkpoints/10_introduction_to_artificial_neural_networks-checkpoint.ipynb".
The first task is:
FNN for MNIST
using tf.learn
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("/tmp/data/")

X_train = mnist.train.images
X_test = mnist.test.images
y_train = mnist.train.labels.astype("int")
y_test = mnist.test.labels.astype("int")

import tensorflow as tf

config = tf.contrib.learn.RunConfig(tf_random_seed=42) # not shown in the config

feature_cols = tf.contrib.learn.infer_real_valued_columns_from_input(X_train)
dnn_clf = tf.contrib.learn.DNNClassifier(hidden_units=[300,100], n_classes=10,
                                         feature_columns=feature_cols, config=config)
dnn_clf = tf.contrib.learn.SKCompat(dnn_clf) # if TensorFlow >= 1.1
dnn_clf.fit(X_train, y_train, batch_size=50, steps=40000)

from sklearn.metrics import accuracy_score

y_pred = dnn_clf.predict(X_test)
accuracy_score(y_test, y_pred['classes'])

from sklearn.metrics import log_loss

y_pred_proba = y_pred['probabilities']
log_loss(y_test, y_pred_proba)


I am working on the file ".ipynb_checkpoints/10_introduction_to_artificial_neural_networks-checkpoint.ipynb".
The first task is:
Now we need to define the directory to write the TensorBoard logs to:
Can you write Python code for it?

from datetime import datetime

def log_dir(prefix=""):
    now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
    root_logdir = "tf_logs"
    if prefix:
        prefix += "-"
    name = prefix + "run-" + now
    return "{}/{}/".format(root_logdir, name)

logdir = log_dir("mnist_dnn")


I am working on the file ".ipynb_checkpoints/10_introduction_to_artificial_neural_networks-checkpoint.ipynb".
The first task is:
Perceptrons
Can you write Python code for it?

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import Perceptron

iris = load_iris()
X = iris.data[:, (2, 3)]  # petal length, petal width
y = (iris.target == 0).astype(np.int)

per_clf = Perceptron(random_state=42)
per_clf.fit(X, y)

y_pred = per_clf.predict([[2, 0.5]])

y_pred

a = -per_clf.coef_[0][0] / per_clf.coef_[0][1]
b = -per_clf.intercept_ / per_clf.coef_[0][1]

axes = [0, 5, 0, 2]

x0, x1 = np.meshgrid(
        np.linspace(axes[0], axes[1], 500).reshape(-1, 1),
        np.linspace(axes[2], axes[3], 200).reshape(-1, 1),
    )
X_new = np.c_[x0.ravel(), x1.ravel()]
y_predict = per_clf.predict(X_new)
zz = y_predict.reshape(x0.shape)

plt.figure(figsize=(10, 4))
plt.plot(X[y==0, 0], X[y==0, 1], "bs", label="Not Iris-Setosa")
plt.plot(X[y==1, 0], X[y==1, 1], "yo", label="Iris-Setosa")

plt.plot([axes[0], axes[1]], [a * axes[0] + b, a * axes[1] + b], "k-", linewidth=3)
from matplotlib.colors import ListedColormap
custom_cmap = ListedColormap(['#9898ff', '#fafab0'])

plt.contourf(x0, x1, zz, cmap=custom_cmap, linewidth=5)
plt.xlabel("Petal length", fontsize=14)
plt.ylabel("Petal width", fontsize=14)
plt.legend(loc="lower right", fontsize=14)
plt.axis(axes)

save_fig("perceptron_iris_plot")
plt.show()


I am working on the file ".ipynb_checkpoints/1_1_dataset_more_features-checkpoint.ipynb".
The first task is:
Let's find the # of investments made by each company
Can you write Python code for it?

investments = pd.read_csv('data/investments.csv')
investments = investments[investments['investor_country_code'] == 'USA']
investments[:3]

#acquirer_permalink
#rounds_agg = df_rounds.groupby(['company_permalink', 'funding_round_type'])['raised_amount_usd'].agg({'amount': [ pd.Series.sum, pd.Series.count]})
number_of_investments = investments.groupby(['investor_permalink'])['investor_permalink'].agg({'amount': [ pd.Series.count]}).reset_index()
number_of_investments.columns = number_of_investments.columns.droplevel()
number_of_investments.columns = ['permalink', 'number_of_investments']
number_of_investments = number_of_investments.set_index('permalink')
number_of_investments[:3]

#Number of different companies in which each company have invested in
number_of_unique_investments = investments.groupby(['investor_permalink'])['company_permalink'].agg({'amount': [ pd.Series.nunique]}).reset_index()
number_of_unique_investments.columns = number_of_unique_investments.columns.droplevel()
number_of_unique_investments.columns = ['permalink', 'number_of_unique_investments']
number_of_unique_investments = number_of_unique_investments.set_index('permalink')
number_of_unique_investments[:3]

number_of_investors_per_round = investments.groupby(['company_permalink', 'funding_round_permalink'])['investor_permalink'].agg({'investor_permalink': [ pd.Series.count]}).reset_index()
number_of_investors_per_round.columns = number_of_investors_per_round.columns.droplevel(0)
number_of_investors_per_round.columns = ['company_permalink', 'funding_round_permalink', 'count']
number_of_investors_per_round = number_of_investors_per_round.groupby(['company_permalink']).agg({'count': [ pd.Series.mean]}).reset_index()
number_of_investors_per_round.columns = number_of_investors_per_round.columns.droplevel(0)
number_of_investors_per_round.columns = ['company_permalink', 'number_of_investors_per_round']
number_of_investors_per_round = number_of_investors_per_round.set_index('company_permalink')
number_of_investors_per_round[:3]

from numpy...


I am working on the file ".ipynb_checkpoints/1_dataset_creation-checkpoint.ipynb".
The first task is:
Comparing investments.csv and rounds.csv
Can you write Python code for it?

#Let's start by comparing and understanding the difference between investments.csv and rounds.csv
df_investments = pd.read_csv('data/investments.csv')
df_investments = df_investments[df_investments['company_country_code'] == 'USA']

df_rounds = pd.read_csv('data/rounds.csv')
df_rounds = df_rounds[df_rounds['company_country_code'] == 'USA']

#companies_USA[companies_USA['permalink'] == '/organization/0xdata']
#df_investments[df_investments['company_permalink'] == '/organization/0xdata' ]
#df_rounds[df_rounds['company_permalink'] == '/organization/0xdata' ]


I am working on the file ".ipynb_checkpoints/1_dataset_creation-checkpoint.ipynb".
The first task is:
Converting categories to dummy variables (selecting top 50)
Can you write Python code for it?

from operator import methodcaller
#Let's analyze category_list and probably expand it as dummy variables

#get a unique list of the categories
categories = list(companies_USA['category_list'].astype('str').unique())

#split each categori by |
categories = map(methodcaller("split", "|"), categories)

#flatten the list of sub categories
categories = [item for sublist in categories for item in sublist]

#total of 60k different categories
#categories
len(categories)

#We'll need to select the most important categories (that appears most of the times, and use Other for the rest)
companies_series = companies_USA['category_list'].astype('str')
categories_splitted_count = companies_series.str.split('|').apply(lambda x: pd.Series(x).value_counts()).sum()

#dummies

dummies = companies_series.str.get_dummies(sep='|')

########### Count of categories splitted first 50)###########
top50categories = list(categories_splitted_count.sort_values(ascending=False).index[:50])

##### Create a dataframe with the 50 top categories to be concatenated later to the complete dataframe
categories_df = dummies[top50categories]
categories_df = categories_df.add_prefix('Category_')


I am working on the file ".ipynb_checkpoints/1_dataset_creation-checkpoint.ipynb".
The first task is:
Let's import the data
Can you write Python code for it?

companies = pd.read_csv('data/companies.csv')

#Having a look to the companies data structure
companies[:3]

#Let's first remove non USA companies, since they usually have a lot of missing data
companies_USA = companies[companies['country_code'] == 'USA']

#Check if there are any missing data for state_code in USA based companies
companies_USA['state_code'].unique()
#companies_USA['state_code'].value_counts()
# No nan values for state_code

#Let's maintain region and city in the dataset but probably they are not gonna be used
#companies_USA['city'].value_counts()


I am working on the file ".ipynb_checkpoints/1_dataset_creation-checkpoint.ipynb".
The first task is:
The difference between investments and rounds is that investments is providing the information of where the money came from. Investments contains information about which investors paid each round. While rounds is grouping and totalizing the information by round.
Analyzing rounds.csv
Can you write Python code for it?

#df_rounds

#Prepare an aggregated rounds dataframe grouped by company and funding type
rounds_agg = df_rounds.groupby(['company_permalink', 'funding_round_type'])['raised_amount_usd'].agg({'amount': [ pd.Series.sum, pd.Series.count]})

#Get available unique funding types
funding_types = list(rounds_agg.index.levels[1])
funding_types

#Prepare the dataframe where all the dummy features for each funding type will be added (number of rounds and total sum for each type)
rounds_df = companies_USA[['permalink']]
rounds_df = rounds_df.rename(columns = {'permalink':'company_permalink'})

#Iterate over each kind of funding type, and add two new features for each into the dataframe
def add_dummy_for_funding_type(df, aggr_rounds, funding_type):
    funding_df = aggr_rounds.iloc[aggr_rounds.index.get_level_values('funding_round_type') == funding_type].reset_index()
    funding_df.columns = funding_df.columns.droplevel()
    funding_df.columns = ['company_permalink', funding_type, funding_type+'_funding_total_usd', funding_type+'_funding_rounds']
    funding_df = funding_df.drop(funding_type,1)

    new_df = pd.merge(df, funding_df, on='company_permalink', how='left')
    new_df = new_df.fillna(0)
    return new_df

#rounds_agg was generated a few steps above
for funding_type in funding_types:
    rounds_df = add_dummy_for_funding_type(rounds_df, rounds_agg, funding_type)

#remove the company_permalink variable, since it's already available in the companies dataframe
rounds_df = rounds_df.drop('company_permalink', 1)

#set rounds_df to have the same index of the other dataframes
rounds_df.index = companies_USA.index

rounds_df[:3]


I am working on the file ".ipynb_checkpoints/1_dataset_creation-checkpoint.ipynb".
The first task is:
Write resulting dataframe to csv file
Can you write Python code for it?

startups_df.index = list(startups_df['permalink'])
startups_df = startups_df.drop('permalink', 1)

startups_df.to_csv('data/startups_1.csv')
#startups_df


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
2.5. Exploration
Frequently you will work with objects with which you are only partially familiar. Here are a few techniques that are helpful for learning about an object and its attributes and methods.
dir()
dir will attempt to display all of the attributes of an object.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
A function is also an object, for which there is a function type.
Can you write Python code for it?

def myfunction():
    return True

type(myfunction)


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
But all is not lost! If I transform it into an int first, I'm golden.
Can you write Python code for it?

print type(mystring)    # mystring is a str
myint = int(mystring)   # But I can turn it into an int!

print type(myint)       # Yup, it's an int.
print myint + 3.4       # And now I can math!


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
In IPython, you can get information about a function or method by writing everything up to the open parens, and then pressing shift + tab. For example, try writing the following in the code-cell below:
python
mystring.count(
and then press shift + tab on your keyboard. It should open a little box with information about the method.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
Just like other functions, instance methods can accept arguments. For example, I can count the number of 'Y' characters in my string object using the count() method:
Can you write Python code for it?

print mystring.count('Y')    # How many Y's are there in my string?


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
Let's unpack the code-block above.
On the first line, I declare that I am about to define a new class (or type) called Car.
python
class Car:
I then define two attributes, and give them default values. When we create a new car, it will have these attributes, and those attributes will have the specified default values.
python
driver = None              # By default, there is no driver.
number_of_passengers = 0   # By default, the car is empty.
Let's try instantiating our Car class. In other words, let's create a new car.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
Sure enough, mycar has an attribute called driver, and another attribute called number_of_passengers.
Can you write Python code for it?

print mycar.driver
print mycar.number_of_passengers


I am working on the file ".ipynb_checkpoints/2. Objects and types-checkpoint.ipynb".
The first task is:
This looks a lot like the function declarations that you saw earlier, but with two important differences:

They are part of the class definition for Car, so they are indented one extra level.
Their first (and, in this case, only) parameter is something called self.

self is how we refer to the instance of a class. When we create an instance of car, and then use that car instance's number_of_passengers method, the self in that function is the instance of the car that we're interacting with.
So when I use the count_passengers method, and it returns self.number_of_passengers, it is retrieving the number_of_passengers attribute from the current car instance.
Can you write Python code for it?

mycar = Car()    # Putting the parens () after the class name instantiates the class.
mycar.driver = 'John'
mycar.number_of_passengers += 5

print mycar.count_passengers()


I am working on the file ".ipynb_checkpoints/2.4 Transfer Learning & Fine-Tuning-checkpoint.ipynb".
The first task is:
Dataset Preparation
Can you write Python code for it?

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# create two datasets one with digits below 5 and one with 5 and above
X_train_lt5 = X_train[y_train < 5]
y_train_lt5 = y_train[y_train < 5]
X_test_lt5 = X_test[y_test < 5]
y_test_lt5 = y_test[y_test < 5]

X_train_gte5 = X_train[y_train >= 5]
y_train_gte5 = y_train[y_train >= 5] - 5  # make classes start at 0 for
X_test_gte5 = X_test[y_test >= 5]         # np_utils.to_categorical
y_test_gte5 = y_test[y_test >= 5] - 5

# define two groups of layers: feature (convolutions) and classification (dense)
feature_layers = [
    Convolution2D(nb_filters, kernel_size, kernel_size,
                  border_mode='valid',
                  input_shape=input_shape),
    Activation('relu'),
    Convolution2D(nb_filters, kernel_size, kernel_size),
    Activation('relu'),
    MaxPooling2D(pool_size=(pool_size, pool_size)),
    Dropout(0.25),
    Flatten(),
]
classification_layers = [
    Dense(128),
    Activation('relu'),
    Dropout(0.5),
    Dense(nb_classes),
    Activation('softmax')
]

# create complete model
model = Sequential(feature_layers + classification_layers)

# train model for 5-digit classification [0..4]
train_model(model,
            (X_train_lt5, y_train_lt5),
            (X_test_lt5, y_test_lt5), nb_classes)

# freeze feature layers and rebuild model
for l in feature_layers:
    l.trainable = False

# transfer: train dense layers for new classification task [5..9]
train_model(model,
            (X_train_gte5, y_train_gte5),
            (X_test_gte5, y_test_gte5), nb_classes)


I am working on the file ".ipynb_checkpoints/2.4 Transfer Learning & Fine-Tuning-checkpoint.ipynb".
The first task is:
Your Turn
Try to Fine Tune a VGG16 Network
Can you write Python code for it?

from keras.applications import VGG16
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
from keras.layers import Input, Flatten, Dense
from keras.models import Model
import numpy as np

#Get back the convolutional part of a VGG network trained on ImageNet
model_vgg16_conv = VGG16(weights='imagenet', include_top=False)
model_vgg16_conv.summary()

#Create your own input format (here 3x200x200)
inp = Input(shape=(48,48,3),name = 'image_input')

#Use the generated model
output_vgg16_conv = model_vgg16_conv(inp)

#Add the fully-connected layers
x = Flatten(name='flatten')(output_vgg16_conv)
x = Dense(4096, activation='relu', name='fc1')(x)
x = Dense(4096, activation='relu', name='fc2')(x)
x = Dense(5, activation='softmax', name='predictions')(x)

#Create your own model
my_model = Model(input=inp, output=x)

#In the summary, weights and layers from VGG part will be hidden, but they will be fit during the training
my_model.summary()


I am working on the file ".ipynb_checkpoints/2.4 Transfer Learning & Fine-Tuning-checkpoint.ipynb".
The first task is:
```python
...
...
Plugging new Layers
model.add(Dense(768, activation='sigmoid'))
model.add(Dropout(0.0))
model.add(Dense(768, activation='sigmoid'))
model.add(Dropout(0.0))
model.add(Dense(n_labels, activation='softmax'))

```
Can you write Python code for it?

import scipy
new_shape = (48,48)
X_train_new = np.empty(shape=(X_train_gte5.shape[0],)+(48,48,3))

for idx in xrange(X_train_gte5.shape[0]):
    X_train_new[idx] = np.resize(scipy.misc.imresize(X_train_gte5[idx], (new_shape)), (48, 48, 3))
    X_train_new[idx] = np.resize(X_train_new[idx], (48, 48, 3))

#X_train_new = np.expand_dims(X_train_new, axis=-1)

print X_train_new.shape

X_train_new = X_train_new.astype('float32')
X_train_new /= 255

print('X_train shape:', X_train_new.shape)
print(X_train_new.shape[0], 'train samples')
print(X_train_new.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train_gte5, nb_classes)
Y_test = np_utils.to_categorical(y_test_gte5, nb_classes)

print y_train.shape

my_model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy'])

my_model.fit(X_train_new, Y_train,
              batch_size=batch_size, nb_epoch=nb_epoch,
              verbose=1)

#print('Training time: %s' % (now() - t))
#score = my_model.evaluate(X_test, Y_test, verbose=0)
#print('Test score:', score[0])
#print('Test accuracy:', score[1])

#train_model(my_model,
#            (X_train_new, y_train_gte5),
#            (X_test_gte5, y_test_gte5), nb_classes)


I am working on the file ".ipynb_checkpoints/2_dataset_preparation-checkpoint.ipynb".
The first task is:
Analzying types of status
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/2_dataset_preparation-checkpoint.ipynb".
The first task is:
Droping some features
We'll drop homepage_url, category_list, region, city, country_code
We'll also move status to the end of the dataframe
Can you write Python code for it?

#drop features
startups_dropped_features = startups.drop(['name','homepage_url', 'category_list', 'region', 'city', 'country_code'], 1)

#move status to the end
cols = list(startups_dropped_features)
cols.append(cols.pop(cols.index('status')))
startups_dropped_features = startups_dropped_features.ix[:, cols]

startups_dropped_features[:3]


I am working on the file ".ipynb_checkpoints/2_dataset_preparation-checkpoint.ipynb".
The first task is:
Normalizing numeric variables
Can you write Python code for it?

from sklearn import preprocessing
min_max_scaler = preprocessing.MinMaxScaler()

startups_normalized = startups_dropped_features.copy()

#Convert '-' to zeros in funding_total_usd
startups_normalized['funding_total_usd'] = startups_normalized['funding_total_usd'].replace('-', 0)

columns_to_scale = list(startups_normalized.filter(regex=(".*(funding_rounds|funding_total_usd)|(number_of|avg_).*")).columns)
startups_normalized[columns_to_scale] = min_max_scaler.fit_transform(startups_normalized[columns_to_scale])
startups_normalized[:3]


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
Frequently you will need to perform a task some specific number of times, but not necessarily because you have an iterator of a particular size. For example, suppose I want to calculate $6!$. I know that I will need to perform four multiplication steps ($65432$). I can use the function xrange to create an iterator over a specified range of integers. For example:
Can you write Python code for it?

factorial = 6

for i in xrange(1, factorial - 1):    # Yields the ints 1, 2, 3, and 4.
    # ``*=`` means "Multiply this value by the value on the right, and store the result in the same variable.
    factorial *= factorial - i    # The same as: factorial = factorial * (factorial - i)

print factorial


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
If I were to win the Powerball AND my friend could come to the LAN party, it would be a pretty great day. I can check for both of those things in one statement, using the and operator.
Can you write Python code for it?

x = random.random()    # Come on, lucky numbers!

if myfriend in invitees and x < 1./292000000:
    print 'Best day ever!'
else:
    print 'meh.'


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
Notice that I threw in some parentheses to group my evaluations. In the example above, the statement
python
if (myfriend in invitees and x < 1./292000000) or is_there_a_puppy:
evaluates as
python
if (False and False) or True:
which gets further simplified to
python
if False or True:
which is just
python
if True:
3.2. elif
Quite often I want to make a decision that has more than two potential outcomes. Sure, if it's sunny outside I'm down for sandals, otherwise I'd rather wear shoes. But if it's a blizzard, I'm going for snow boots. Consider the elif operator (short for "else if"):
Can you write Python code for it?

weather = 'blizzard'
footwear = None    # I know that I'm wearing something, but I'm not sure what yet.

if weather == 'sunny':      # ``==`` means "equal to".
    footwear = 'sandals'    # ``=`` means "assign the expression on the right to the variable on the left."
elif weather == 'blizzard':
    footwear = 'snow boots'
else:
    footwear = 'shoes'

print footwear


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
Now I need to turn those interaction counts into interaction rates, expressed in interactions per minute. I can iterate over the list of interaction counts, perform my math, and store the result in a new list. Notice: my counts are ints, so in order to get a precise result during division I'll need to coerce them into floats.
Can you write Python code for it?

interaction_frequencies = []    # Make a new, empty list.
for count in interactions:      # Each iteration, I get the next item, and I refer to it as ``count``.
    frequency = float(count)/20.    # Interactions per minute.
    interaction_frequencies.append(frequency)    # ``append()`` adds the value to the end of the list.

print interaction_frequencies


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
Right, pretty unlucky. But I could get over my disappointment if I saw a puppy. So I'll throw in an or operator.
Can you write Python code for it?

x = random.random()    # I gamble for the thrill.
is_there_a_puppy = True

if (myfriend in invitees and x < 1./292000000) or is_there_a_puppy:
    print "This was much more likely, and I've adjusted my expectations."
else:
    print ':-|'


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
Strings are also iterators! I can get one character at a time:
Can you write Python code for it?

for character in 'this string':
    print character


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
You can use as many elifs as you like!
Can you write Python code for it?

weather = 'cows'
footwear = None

if weather == 'sunny':
    footwear = 'sandals'
elif weather == 'blizzard':
    footwear = 'snow boots'
elif weather == 'rain':
    footwear = 'galoshes'
elif weather == 'cows':
    footwear = 'run away!'
else:
    footwear = 'shoes'

print footwear


I am working on the file ".ipynb_checkpoints/3. Flow control. if, elif, else, and friends-checkpoint.ipynb".
The first task is:
if says, "if the following statement is True*, then do the following...". Don't foget the colon (:) afterwards, too. The part after the if statement, print 'I won...., is indented, so that Python knows which part should be conditionally executed.
else says, "otherwise, do this...". Don't forget the colon! else is at the same level of indentation as the if statement, so that Python knows they go together. The part afterward, print ':-(', is indented, so that Python knows it's the part of the code that will get run in the "otherwise" case.
* In Python, the statement need only by "truthy" to be considered True. That is, it could be True, not 0 (or 0.0), or not an empty string.
There are all kinds of comparative operators. For a complete list, see this page.
Here's an example of how to check whether or not a particular item is contained within a list. Say, for example, that I have a list of invitees for a super-secret-underground LAN party, and I want to know if my friend is on the list.
Can you write Python code for it?

invitees = ['Jack', 'Jane', 'Jim', 'Jill', 'Joseph', 'Jay', 'Josephus']
myfriend = 'Greg'

if myfriend in invitees:
    print 'w00t!'
else:
    print 'l4me'


I am working on the file ".ipynb_checkpoints/3_data_visualization-checkpoint.ipynb".
The first task is:
(135.000 likes) Ragazzi, da non credere! Ascoltate e divulgate.Sabato e domenica tutti in piazza, vieni a firmare. (nel video vengono intervistate due minorenni rom che si vantavano di rubare, video scoperto poi essere un falso)
"Una mamma di 41 anni, separata e con due figli, si è impiccata vicino a Bologna. Le avevano staccato il gas, e per luglio rischiava lo sfratto.Una preghiera per questa mamma, un abbraccio ai suoi due cuccioli di 10 e 11 anni che non lasceremo soli, e tanta rabbia. Stato italiano, dove sei?",
Da FARE SUBITO.Sostegno militare alla Russia per annientare l’ISIS, controllo delle frontiere, blocco degli sbarchi ed espulsione dei clandestini, verifica a tappeto di tutte le occupazioni abusive nei nostri quartieri popolari, da Milano a Palermo. Ci hanno dichiarato GUERRA. E alla guerra non si risponde con le chiacchiere di Renzi e dell’inutile Alfano!",
Can you write Python code for it?

df_post_16 = df_posts['20160101':'20161231']
tp_16 = df_post_16.plot()
fig_posts_16 = tp_16.get_figure()
fig_posts_16.tight_layout()
fig_posts_16.savefig(os.path.join(dir_out,'posts_2016.png'), format='png', dpi=300)


I am working on the file ".ipynb_checkpoints/3_data_visualization-checkpoint.ipynb".
The first task is:
1. Confronto Salvini, Renzi e M5S
Can you write Python code for it?

# Dimensione Temporale -> ANNO
df_posts_m5s['Post_Date'] = df_posts_m5s['Post_Date'].str[:4]
df_posts_salvini['Post_Date'] = df_posts_m5s['Post_Date'].str[:4]
df_posts_renzi['Post_Date'] = df_posts_m5s['Post_Date'].str[:4]

df_posts_m5s = df_posts_m5s.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})
df_posts_salvini = df_posts_salvini.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})
df_posts_renzi = df_posts_renzi.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})

df_posts_m5s.rename(columns={'ID': 'Posts_M5S', 'Likes': 'Likes_M5S'}, inplace=True)
df_posts_m5s = df_posts_m5s.set_index(['Post_Date'])
df_posts_m5s.head(2)

df_posts_renzi.rename(columns={'ID': 'Posts_Renzi', 'Likes': 'Likes_Renzi'}, inplace=True)
df_posts_renzi = df_posts_renzi.set_index(['Post_Date'])
df_posts_renzi.head(2)

df_posts_salvini.rename(columns={'ID': 'Posts_Salvini', 'Likes': 'Likes_Salvini'}, inplace=True)
df_posts_salvini = df_posts_salvini.set_index(['Post_Date'])
df_posts_salvini.head(2)

# Numero Posts
result_post = pd.concat([df_posts_renzi, df_posts_salvini, df_posts_m5s], axis=1)
del result_post['Likes_Renzi']
del result_post['Likes_Salvini']
del result_post['Likes_M5S']
result_post.rename(columns={'Posts_Renzi': 'Renzi', 'Posts_Salvini': 'Salvini', 'Posts_M5S': 'M5S'}, inplace=True)

result_post.plot(
        kind='bar'
)

result_post.to_csv(os.path.join(dir_out,r'Distr_Posts.csv'),header=True, index=True)

# Numero Likes
result_likes = pd.concat([df_posts_renzi, df_posts_salvini, df_posts_m5s], axis=1)
del result_likes['Posts_Renzi']
del result_likes['Posts_Salvini']
del result_likes['Posts_M5S']
result_likes.rename(columns={'Likes_Renzi': 'Renzi', 'Likes_Salvini': 'Salvini', 'Likes_M5S': 'M5S'}, inplace=True)

result_likes.plot(
        kind='bar'
)

result_likes.to_csv(os.path.join(dir_out,r'Distr_Likes.csv'),header=True, index=True)


I am working on the file ".ipynb_checkpoints/3_data_visualization-checkpoint.ipynb".
The first task is:
2. Dettaglio Salvini
Can you write Python code for it?

# Dataset Salvini
df_filename = r'df_posts_likes_salvini.pkl'
df_fullpath = os.path.join(dir_df, df_filename)
df_posts = pd.read_pickle(df_fullpath)

# Estraggo la Data da Str
df_posts['Post_Date'] = df_posts['Post_Date'].str[:10]

# Converto in Date
df_posts['Post_Date'] = pd.to_datetime(df_posts['Post_Date'])

# Ordino per Data
df_posts = df_posts.sort_values(by='Post_Date')

# Mi tengo DS totale per Data per analisi successive
df_posts_dett = df_posts
df_posts_dett = df_posts_dett.set_index(['Post_Date'])

# Raggruppo per Data
df_posts = df_posts.groupby('Post_Date',as_index=False).agg({'ID':'count', 'Likes': 'sum'})

# Elimino i le date per cui non ho post con likes (privacy ?)
df_posts = df_posts[np.isfinite(df_posts['Likes'])]

# Setto indice la Data
df_posts = df_posts.set_index(['Post_Date'])

# Lavoro con TimeSeries, raggruppo tutto per Anno/Mese (la data era per giorno)
df_posts = df_posts.groupby(pd.TimeGrouper("M")).sum()

# Elimino Numero di Posts
del df_posts['ID']

# Ok, i numeri tornano dopo le elaborazioni
df_posts['Likes'].sum()


I am working on the file ".ipynb_checkpoints/3_data_visualization-checkpoint.ipynb".
The first task is:
2. Focus su Anni 2014, 2015, 2016
Can you write Python code for it?

df_post_14 = df_posts['20140101':'20141231']
tp_14 = df_post_14.plot()
fig_posts_14 = tp_14.get_figure()
fig_posts_14.tight_layout()
fig_posts_14.savefig(os.path.join(dir_out,'posts_2014.png'), format='png', dpi=300)

# Dettaglio 2014
df_posts_dett['20140101':'20141231'].sort_values(by=['Likes'],ascending=False).head(1)
# Analizzo gli ID direttamente dall API Graph Tool di Facebook


I am working on the file ".ipynb_checkpoints/3_data_visualization-checkpoint.ipynb".
The first task is:
2.1. Distribuzione Totale dei Likes ai Post di Salvini
Can you write Python code for it?

df_posts.sort_values(by='Likes').head(5)

# Costruisco il Grafico, l'obiettivo è analizzare i picchi e capire a quale evento è collegato
tp = df_posts.plot(
        marker='o',
        markersize=7,
        # x-axis da 0 a 84
        markevery=[58,60,63,65,70])
tp.set_xlabel("Data del Post")
vals = tp.get_yticks()
tp.set_yticklabels(['{:,.0f}'.format(x) for x in vals])

fig_posts = tp.get_figure()
fig_posts.tight_layout()
fig_posts.savefig(os.path.join(dir_out,'Distr_Posts_Salvini.png'), format='png', dpi=300)


I am working on the file ".ipynb_checkpoints/3_data_visualization-checkpoint.ipynb".
The first task is:
Principali post tra Ottobre e Novembre

(90.000 likes) Uno STUPRATORE tunisino di 28 anni, già in galera per violenza sessuale, è evaso dal carcere di Pordenone e ha violentato una ragazza di 28 anni. È stato arrestato. Fossi ministro, applicherei (come già sperimentato in numerosi Paesi europei) la CASTRAZIONE CHIMICA e poi lo rimanderei in Tunisia. Che dite?",
Can you write Python code for it?

df_post_15 = df_posts['20150101':'20151231']
tp_15 = df_post_15.plot()
fig_posts_15 = tp_15.get_figure()
fig_posts_15.tight_layout()
fig_posts_15.savefig(os.path.join(dir_out,'posts_2015.png'), format='png', dpi=300)

# Controllo 2015
df_posts_dett['20150101':'20151231'].sort_values(by=['Likes'],ascending=False).head(3)
# Analizzo gli ID direttamente dall API Graph Tool di Facebook


I am working on the file ".ipynb_checkpoints/3_dataset_exploration-checkpoint.ipynb".
The first task is:
Let's start exploring the numerical features
Let's see a heatmap chart of  the average features for 'acquired' startups against the complete set of startups
Can you write Python code for it?

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

def plot_avg_status_against_avg_total(df, status):
    startups_numeric = df.filter(regex=('(number_of|avg_).*|.*(funding_total_usd|funding_rounds|_at|status)'))
    startups_acquired = startups_numeric[startups_numeric['status'] == status]

    startups_numeric = startups_numeric.drop('status', 1)
    startups_acquired = startups_acquired.drop('status', 1)

    fig, ax = plt.subplots(figsize=(20,20))
    ax.set_title(status+' startups heatmap')
    sns.heatmap((pd.DataFrame(startups_acquired.mean()).transpose() -startups_numeric.mean())/startups_numeric.std(ddof=0), annot=True, cbar=False, square=True, ax=ax)

plot_avg_status_against_avg_total(startups, 'acquired')


I am working on the file ".ipynb_checkpoints/3_dataset_exploration-checkpoint.ipynb".
The first task is:
Let's try some DecisionTrees for categories and see which performance we get.
Can you write Python code for it?

startups_non_numeric['status'].value_counts()
startups_non_numeric['acquired'] = startups_non_numeric['status'].map({'operating': 0, 'acquired':1, 'closed':0, 'ipo':0})
startups_non_numeric = startups_non_numeric.drop('status', 1)
startups_non_numeric[:3]

from sklearn import tree
def visualize_tree(tree_model, feature_names):
    """Create tree png using graphviz.

    Args
    ----
    tree_model -- scikit-learn DecsisionTree.
    feature_names -- list of feature names.
    """
    with open("dt.dot", 'w') as f:
        tree.export_graphviz(tree_model, out_file=f,
                        feature_names=feature_names)

    command = ["dot", "-Tpng", "dt.dot", "-o", "dt.png"]
    try:
        subprocess.check_call(command)
    except:
        exit("Could not run dot, ie graphviz, to "
             "produce visualization")

#import visuals_tree as vs_tree
#vs_tree.ModelLearning(startups_non_numeric.drop(['acquired','state_code'], 1), startups_non_numeric['acquired'])
from sklearn import tree
from sklearn.cross_validation import cross_val_score
from sklearn import tree
from sklearn import grid_search
from sklearn import preprocessing

#clf = tree.DecisionTreeClassifier(random_state=0)
#cross_val_score(clf, startups_non_numeric.drop(['acquired','state_code'], 1),  startups_non_numeric['acquired'], cv=10)

#Drop state_code feature
features = startups_non_numeric.drop(['acquired','state_code'], 1)

#Convert state_code feature to number
#features = startups_non_numeric.drop(['acquired'], 1)
#features['state_code'] = preprocessing.LabelEncoder().fit_transform(features['state_code'])

#Convert state_code to dummy variables
features = pd.get_dummies(startups_non_numeric.drop(['acquired'], 1), prefix='state', columns=['state_code'])

#Merge numeric_features to non-numeric-features
features_all = pd.concat([features, startups_numeric], axis=1, ignore_index=False)
#features = features_all

features = startups_numeric

parameters = {'max_depth':range(5,20)}
clf =...


I am working on the file ".ipynb_checkpoints/3_dataset_exploration-checkpoint.ipynb".
The first task is:
Okay. We see now that some features tend to express more variance than others.
We also see that funding_rounds variable tend to dominate against funding_total_usd values.
And also, that last_funding_at is a very expressing variable.
Let's start playing with non-numerical variables: dates and Categories
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/3_dataset_exploration-checkpoint.ipynb".
The first task is:
The most important variables here are:
Dimension1: funding_rounds, -last_funding_at, debt_financing_funding_rounds, venture_funding_rounds
Dimension2: -funding_rounds, -last_funding_at, -seed_funding_rounds, venture_funding_rounds
Dimension3: -last_funding_at, equity_crowdfunding_funding_rounds, -seed_funding_rounds
Dimension4: last_funding_at, equity_crowdfunding_funding_rounds, seed_funding_rounds
Now I'll apply the same PCA algorithm, but just for startups with acquired status
Can you write Python code for it?

startups_numeric_acquired = startups.filter(regex=('(number_of|avg_).*|.*(funding_total_usd|funding_rounds|_at|status)'))
startups_numeric_acquired = startups_numeric_acquired[startups_numeric_acquired['status'] == 'acquired']
startups_numeric_acquired = startups_numeric_acquired.drop('status', 1)

pca = PCA(n_components=4)
pca.fit(startups_numeric_acquired)

# Generate PCA results plot
pca_results = vs.pca_results(startups_numeric_acquired, pca)


I am working on the file ".ipynb_checkpoints/3_dataset_exploration-checkpoint.ipynb".
The first task is:
The same for 'closed':
Can you write Python code for it?

plot_avg_status_against_avg_total(startups, 'closed')

plot_avg_status_against_avg_total(startups, 'ipo')

plot_avg_status_against_avg_total(startups, 'operating')


I am working on the file ".ipynb_checkpoints/4. Time-variant networks-checkpoint.ipynb".
The first task is:
Creating a GraphCollection
As the name suggests, a GraphCollection is a container for graphs. The GraphCollection class gives us some convenient methods for generating and interrogating time-variant networks. The GraphCollection.build method allows us to build a series of graphs from our Corpus -- one graph per slice -- all in one step.
First, import GraphCollection directly from the tethne package.
Can you write Python code for it?

from tethne import GraphCollection

from tethne.networks import coauthors


I am working on the file ".ipynb_checkpoints/4. Time-variant networks-checkpoint.ipynb".
The first task is:
Introduction to Tethne: Time-Variant Networks
Now that we can index our Corpus temporally using the slice method, we can start to build time-variant networks. In this workbook we'll build a time-variant coauthor network using a GraphCollection.
Before you start

Download the Web of Science practice dataset from here, and store it in a place where you can find it. You'll need the full path to your dataset.
Complete the tutorial "Time and Change, Working with Corpora"

Loading data
We'll use the same WoS dataset that we've used in previous workbooks. Load it up using read().
Can you write Python code for it?

from tethne.readers.wos import read
datadirpath = '/Users/erickpeirson/Projects/tethne-notebooks/data/wos'
MyCorpus = read(datadirpath)


I am working on the file ".ipynb_checkpoints/4. Time-variant networks-checkpoint.ipynb".
The first task is:
Node indexing
Note that in the example above we had to convert our author name, (u'WARWICK', u'SI'), to an id, node_id. That's because the GraphCollection indexes all of the nodes so that we can track them across graphs. The index is stored in GraphCollection.node_index:
Can you write Python code for it?

MyGraphCollection.node_index.items()[0:10]    # The first ten nodes in the index.

MyGraphCollection.node_index[2467]    # Get the name of a specific node.


I am working on the file ".ipynb_checkpoints/4. Time-variant networks-checkpoint.ipynb".
The first task is:
Some algorithms, like "degree_centrality" and "betweenness_centrality" return a value for each node in each graph. In that case, the nodes in each graph are updated with those values.
Can you write Python code for it?

MyGraphCollection[2008].nodes(data=True)[15:17]   # Shows the attributes for two of the nodes in the 2008 graph.


I am working on the file ".ipynb_checkpoints/4. Time-variant networks-checkpoint.ipynb".
The first task is:
The method plot_attr_distribution can help to visualize the results of an algorithm across the graphs in the GraphCollection. In the example below, attr='degree_centrality' selects the degree_centrality attribute, etype='node' indicates that the attribute belongs to nodes (not edges), and stat=mean specifies that the Python function mean should be applied to the collection of values in each graph.
We can use node_history to look at how the attribute of a particular node changes across graphs. In the example below, the specified node appears first in 2008, and its centrality increases through 2011.
Can you write Python code for it?

node_id = MyGraphCollection.node_lookup[(u'WARWICK', u'SI')]
warwick_centrality = MyGraphCollection.node_history(node_id, 'degree_centrality')

warwick_centrality.items()[:20]    # First 20 years.

plt.plot(warwick_centrality.keys(), warwick_centrality.values(), 'ro')
plt.ylabel('Degree Centrality')
plt.show()


I am working on the file ".ipynb_checkpoints/5GB1_Cu_transition_timecourse_transctriptomics-checkpoint.ipynb".
The first task is:
Aside to remove all columns containing the string "QC"
Can you write Python code for it?

#many columns with QC runs. Gotta filter those out.
list(data_2.columns)

data_2.select(lambda x: not re.search("QC", x), axis = 1) #ok this is what I need, not lets break it down.

# re is a regex (regular expression) module. It is useful in selecting strings or parts of strings. Here is an example.

str_1 = 'an example word:cat12, word:cat!!, word:cattt165'
match = re.search(r'word:cat\d+', str_1) #the r ignores slashes (google education has a nice tutorial with re module)

if match:
    print("found", match.group())

else:
    print("did not find")

#lets find multiple words, the precious example on found 1

#say you have a text with many email addresses

str_2 = "purple alice@gmail.com, and many other like bob@yahoo.com and also a dishwasher"
emails = re.findall(r"\w+@\w+\.\w+", str_2)
for email in emails:
    print (email)

print(emails)


I am working on the file ".ipynb_checkpoints/5GB1_Cu_transition_timecourse_transctriptomics-checkpoint.ipynb".
The first task is:
Aside, how to keep only columns with FM40, and FM34
Can you write Python code for it?

#can individualy identify 1 column into a new dataframe
df = data["locus_tag"]
df

# for multiple column selection must use the __getitem__ syntax []
df1 = data[["locus_tag","type"]]
df1
#Dont want to manually enter all the column names


I am working on the file ".ipynb_checkpoints/5GB1_Cu_transition_timecourse_transctriptomics-checkpoint.ipynb".
The first task is:
Taking an aside to understanding iloc, loc, ix
Can you write Python code for it?

data.ix[1:3] #supports mixed integer and label based access. It is
            #primarily label based, but will fall back to integer positional
            #access unless the corresponding axis is of integer type.

data.loc[1] #returns the first row, Note: "1" is
            #interpreted as a *label* of the index, and **never** as an
            # integer position along the index).
data.loc[0:2,"5GB1_FM34_T0_TR1_QC"] #the integers are interpreted as names, not positions.

data.iloc[1:3] #returns the data for axis 0 = 1,2,3 (or the first three rows of data)
data.iloc[3] #returns the data for the third row
data.iloc[[1,2,5]] #returns data for rows 1,2,5

type(data.iloc[4])


I am working on the file ".ipynb_checkpoints/5GB1_Cu_transition_timecourse_transctriptomics-checkpoint.ipynb".
The first task is:
identifying the column index in order to remove unnecessary columns from data
Can you write Python code for it?

list(data.columns)

print(data.columns.get_loc("translation"))
print(data.columns.get_loc("5GB1_FM40_T0m_TR2"))
print(data.columns.get_loc("5GB1_FM40_T90m_TR2_QC"))

columns_list = list(range(0,9)) + list(range(20,42))

columns_list

data_1 = data.iloc[:, columns_list] #slicing the column index the way I want.


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
1. Load the Network
Here, we use the Zachary's karate club network $K$ to illustrate some concepts on network structures. This network was first presented in [1].
[1] W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of Anthropological Research 33, 452-473 (1977).
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
3. Divisive Method: The Girvan-Newman Algorithm
Edge Betweenness
The betweenness of an edge $e$ is the fraction of shortest-paths that course through it.
$$c_B(e)=\sum _{i,j \in V}\frac{\sigma (i,j)|e}{\sigma (i,j)},$$
where $\sigma(i,j)$ is the total number of shortest paths and $\sigma (i,j)|e$ is the number of shortest paths that pass through edge $e$. In networkx, the function is edge_betweenness_centrality(G). Below, we illustrate the function using Zachary's karate club $K$.
Can you write Python code for it?

#order the dictionary by value, which is the edge betweeness of two nodes
ebet = nx.edge_betweenness_centrality(K)
sorted_ebet = sorted(ebet.items(), key=operator.itemgetter(1), reverse=True)
sorted_ebet[0:5]


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
4. Agglomerative Method: Hierarchical Clustering
In this method, we start with a distance matrix $D$, which is a matrix that provides the distance between two nodes $i$ and $j$ in network $G$. Using $D$, we then perform hierarchical clustering in a recursive manner, group similar nodes at each step. There are various ways to perform the similarity grouping using agglomerative procedures such as single, complete, and average, to name a few. In the example below, we implement the average/UPGMA linkage, which is the one implemented in the Ravasz Algorithm published in Science in 2002.
In the following cells, we perform hierarchical clustering and visualize the order in which the nodes are grouped together in communities.
Can you write Python code for it?

__author__ = """\n""".join(['Erika Fille Legara <legareft@ihpc.a-star.edu.sg>',
                            'Maksim Tsvetovat <maksim@tsvetovat.org',
                            'Drew Conway <drew.conway@nyu.edu>',
                            'Aric Hagberg <hagberg@lanl.gov>'])

'''
The original code was written by Drew Conway and Aric Hagberg and was later
modified by Maksim Tsvetovat for his book titled “Social Network Analysis for Startups”.
EF Legara slightly modified Tsvetovat's version to address some issues
on the ordering of keys within the path_length dictionary. Moreover,
in this code, a different agglomerative procedure is implemented.
'''

def create_hc(G, t):
    ## Set-up the distance matrix D
    labels=G.nodes()    # keep node labels
    path_length=nx.all_pairs_shortest_path_length(G)
    distances=np.zeros((len(G),len(G)))
    for u,p in path_length.items():
        for v,d in p.items():
            distances[G.nodes().index(u)][G.nodes().index(v)] = d
            distances[G.nodes().index(v)][G.nodes().index(u)] = d
            if u==v: distances[G.nodes().index(u)][G.nodes().index(u)]=0

    # Create hierarchical cluster (HC)
    # There are various other routines for agglomerative clustering,
    # but here we create the HCs using the complete/max/farthest point linkage
    Y = distance.squareform(distances) ## the upper triangular of the distance matrix
    Z = hierarchy.average(Y)

    # This partition selection (t) is arbitrary, for illustrive purposes
    membership=list(hierarchy.fcluster(Z,t=t))

    # Create collection of lists for blockmodel
    partition = defaultdict(list)
    for n,p in zip(list(range(len(G))),membership):
        partition[p].append(labels[n])

    return Z, membership, partition

Z, membership, partition = create_hc(K, t=1.15)
partition.items()

partition = {}
i = 0
for i in range(len(membership)):
    partition[i]=membership[i]

plt.figure(figsize=(10,10))
plt.axis('off')

nx.draw_networkx_nodes(K, pos, cmap=plt.cm.RdYlBu,...


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
5. Modularity-Based Community Detection: The Louvain Method
The community detection algorithm that we implement in this notebook is a modularity-based algorithm. The modularity $M_c$ quantifies how good a "community" or partition is, and is given by
\begin{equation}
M_c = \sum _{c=1} ^{n_c} \left[ \frac{L_c}{L} - \left (\frac{k_c}{2L} \right)^2\right]
\end{equation}
where $n_c$ is the number of communiies, $L_c$ is the number of links within the community, $k_c$ is the total degree of nodes belonging to the community, and $L$ is the total number of links in the network. The illustration below that is taken from the slide deck shows how modularity is calculated.

The higher the network modularity is, the more optimal the partitionioning. And, if you only have a single community, $M=0$.

In this recipe, we implement a modularity-maximization algorithm called the Louvain algorithm. We go back to Zachary's karate club and find the best partitions.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
Dendrogram
The dendrogram allows us to visualize the order in which nodes are grouped together in the clustering process.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
Sample 3: Political Blogs
Here, we use the dataset utilized in the paper titled "The political blogosphere and the 2004 US Election" by L. A. Adamic and N. Glance (2005). It's a directed network of hyperlinks between weblogs on US politics.
Can you write Python code for it?

# P = nx.read_gml("./Module 6 Datasets/polblogs.gml")
P = nx.read_gml("./Module 6 Datasets/polblogs copy.gml")


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
Some Network Properties
Can you write Python code for it?

P.nodes(data=True)[0:2]

P.order(), P.size()

pos = nx.spring_layout(P)
nx.draw(P, pos, node_size=30)
plt.show()


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
The algorithm produces four (4) partitions. Let's compare this partition to the actual.
Can you write Python code for it?

plt.figure(figsize=(8,8));
plt.axis("off");
nx.draw_networkx_nodes(K,pos, nodelist=mrhi, node_color='r',
                       node_size=300, alpha=0.8);
nx.draw_networkx_nodes(K,pos, nodelist=johna, node_color='b',
                       node_size=300, alpha=0.8);
nx.draw_networkx_edges(K, pos, alpha=0.5);
nx.draw_networkx_labels(K, pos, font_color="white");


I am working on the file ".ipynb_checkpoints/6. Community Detection-checkpoint.ipynb".
The first task is:
The code below may take a while to run since we are looking at a relatively large network, so we're commenting it out.'
```python
plt.figure(figsize=(10,10))
plt.axis('off')
pos = nx.spring_layout(M)
nx.draw_networkx_nodes(M, pos,cmap=plt.cm.RdYlBu, node_color=partition.values())
nx.draw_networkx_edges(M, pos, alpha=0.4)
plt.savefig("Marvel Universe.png")
```
But this is how it looks using Gephi.
<img src="./figs/marvel.png">
What are these communities? Which community is the largest? Sneak a look into the communities.
Can you write Python code for it?

comm = defaultdict(list)
for k, v in partition.items():
    comm[v].append(k)

print comm.items()[11]


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Applying a stoplist
In many cases you may wish to apply a stoplist (a list of features to exclude from analysis) to a featureset. In our DfR dataset, the most common words are prepositions and other terms that don't really have anything to do with the topical content of the corpus.
Can you write Python code for it?

corpus.features['wordcounts'].top(5)    # The top 5 words in the FeatureSet.


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
But since not all of our WoS records come from >= 1990, there are a handful for which there are no abstract terms.
Can you write Python code for it?

print 'Only {0} of {1} papers have abstracts, however.'.format(len(wosCorpus.features['abstract'].features), len(wosCorpus.papers))

filter = lambda f, v, c, dc: f not in stoplist and 2 < dc < 400
wosCorpus.features['abstract_filtered'] = wosCorpus.features['abstract'].transform(filter)

print 'There are {0} features in the abstract_filtered featureset.'.format(len(wosCorpus.features['abstract_filtered'].index))

type(wosCorpus.features['abstract_filtered'])


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Generating networks from topic models
The features module in the tethne.networks subpackage contains some useful methods for visualizing topic models as networks. You can import it just like the authors or papers modules.
Can you write Python code for it?

from tethne.networks import topics

termGraph = topics.terms(model, threshold=0.01)

termGraph.name = ''


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Introduction to Tethne: Words and Topic Modeling
In this workbook we'll start working with word-based FeatureSets, and use the model.corpus.mallet module to  fit a Latent Dirichlet Allocation (LDA) topic model.
Before you start

Download the practice dataset from here, and store it in a place where you can find it. You'll need the full path to your dataset.

Loading JSTOR DfR datasets
First, import the dfr module from tethne.readers.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
It's not uncommon for featuresets to be incomplete; sometimes data won't be available for all of the Papers in the Corpus. An easy way to check the number of Papers for which data is available in a FeatureSet is to get the size (len) of the features attribute of the FeatureSet. For example:
Can you write Python code for it?

print 'There are %i papers with unigram features in this corpus.' % len(corpus.features['wordcounts'].features)


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Next we'll fit the model. We need to tell MALLET how many topics to fit (the hyperparameter Z), and how many iterations (max_iter) to perform. This step may take a little while, depending on the size of your corpus.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Now we'll create a new LDAModel for our Corpus. The featureset_name parameter tells the LDAModel which FeatureSet we want to use. We'll use our filtered wordcounts.
Can you write Python code for it?

model = mallet.LDAModel(corpus, featureset_name='wordcounts_filtered')


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
So you can retrieve specific Papers from Corpus.papers using their DOIs:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
The 'abstract' featureset is similar to the 'unigrams' featureset from the JSTOR DfR dataset, so we can perform topic modeling with it, too.
Can you write Python code for it?

wosModel = mallet.LDAModel(wosCorpus, featureset_name='abstract_filtered')

wosModel.fit(Z=50, max_iter=500)

wosModel.print_topics(Nwords=5)


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
The network visualization below was generated in Cytoscape. Edge width is a function of the 'weight' attribute. Edge color is based on the 'topics' attribute, to give some sense of how which clusters of terms belong to which topics. We can see right away that terms like plants, populations, species, and selection are all very central to the topics retrieved by this model.

WoS abstracts
JSTOR DfR is not the only source of wordcounts with which we perform topic modeling. For records more recent than about 1990, the Web of Science includes abstracts in their bibliographic records.
Let's first spin up our WoS dataset.
Can you write Python code for it?

from tethne.readers import wos
wosCorpus = wos.read('/Users/erickpeirson/Projects/tethne-notebooks/data/wos')


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
The topic_coupling function generates a network of words connected on the basis of shared affinity with a topic. If two words i and j are both associated with a topic z with $\Phi(i|z) >= 0.01$ and $\Phi(j|z) >= 0.01$, then an edge is drawn between them.
The resulting graph will be smaller or larger depending on the value that you choose for threshold. You may wish to increase or decrease threshold to achieve something interpretable.
Can you write Python code for it?

print 'There are {0} nodes and {1} edges in this graph.'.format(
            len(termGraph.nodes()), len(termGraph.edges()))


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
To check the number of features in a featureset (e.g. the size of a vocabulary), look at the size of the index attribute.
Can you write Python code for it?

print 'There are %i words in the unigrams featureset' % len(corpus.features['wordcounts'].index)


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Topic modeling with featuresets
Latent Dirichlet Allocation is a popular approach to discovering latent "topics" in large corpora. Many digital humanists use a software package called MALLET to fit LDA to text data. Tethne uses MALLET to fit LDA topic models.
Start by importing the mallet module from the tethne.model.corpus subpackage.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Unlike WoS datasets, DfR datasets can contain wordcounts, bigrams, trigrams, and quadgrams in addition to bibliographic data. read will automagically load those data as FeatureSets.
Can you write Python code for it?

corpus = dfr.read('/Users/erickpeirson/Projects/tethne-notebooks/data/dfr')

print 'There are %i papers in this corpus.' % len(corpus.papers)

print 'This corpus contains the following features: \n\t%s' % '\n\t'.join(corpus.features.keys())


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
We apply the stoplist using the transform() method. FeatureSets are not modified in place; instead, a new FeatureSet is generated that reflects the specified changes. We'll call the new FeatureSet 'wordcounts_filtered'.
Can you write Python code for it?

corpus.features['wordcounts_filtered'] = corpus.features['wordcounts'].transform(apply_stoplist)

print 'There are %i words in the wordcounts_filtered FeatureSet' % len(corpus.features['wordcounts_filtered'].index)


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
We can also look at the representation of a topic over time using the topic_over_time() method. In the example below we'll print the first five of the topics on the same plot.
Can you write Python code for it?

plt.figure(figsize=(15, 5))
for k in xrange(5):
    x, y = model.topic_over_time(k)
    plt.plot(x, y, label='topic {0}'.format(k), lw=2, alpha=0.7)
plt.legend(loc='best')
plt.show()


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
We then create a function that will evaluate whether or not a word is in our stoplist. The function should take three arguments:

f -- the feature itself (the word)
v -- the number of instances of that feature in a specific document
c -- the number of instances of that feature in the whole FeatureSet
dc -- the number of documents that contain that feature

This function will be applied to each word in each document. If it returns 0 or None, the word will be excluded. Otherwise, it should return a numeric value (in this case, the count for that document).
In addition to applying the stoplist, we'll also exclude any word that occurs in more than 50 of the documents and less than 3 documents.
Can you write Python code for it?

def apply_stoplist(f, v, c, dc):
    if f in stoplist or dc > 50 or dc < 3:
        return 0
    return v


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
Whereas Corpora generated from WoS datasets are indexed by wosid, Corpora generated from DfR datasets are indexed by doi.
Can you write Python code for it?

corpus.indexed_papers.keys()[0:10]    # The first 10 dois in the Paper index.


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
index maps words to an integer representation. Here are the first ten words in the FeatureSet:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
topics.cotopics() creates a network of topics, linked by virtue of their co-occurrence in documents. Use the threshold parameter to tune the density of the graph.
Can you write Python code for it?

coTopicGraph = topics.cotopics(wosModel, threshold=0.15)

print '%i nodes and %i edges' % (coTopicGraph.order(), coTopicGraph.size())

graph.write_graphml(coTopicGraph, '/Users/erickpeirson/Projects/tethne-notebooks/output/lda_coTopics.graphml')


I am working on the file ".ipynb_checkpoints/6. Words and topic modeling-checkpoint.ipynb".
The first task is:
topics.topicCoupling() creates a network of documents, linked by virtue of containing shared topics. Again, use the threshold parameter to tune the density of the graph.
Can you write Python code for it?

topicCoupling = topics.topic_coupling(wosModel, threshold=0.2)

print '%i nodes and %i edges' % (topicCoupling.order(), topicCoupling.size())

graph.write_graphml(topicCoupling, '/Users/erickpeirson/Projects/tethne-notebooks/output/lda_topicCoupling.graphml')


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
...or more succinctly...
Try grabbing the bottom left corner (3x3)...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
A few more practical uses of nparray:
Can you write Python code for it?

data.mean(), data.max(), data.min()

data[3].max()  # Max inflammation for 4th patient


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
A lot of the magic methods here, those with double underscores on either side, allow Python to treat this object more like a built-in. For instance, the __neg__ method will get called when we enter -data. If we try data1 < data2, then data1.__lt__ will be called with data2 as an argument, expecting a boolean return. The behaviour when the user writes data.previously_unknown_attr is defined by the __getattr__ method - this need not be an error. In some ways, you can think of this as an expansion of the concept of operator overloading.
The ndarray class uses those methods very effectively, to help us write clear and efficient code.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
As you can see, the second day does indeed have max of 1.0. Notice that it returns an array (this is a superclass of ndarray), so we can do...
An aside: this is the first time you have seen a named argument. In an argument list of a function, you can supply a name for arguments:
python
def some_function(arg1, info=38, axis=None):
   ...
To write a function like this, you must provide a default for the argument. You can then call the function only passing a subset of arguments, as you have just seen.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
Challenge
Put up your stars!!
Modify the previous cell to give the max line with a dash, the blue line with dots and the red line solid.
Even better, add a title at the top.
When done, swap to the arrow. If you find this very easy and are waiting, leave your arrow up and have a go at plotting all the patients in very light grey behind your three coloured lines. See how long it takes to run for those 63 lines.
RMV: To get you started, Google pyplot line style and pick the pylab_examples example code link...
Bear in mind, and this is particularly the case with matplotlib, that there are multiple similar tools with slightly different configurations and long lists of technical documentation - try not to get lost in them and use Etherpad to flags things up. However, short examples are findable via Google, especially via Stack Overflow, that get you going fast.
But...
Not so interactive
If we are using Jupyter, it is probably because we want to have something interactive. Fixed plots in a web-page make it seem like something is missing.
Bokeh
Dynamic plots
Can you write Python code for it?

import bokeh.plotting as bplot
from bokeh.io import output_notebook
output_notebook()


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
Don't forget your star!
Can you write Python code for it?

# RMV
# Start out with days vs average
initial_coordinates = {'x': days, 'y': data.mean(axis=0)}
source = ColumnDataSource(initial_coordinates)

# Define a callback to update the plot when we
# pick something else
def update_plot(N):
    source.data['x'] = np.arange(N)
    push_notebook()

fig = bplot.figure()

days = np.arange(data.shape[1])

fig.xaxis.axis_label = 'Day'
fig.yaxis.axis_label = 'Inflammation'

fig.line(initial_coordinates['x'], initial_coordinates['y'], source=source)

bplot.show(fig)

interact(update_plot, N=(1, 60, 1))


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
If this works, you should see an icon like a rainbow-coloured camera shutter.
Can you write Python code for it?

fig = bplot.figure()
days = np.arange(data.shape[1])
fig.line(days, data.mean(axis=0))
fig.xaxis.axis_label = "Day"
bplot.show(fig)


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
Now for inline plotting we can use the matplotlib module, the go-to for Python plotting (but not in the Dijkstra sense)...
Can you write Python code for it?

import matplotlib
pretty_pic = matplotlib.pyplot.imshow(data)
matplotlib.pyplot.show(pretty_pic)


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
Now we need a widget to use this...
Can you write Python code for it?

from ipywidgets import interact

interact(update_plot_statistic, statistic=["Average", "Max", "Min"])


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
RIGHT
Enough text.
Programming isn't about writing reams of code, it's about getting stuff done and having fun while doing it. Right? Right. Right. RIGHT.
In any case, when examining data, it's a whole lot more pleasant and efficient to see visualizations, so lets get started with some plotting.
Plotting
Like we're Brutus the Younger
First, we use a magic method that works in Jupyter (strictly, in its IPython backend)
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
Reproducible Plots
If you come from a MATLAB background, or use other plotting tools, you will be used to saving/re-opening a plot...
...in Python you normally have a plotting script that you run each time (and save the image).
Cons

You have no standard GUI for designing plots
If your plot involves lots of processing, it's slow to open

Pros

Changes are trivially easy
Every time you open the plot, it contains current data

Con-workarounds

There are libraries for adding GUI elements to plots
You can split into two scripts
output processed data
plot

Example plotting script
Can you write Python code for it?

import numpy as np
from matplotlib import pyplot

data = np.loadtxt(fname='data/inflammation-01.csv', delimiter=',')
pyplot.figure(figsize=(5.0, 3.0))

pyplot.xlabel('Day')
pyplot.ylabel('Inflammation')

pyplot.plot(data.mean(axis=0), label='Average')
pyplot.plot(data.max(axis=0), label='Max')
pyplot.plot(data.min(axis=0), label='Min')
pyplot.legend()

pyplot.show()

exec(In[42])  # Cheeky way to reshow output... forget immediately.


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
Suppose we want the max for every day...
We could create a new list and write a loop to go through it, but it seems there should be a more transparent, succinct, Pythonic approach
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
This a very basic approach - the widget tool guesses you want a drop-down because you pass a list. Alternatively, you could pass a tuple with two floats as limits and get a slider back.
It seems you do have to output the image before the widgets, if you define the callback in Python, at least. However, there are a lot of other interactors - you can pass a JS callback (as a string), assign URLs to various objects in your plot, animate it, etc. etc. etc.
Final Challenge
Add a slider for variable N from 1 to 60 that reduces the dataset from all Patients to Patients 1 to N. Unless you're feeling particularly ambitious, you can replace the drop-down from the previous example with this slider.
Can you write Python code for it?

# Use this cell for the plot

# And this for the one line `interact` call afterwards


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
This could also be written data[3,:] - colon on its own just signifies all entries in that axis, from first to last
Can you write Python code for it?