I am working on the file "Code/IPython/Bootcamp_1_Examples.ipynb".
The first task is:
Example 2: Fama-French equity "factors"
Gene Fama and Ken French are two of the leading academics studying (primarily) equity returns.  Some of this work is summarized in the press release and related material for the 2013 Nobel Prize in economics, which was shared by Fama with Lars Hansen and Robert Shiller.  For now, it's enough to say that Ken French posts an extensive collection of equity data on his website.
We'll look at what have come to be called the Fama-French factors.  The data includes:

xsm:  the return on the market (aggregate equity) minus the riskfree rate
smb (small minus big):  the return on small firms minus the return on big firms
hml (high minus low):  the return on firms with high book-to-market ratios minus those with low ratios.
rf:  the riskfree rate.

We download all of these at once, monthly from 1926.  Each is reported as a percentage.
Since they're monthly, you can get a rough annual number if you multiply by 12.
Question(s).
The question we address is how the returns compare:  their means, their variability, and so on.
[Ask yourself:  how would I answer this?  What would I like to do with the data?]
References

http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html
http://quant-econ.net/pandas.html
http://pandas.pydata.org/pandas-docs/dev/remote_data.html#fama-french
http://pandas.pydata.org/pandas-docs/stable/10min.html#selection
http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.boxplot
http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.hist.html
Can you write Python code for it?

# load packages (if it's redundant it'll be ignored)
import pandas.io.data as web

# read data from Ken French's website
ff = web.DataReader('F-F_Research_Data_Factors', 'famafrench')[0]
# NB:  ff.xs is a conflict, rename to xsm
ff.columns = ['xsm', 'smb', 'hml', 'rf']

# see what we've got
print(ff.head(3))
print(ff.describe())

# compute and print summary stats
moments = [ff.mean(), ff.std(), ff.skew(), ff.kurtosis() - 3]

# \n here is a line break
print('Summary stats for Fama-French factors (mean, std, skew, ex kurt)') #, end='\n\n')
print(moments)
#[print(moment, end='\n\n') for moment in moments]

# try some things yourself
# like what?  type ff.[tab]
import pandas as pd
pd.__version__

# some plots
ff.plot()
plt.show()

ff.hist(bins=50, sharex=True)
plt.show()

ff.boxplot(whis=0, return_type='axes')
plt.show()


I am working on the file "Code/IPython/Bootcamp_1_Examples.ipynb".
The first task is:
Example 4: Option prices
A financial option gives its owner the right to buy or sell an asset (the "underlying") at a preset price (the "strike") by a specific date (the "expiration date").  Puts are options to sell, calls are options to buy.  We explore option prices with Yahoo Finance, specifically options on the S&P 500 exchange-traded fund, ticker SPY.
We illustrate its usefulness with a scatterplot of life expectancy v GDP per capita.
Question(s).  How do put and call prices vary with their strike price?  [Think about this.  What would you expect?]
Warning. This won't work in Python 2.7 or, in fact, in any environment that uses versions of Pandas prior to 0.14.1.  The Yahoo Option API is labeled experimental and it seems the earlier versions don't allow easy access to the strike prices.
References

http://finance.yahoo.com/q/op?s=SPY+Options
http://pandas.pydata.org/pandas-docs/stable/remote_data.html#yahoo-finance
http://pandas.pydata.org/pandas-docs/stable/remote_data.html#yahoo-finance-options
Can you write Python code for it?

# load packages
import pandas as pd
import pandas.io.data as web
from pandas.io.data import Options
import datetime as dt
import matplotlib.pylab as plt

# ticker
ticker = 'spy'

# load stock price first (the underlying)
# pick a recent date and subtract seven days to be sure we get a quote
# http://pymotw.com/2/datetime/#date-arithmetic
today = dt.date.today()
one_week = dt.timedelta(days=7)
start = today - one_week
stock = web.DataReader(ticker, 'yahoo', start)
print(stock)        # just to see what we have

# take the last close (-1 is the last, 'Close' is the close)
# this shows up in our figure
atm = stock.ix[-1,'Close']      # the -1 takes the last observation

# get option prices for same ticker
option = Options(ticker, 'yahoo')
expiry = dt.date(2014, 11, 20)
data_calls = option.get_call_data(expiry=expiry).dropna()
data_puts  = option.get_put_data(expiry=expiry).dropna()

# check what we have
print(data_calls.index)
print(data_calls.tail())

# compute mid of bid and ask and arrange series for plotting
calls_bid = data_calls['Bid']
calls_ask = data_calls['Ask']

calls_strikes = data_calls['Strike']
calls_mid = (data_calls['Bid'] + data_calls['Ask'])/2
puts_strikes = data_puts['Strike']
puts_mid = (data_puts['Bid'] + data_puts['Ask'])/2


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
As per usual, we begin by listing the versions of each package that is used in this notebook.
Can you write Python code for it?

# check versions (overkill, but why not?)
print('Python version:', sys.version)
print('Pandas version: ', pd.__version__)
print('Matplotlib version: ', mpl.__version__)
print('Seaborn version: ', sns.__version__)


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Better Defaults
Recall that in our previous notebook that we used plt.style.use to set styles. We will begin by setting the style to "classic"; this sets all of our default settings back to matplotlib's default values.
Below we plot open and closing prices on the top axis and the implied returns on the bottom axis.
Can you write Python code for it?

plt.style.use("classic")

def plot_tips():
    fig, ax = plt.subplots(2, figsize=(8, 6))

    tips[tips["sex"] == "Male"].plot(x="total_bill", y="tip", ax=ax[0], kind="scatter",
                                     color="blue")
    tips[tips["sex"] == "Female"].plot(x="total_bill", y="tip", ax=ax[1], kind="scatter",
                                       color="#F52887")

    ax[0].set_xlim(0, 60)
    ax[1].set_xlim(0, 60)
    ax[0].set_ylim(0, 15)
    ax[1].set_ylim(0, 15)

    ax[0].set_title("Male Tips")
    ax[1].set_title("Female Tips")

    fig.tight_layout()

plot_tips()
# fig.savefig("/home/chase/Desktop/foo.png")

# sns.set() resets default seaborn settings
sns.set()

plot_tips()


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Exercise: Find a style you like and recreate the plot above using that style.
The Juicy Stuff
While having seaborn set sensible defaults is convenient, it isn't a particularly large innovation. We could choose sensible defaults and set them to be our default. The main benefit of seaborn is the types of graphs that it gives you access to -- All of which could be done in matplotlib, but, instead of 5 lines of code, it would require possibly hundreds of lines of code. Trust us... This is a good thing.
We don't have time to cover everything that can be done in seaborn, but we suggest having a look at the gallery of examples.
We will cover:

kdeplot
jointplot
violinplot
pairplot
...
Can you write Python code for it?

# Move back to seaborn defaults
sns.set()


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Exercise: We might also want to look at the distribution of prices across ticket classes. Make a violin plot of the prices over the different ticket classes.
Can you write Python code for it?

sns.swarmplot(x="sex", y="age", hue="survived", data=tita)


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Like an contour map
Exercise: Create your own jointplot. Feel free to choose your own x and y data (if you can't decide then use x=size and y=tip). Interpret the output of the plot.
Can you write Python code for it?

sns.jointplot(x="size", y="tip", data=tips,
              kind="kde")

sns.jointplot(x="size", y="tip", data=tips)


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Pairplot
Pair plots show us two things. They show us the histograms of the variables along the diagonal and then the scatter plot of each pair of variables on the off diagonal pictures.
Why might this be useful? It allows us to look get an idea of the correlations across each pair of variables and gives us an idea of their relationships across the variables.
Can you write Python code for it?

sns.pairplot(tips[["tip", "total_bill", "size"]], size=2.5)


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
We could do the same for a different style (like ggplot)
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
What did you notice about the differences in the settings of the plot?
Which do you like better? We like the second better.
Investigate other styles and create the same plot as above using a style you like. You can choose from the list in the code below.
If you have additional time, visit the seaborn docs and try changing other default settings.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
What's different about this plot?
Different colors for each company.
Can you write Python code for it?

tips.head()

sns.pairplot(tips[["tip", "total_bill", "size", "time"]], size=3.5,
             hue="time", diag_kind="kde")


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
kdeplot
What does kde stand for?
kde stands for "kernel density estimation." This is (far far far) beyond the scope of this class, but the basic idea is that this is a smoothed histogram. When we are trying to get information about distributions it sometimes looks nicer than a histogram does.
Can you write Python code for it?

fig, ax = plt.subplots()

ax.hist(tips["tip"], bins=25)

ax.set_title("Histogram of tips")
plt.show()

fig, ax = plt.subplots()

sns.kdeplot(tips["tip"], ax=ax)
ax.hist(tips["tip"], bins=25, alpha=0.25, normed=True, label="tip")
ax.legend()

fig.suptitle("Kernel Density with Histogram");


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
lmplot
We often want to think about running regressions of variables. A statistician named Francis Anscombe came up with four datasets that:

Same mean for $x$ and $y$
Same variance for $x$ and $y$
Same correlation between $x$ and $y$
Same regression coefficient of $x$ on $y$

Below we show the scatter plot of the datasets to give you an idea of how different they are.
Can you write Python code for it?

fig, ax = plt.subplots(2, 2, figsize=(8, 6))

ansc[ansc["dataset"] == "I"].plot.scatter(x="x", y="y", ax=ax[0, 0])
ansc[ansc["dataset"] == "II"].plot.scatter(x="x", y="y", ax=ax[0, 1])
ansc[ansc["dataset"] == "III"].plot.scatter(x="x", y="y", ax=ax[1, 0])
ansc[ansc["dataset"] == "IV"].plot.scatter(x="x", y="y", ax=ax[1, 1])

ax[0, 0].set_title("Dataset I")
ax[0, 1].set_title("Dataset II")
ax[1, 0].set_title("Dataset III")
ax[1, 1].set_title("Dataset IV")

fig.suptitle("Anscombe's Quartet")


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
lmplot plots the data with the regression coefficient through it.
Can you write Python code for it?

sns.lmplot(x="x", y="y", data=ansc, col="dataset", hue="dataset",
           col_wrap=2, ci=None)

sns.lmplot(x="total_bill", y="tip", data=tips, hue="sex")


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
regplot
regplot also shows the regression line through data points
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
violinplot
Some of the story of this notebook is that distributions matter and how we can show them. Violin plots are similar to a sideways kernel density and it allows us to look at how distributions matter over some aspect of the data.
Can you write Python code for it?

tita.head()

tips.head()

sns.violinplot?

sns.violinplot(x="fare", y="deck", hue="survived", split=True, data=tita)
# sns.swarmplot(x="class", y="age", hue="sex", data=tita)

sns.boxplot(x="class", y="age", hue="sex", data=tita)


I am working on the file "Code/IPython/bootcamp_exam_practice.ipynb".
The first task is:
IPython review
We review some of the basics of IPython.  You won't be asked about IPython on the exam, but since the exam is an IPython notebook, it's essential for you to be able to work with one.
Question 1.

How do you set/choose the current cell?
How do you edit the current cell?
How do you add a new cell below the current cell?
How do you specify the current cell as code or text?
How do you delete the current cell?
How do you move the current cell up or down?
How do you run the current cell?
Add your name in bold to the bottom of the first cell in this notebook.  Bonus: Add a link to your LinkedIn or Facebook page.
How do you save the contents of your notebook?

Answers.  Enter your answers below:

Python fundamentals
Question 2. Describe the type and content of these expressions:

x = 2
y = 3.0
z = "3.0"
x/y
letters = 'abcd'
letters[-1]
xyz = [x, y, z]
xyz[1]
abcd = list(letters)
abcd[-2]
case = {'a': 'A', 'b': 'B', 'c': 'C'}
case['c']
2 >= 1
x == 2

Answers. Enter your answers below:
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_exam_practice.ipynb".
The first task is:
Question 3. These get progressively more difficult:

What type is dollars = '$1,234.5'?
Find and apply a method that eliminates the dollar sign from dollars.
Find and apply a method that eliminates the comma from dollars.
Eliminate both the dollar sign and comma from dollars and covert the result to a float.
Combine the last three steps in one line.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Question 4.
For this problem we set letters = 'abcd' as in problem 2.

Find and apply a method that converts the lower case letter 'a' to the upper case letter 'A'.
Write a loop that goes through the elements of letters and prints their upper case versions.
Bonus: Write a loop that goes through the elements of letters.  On each interation, print a string consisting of the upper and lower case versions together; eg, 'Aa'.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Question 5.
For this problem xyz is the same as defined in problem 2

Write a loop that goes through the elements of xyz and prints them.
Modify the loop to print both the elements of xyz and their type.
Modify the loop to print only those elements that are not strings.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Data input with Pandas
We explore the public indebtedness of Argentina (country code ARG), Germany (DEU), and Greece (GRC).  For each one, we provide the ratio of government debt to GDP for every second year starting in 2002.  The data come from the IMF's World Economic Outlook.
Question 6. Write code in the cell below that reads the csv file we posted at
http://pages.stern.nyu.edu/~dbackus/Data/debt.csv
Assign the contents of the file to the object debt.
The rest of the questions in this notebook will refer to the object debt you create below.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
IPython review
We review some of the basics of IPython.  You won't be asked about IPython on the exam, but since the exam is an IPython notebook, it's essential for you to be able to work with one.
Question 1.

How do you set/choose the current cell?
How do you edit the current cell?
How do you add a new cell below the current cell?
How do you specify the current cell as code or text?
How do you delete the current cell?
How do you move the current cell up or down?
How do you run the current cell?
Add your name in bold to the bottom of the first cell in this notebook.  Bonus: Add a link to your LinkedIn or Facebook page.
How do you save the contents of your notebook?

Answers.  Enter your answers below:

click on it
click again
click on the plus (+) at the top
choose the appropriate one in the menu below Help at the top
choose the cell and click on the scirrors at the top
choose the cell and click on the up or down arrow at the top
two ways:  click on the run cell icon at the top, or shift-enter
**name**
two ways:  File, Save and Checkpoint, or cntl-S.

Python fundamentals
Question 2. Describe the type and content of these expressions:

x = 2
y = 3.0
z = "3.0"
x/y
letters = 'abcd'
letters[-1]
xyz = [x, y, z]
xyz[1]
abcd = list(letters)
abcd[-2]
case = {'a': 'A', 'b': 'B', 'c': 'C'}
case['c']
2 >= 1
x == 2

Answers. Enter your answers below:
By content and type we mean the content of the variable or expression and its type as return by the type() function.

content 2, type int
content 3.0, type float
content '3.0', type str
content 0.66666, type float
content 'abcd', type str
content 'd', type str
content [x, y, z], type list
content y=3.0, type float
content ['a', 'b', 'c', 'd'], type list
content 'c', type str
content as stated, type dictionary or dict
content 'C', type str
content True, type bool
content True, type bool
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Python graphics with Matplotlib
We'll continue to use the data in debt.  Make sure the index is the year.
Question 9.

Create figure and axis objects with plt.subplots().
Graph public indebtedness over time using our debt data and the axis object we just created.
Change the line width to 2.
Change the colors to ['red', 'green', 'blue'].
Change the lower limit on the y axis to zero.
Add a title to the graph.
Add a label to the y axis -- something like "Public Debt (% of GDP)".
Bonus: Make the line for Argentina thicker than the others.  Hint: Do this by plotting a separate line applied to the same axis object.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Can you write Python code for it?

fig, ax = plt.subplots()
debt.plot(ax=ax,
         linewidth=2,
         color=['red', 'green', 'blue'])
ax.set_ylim(0)
ax.set_title('Public debt')
ax.set_ylabel('Public Debt (% of GDP)')
debt['Argentina'].plot(ax=ax, linewidth=4, color='red')


I am working on the file "Code/IPython/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Question 11.  We ran across this one in the OECD healthcare data.  The country names had numbers appended, which served as footnotes in the original spreadsheet but looked dumb when we used them as index labels.  The question is how to eliminate them.  A short version of the country names is
names = ['Australia 1', 'Canada 2', 'Chile 3', 'United States 1']
Do each of these in a separate code cell:

Apply the rsplit() method to us = names[-1].  What do you get?
Consult the documentation for rsplit to split us into two pieces, the country name and the number 1.  How would you extract just the country name?
Use a loop to strip the numbers from all of the elements of names.
Use a list comprehension to strip the numbers from all of the elements of names.

Hints. rsplit means split from the right.  One input is the number of splits.
Can you write Python code for it?

names = ['Australia 1', 'Canada 2', 'Chile 3', 'United States 1']

names[-1].rsplit()

names[-1].rsplit(maxsplit=1)

# apologies, this is harder than we thought
for n in range(len(names)):
    item = names[n]
    names[n] = item.rsplit(maxsplit=1)[0]

print(names)

# this one's easier
names = ['Australia 1', 'Canada 2', 'Chile 3', 'United States 1']
[item.rsplit(maxsplit=1)[0] for item in names]


I am working on the file "Code/IPython/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Question 8. Do the following with debt:

Set Year as the index.
Change the column labels from country codes to country names. Do this using both a dictionary and a list.
Print the result to verify your changes.

The next three get progressively more difficult:

Compute the mean (average) debt for each country.
Bonus: Compute the mean debt for each year.
Bonus: Compute the mean debt over both countries and years.

Some simple plots:

Plot each country's debt against Year using a plot method.
Change the linewidth to 2.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Can you write Python code for it?

debt = debt.set_index('Year')

rn = {"ARG": "Argentina", "DEU": "Germany", "GRC": "Greece"}
debt.rename(columns=rn)

debt.columns = ['Argentina', 'Germany', 'Greece']
debt

debt.mean()

debt.mean(axis=1)

debt.mean().mean()


I am working on the file "Code/IPython/bootcamp_exam_s16_answerkey.ipynb".
The first task is:
Changed cell to Markdown with menu at top
The code from if on:

if torf is True, set x=1
if torf is False, we set x=2

At the top, torf is True, so we do the first one (x=1).  If we change it to False, we do the second one (x=2).
Question 4
Take the first and last variables defined in the cell below and do the following with them:
(a) Extract the first letter of last.
(b) Find a method to split last into two components at the hyphen.
(c) Define a new string variable named combo consisting of first (the first name), a space, the first letter of last, and a period.
(d) Define a function that takes as inputs first and last names (both strings) and returns combo (also a string, consisting of the first name plus the first letter of the last name and a period).  Apply it to the variables first and last and to your own first and last names.
(20 points)
Can you write Python code for it?

first = 'Sarah'
last  = 'Beckett-Hile'

# (a)
firstoffirst = first[0]
firstoffirst

# (b)
last.split('-')

# (c)
combo = first + ' ' + last[0] + '.'
combo

# (d)
def lastinitial(name1, name2):
    combo = name1 + ' ' + name2[0] + '.'
    return combo

lastinitial(first, last)

lastinitial('Chase', 'Coleman')


I am working on the file "Code/IPython/bootcamp_exam_s16_answerkey.ipynb".
The first task is:
Question 2
As above describe the value and type of each variable.  (These are more challenging.)
(f) f = (1, 2, 3)
(g) g = {1: 'Chase', 2: 'Dave', 3: 'Spencer'}
(h) h = 'foo' + 'bar'
(i) i = (1 != 0)   # parens not needed, but they make code more understandable
(20 points)
Can you write Python code for it?

f = (1, 2, 3)
g = {1: 'Chase', 2: 'Dave', 3: 'Spencer'}
h = 'foo' + 'bar'
i = (1 != 0)

# (f)
valuetype(f)

# (g)
valuetype(g)

# (h)
valuetype(h)

# (i)
valuetype(i)


I am working on the file "Code/IPython/bootcamp_exam_s16_answerkey.ipynb".
The first task is:
Question 3
Explain the code below -- briefly -- in a Markdown cell.  What happens if we change the first line to torf = False?
(10 points)
Can you write Python code for it?

torf = True

if torf:
    x = 1
else:
    x = 2

print('x =', x)


I am working on the file "Code/IPython/bootcamp_exam_s16_answerkey.ipynb".
The first task is:
Question 7
Continuing with the same data:
(a) Create a new variable AvgEmp equal to the ratio of Emp to Firms and add it as a new column in bsd.
(b) Use a dataframe method to change the name of Emp to Employees.
(c) Create a bar chart of the number of employees in each size category.
(15 points)
Can you write Python code for it?

# (a)
bds['AvgEmp'] = bds['Emp']/bds['Firms']
bds.head(3)

# (b)
bds = bds.rename(columns={'Emp': 'Employment'})
bds.head(3)

# (c)
bds['Employment'].plot.bar()


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Answer(s)?  Aren't the boxplots in the last figure cool?  The histograms above them?  What do you see in them?  How do the various returns compare?

Example 3: GDP per capita and life expectancy
The World Bank collects a broad range of economic and social indicators for most countries in the World.  They also have a nice interface.  It's a good source for basic information about the economic climate compares across countries.
We illustrate its usefulness with a scatterplot of life expectancy v GDP per capita.
Question(s).  How closely are these two indicators of quality of life are related.
References

http://data.worldbank.org/
http://pandas.pydata.org/pandas-docs/stable/remote_data.html#world-bank
http://matplotlib.org/examples/shapes_and_collections/scatter_demo.html
Can you write Python code for it?

# load package under name wb
from pandas.io import wb

# find the codes for the variables of interest
wb.search
wb.search(string='gdp.*capita').iloc[:2]

# specify dates, variables, and countries
start = 2011
# GDP per capita, population, life expectancy
variable_list = ['NY.GDP.PCAP.KD', 'SP.POP.TOTL', 'SP.DYN.LE00.IN']
country_list  = ['US', 'FR', 'JP', 'CN', 'IN', 'BR', 'MX']

# Python understands we need to go to the second line because ( hasn't been closed by )
data = wb.download(indicator=variable_list,
                   country=country_list, start=start, end=start).dropna()

# see what we've got
print(data)

# check the column labels, change to something simpler
print(data.columns)
data.columns = ['gdppc', 'pop', 'le']
print(data)

# scatterplot
# life expectancy v GDP per capita
# size of circles controlled by population

# load packages (ignored if redundant)
import numpy as np
import matplotlib.pyplot as plt

plt.scatter(data['gdppc'], data['le'], s=0.000001*data['pop'], alpha=0.5)
plt.ylabel('Life Expectancy')
plt.xlabel('GDP Per Capita')
plt.show()

# Note:  size of circles based on population


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Comment. Chase likes heatmaps.   The one that follows needs work, but it's an attempt to reproduce the previous graph as a heatmap, where the value of the variable is represented by the color and its intensity.  Dark red:  large postive value.  Dark blue:  large negative value.  Light colors:  small values.
Can you write Python code for it?

import seaborn as sns      # graphics packages
fig, ax = plt.subplots()
sns.heatmap(g_std, linewidths=.5, ax=ax)
ax.set_yticklabels(g_std.index, visible=False)


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Example 1:  Comparing countries by GDP and GDP per capita
GDP is the total value of output produced in a country.  GDP per capita is the per person version:  GDP divided by the population.  Both are expressed in 2011 US dollars with an adjustment for the cost of living.  They come from the World Bank's World Development Indicators, a great source of information about economic and social conditions around the world.  See the World Bank's data catalog.
For more information about accessing World Bank data in Python, see the Pandas documentation.
Can you write Python code for it?

# load packages (redundancy is ok)
import pandas as pd                # data management tools
from pandas.io import wb           # World Bank api
import matplotlib.pyplot as plt    # plotting tools

# look for variable codes
wb.search('gdp.*capita.*const').iloc[:,:2]

# read data from World Bank
var = ['NY.GDP.PCAP.PP.KD', 'NY.GDP.MKTP.PP.KD']          # variable list
iso  = ['USA', 'FRA', 'JPN', 'CHN', 'IND', 'BRA', 'MEX']  # country list (ISO codes)
year = 2014
df = wb.download(indicator=var, country=iso, start=year, end=year)
df

# massage data
df = df.reset_index(level='year', drop=True)
df.columns = ['gdppc', 'gdp']
df['gdp'] = df['gdp']/10**12           # convert to trillions
df['gdppc'] = df['gdppc']/10**3        # convert to thousands
df['order'] = [5, 3, 1, 4, 2, 6, 0]    # reorder countries
df = df.sort(columns='order', ascending=False)

ax = df['gdp'].plot(kind='barh')
ax.set_title('GDP', loc='left', fontsize=14)
ax.set_xlabel('Trillions of US Dollars')
ax.set_ylabel('')

ax = df['gdppc'].plot(kind='barh', alpha=0.5)
ax.set_title('GDP Per Capita', loc='left', fontsize=14)
ax.set_xlabel('Thousands of US Dollars')
ax.set_ylabel('')


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Example:  Employment trends
Not long ago, the US had more people working (a higher fraction of the adult population) than many other developed countries.  Over the past 15 years things have flipped.  The deep question is why, but here we simply report what we know.
The key variables are the employment rate (fraction of people aged 25-54) and the participation rate (fraction either working or unemployed).  They're similar but the latter irons out some of the cyclical fluctuations.  We get the data from FRED, but they get it from the OECD's Main Economic Indicators, which covers mostly rich countries.
Can you write Python code for it?

import pandas as pd                   # data management
import datetime as dt                 # time and date tool
import pandas.io.data as web          # data interface (soon to be moved, see warning)

# countries = AU, CA, CH, DE, ES, EU, EZ, FR, GB, JP, KR, NL, O1 (OECD), SE, US, ZA
countries = ['CA', 'DE', 'GB', 'JP', 'US']
emcodes = ['LREM25TT' + c + 'Q156S' for c in countries]
lrcodes = ['LRAC25TT' + c + 'Q156S' for c in countries]

start_date = dt.datetime(2000, 1, 1)
em = web.DataReader(emcodes, 'fred', start_date)
lr = web.DataReader(lrcodes, 'fred', start_date)
em.columns = countries
lr.columns = countries
em.head(3)

def emplot(em, title):
    fig, ax = plt.subplots()
    em.plot(ax=ax, lw=2)
    ax.set_title(title, fontsize=14, loc='left')
    ax.set_xlabel('')
    ax.set_ylabel('Percent')
    ax.legend(loc='best', fontsize=8, handlelength=2, labelspacing=0.1)

emplot(em, 'Employment rates, age 25-54')
emplot(lr, 'Participation rates, age 25-54')


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Note to self.
In older versions of Pandas, prior to 0.14.1, the option input puts the strike in the index, not as a column of data.
The next two lines check the versions of pandas and python on the off chance we want to check:  print(pd.version),
! python --version
Can you write Python code for it?

# plot call and put prices v strike
plt.plot(calls_strikes, calls_mid, 'r', lw=2, label='calls')
plt.plot(puts_strikes, puts_mid, 'b', lw=2, label='puts')

# prettify it
#plt.axis([120, 250, 0, 50])
plt.axvline(x=atm, color='k', linestyle='--', label='ATM')
plt.legend(loc='best')
plt.show()

# rerun the figure above with different color lines.  Or dashed lines for call and put prices.
# or change the form of the vertical ATM line:  solid? another color?


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Open questions

Why such large differences between countries?
What other country indicators would you like to see?
How would you improve these graphs?

Example 2:  US economic conditions
Fluctuations...  Investors -- and others -- keep a close eye on the state of the economy because it affects the performance and value of firms.  We'll go into this more extensively later, but for now we want to see what the economy has done in the past, especially the recent past.  We use the St Louis Fed's FRED database (Federal Reserve Economic Data), one of the best sources of macroeconomic data.
For more information about accessing FRED data in Python, see the Pandas documentation.
Can you write Python code for it?

# get data from FRED
fred_series = ["GDPC1"]
start_date = dt.datetime(1960, 1, 1)
fred = web.DataReader(fred_series, "fred", start_date)/10**3

# print last 3 data points to see what we've got
# note:  quarterly data
print(fred.tail(3))
# plot
ax = fred.plot(legend=False)
ax.set_title('US Real GDP', fontsize=14, loc='left')
ax.set_xlabel('')
ax.set_ylabel('Trillions of Chained US Dollars')
ax.legend('')

# compute quarterly growth rates
g = 400*fred.pct_change()           # 400 makes this an annual percentage
print(g.tail(5))
# change label
g.columns = ['US GDP Growth']

# get mean and standard deviation
gmean = g.mean()
gstd  = g.std()

g[DATE>=(1990, 1, 1)].plot(kind='bar')

# simple plot
# note the financial crisis:  GDP fell 8% one quarter (annual rate, so really 2%)
ax = g.plot()

# fix up plot
plt.title('US GDP Growth')
#plt.show()

# more complex plot, bar chart for last 6 quarters
# put mean


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  Government debt
One of the traditional ways for a country to get into trouble is to issue so much debt that investors worry about getting paid back.  How much is that?  Hard to say, but we'll look at some debt numbers from the IMF's World Economic Outlook or WEO, a popular source of international data on debt, deficits, and other macroeconomic indicators.  We use numbers for the ratio of government debt to GDP, a standard indicator of public indebtedness.
Given recent events, we pay special attention to Argentina and Greece.
Can you write Python code for it?

%%time
import pandas as pd
import matplotlib.pyplot as plt

url1 = 'http://www.imf.org/external/pubs/ft/weo/2015/01/weodata/'
url2 = 'WEOApr2015all.xls'
url = url1 + url2
weo = pd.read_csv(url, sep='\t', thousands=',', na_values=['n/a', '--'])
weo.shape

list(weo[list(range(12))])

country_guide = weo[['ISO', 'Country']].drop_duplicates().set_index('ISO')

variable_guide = weo[['WEO Subject Code', 'Subject Descriptor', 'Subject Notes']].drop_duplicates().set_index('WEO Subject Code')
variable_guide.head(3)

variables = ['GGXWDG_NGDP']
countries = ['ARG', 'DEU', 'FRA', 'GRC', 'USA']
debt = weo[weo['ISO'].isin(countries) & weo['WEO Subject Code'].isin(variables)]
some = [3] + list(range(9,44))
debt = debt[some].set_index('Country').T.dropna()
debt.head(3)

fig, ax = plt.subplots()
debt.plot(ax=ax, lw=2)
ax.set_title('Ratio of government debt to GDP', fontsize=14, loc='left')
ax.set_ylabel('Government Debt (Percent of GDP)')
ax.legend(loc='best', fontsize=10, handlelength=2, labelspacing=0.15)


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  Japan's aging population
Data from the UN's Population Division.  One of our favorite quotes:

Last year, for the first time, sales of adult diapers in Japan exceeded those for babies.

This is what the numbers look like.  They're UN projections, what they call the "medium variant."
Can you write Python code for it?

url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2015_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLS'
url = url1 + url2 + url3

cols = [2, 4, 5] + list(range(6,28))
#est = pd.read_excel(url, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])

"""
for later:  change cols for the two sources, rename 80+ to 80-84, then concat
#pop = pd.concat([est, prj], axis=0, join='outer')
"""
pop = prj

pop.head()

# rename some variables
pop = pop.rename(columns={'Reference date (as of 1 July)': 'Year',
                          'Major area, region, country or area *': 'Country',
                          'Country code': 'Code'})
# select Japan and years
countries = ['Japan']
years     = [2015, 2025, 2035, 2045, 2055, 2065]
pop = pop[pop['Country'].isin(countries) & pop['Year'].isin(years)]
pop = pop.drop(['Country', 'Code'], axis=1)
pop.head()

pop = pop.set_index('Year').T
pop.head()

ax = pop.plot(kind='bar',
         alpha=0.5, subplots=True, sharey=True, figsize=(6, 8))
for axnum in range(len(ax)):
    ax[axnum].set_title('')

ax[0].set_title('Japanese population by age', fontsize=14, loc='left')


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  Options data
The Pandas package is spinning off the remote data access tools, including FRED, World Bank, and so on.  Here we use the new tool, the Pandas DataReader.  This requires installation:  on the command line, enter
conda install pandas-datareader html5lib
and follow instructions.
You can run commands on the command line from within jupyter by putting a ! at the beginning of the line. To run the commands above you can do enter the following in a code cell:
!conda install -y pandas-datareader html5lib
We'll use the Options tool, which reads in prices of stock options from Yahoo Finance.  The options come with a variety of contract specifications:

Type.  Calls are options to buy, puts are options to sell.
Expiration.  The date when the option expires.
Underlying.  The asset you have a right to buy or sell.  In this data, it's a stock.
Strike price.  The price at which you are able to buy (calls) or sell (puts) the underlying.

We give an illustration using options on Amazon stock.
The first version of this module was written by our very own Spencer Lyon.
Can you write Python code for it?

import pandas_datareader.yahoo.options as yho

"""
supply ticker, get option prices
"""
ticker = 'amzn'
otk = yho.Options(ticker)
exp = otk.expiry_dates     # get expiration dates
exp

# get option prices
cols = [0, 1, 2, 7]
opexp = 2
print('Options with expiry', exp[opexp])
calls = otk.get_call_data(expiry=exp[opexp])[cols]
puts  = otk.get_put_data(expiry=exp[opexp])[cols]

calls.head()

# drop extra index levels
calls = calls.reset_index(level=[1,2,3], drop=True)
puts  = puts.reset_index(level=[1,2,3], drop=True)

# cut off extremes
spot = otk.underlying_price
print('Spot price', spot)
delta = 0.25
calls  = calls[(calls.index >= (1-delta)*spot) & (calls.index <= (1+delta)*spot)]
puts  = puts[(puts.index >= (1-delta)*spot) & (puts.index <= (1+delta)*spot)]

# compute avg of bid and ask
calls['Mid'] = (calls['Bid'] + calls['Ask'])/2
puts['Mid']  = (puts['Bid'] + puts['Ask'])/2

# plot put and call prices
which = 'Mid'

fig, ax = plt.subplots()
calls[which].plot(lw=2, color='blue', alpha=0.6, ax=ax)
puts[which].plot(lw=2, color='magenta', alpha=0.6, ax=ax)
ymin, ymax = ax.get_ylim()
xmin, xmax = ax.get_xlim()
ax.set_title('Prices of Amazon options (bid-ask avg)', fontsize=14, loc='left')
ax.set_ylabel('Option Prices')
ax.set_xlabel('Strike Price')
ax.vlines(x=spot, ymin=ymin, ymax=ymax, linestyle='dashed')
ax.text(1.01*spot, 0.9*ymax, 'Stock price', horizontalalignment='left')
ax.text(0.875*spot, 0.13*ymax, 'Put prices', horizontalalignment='right', color='m')
ax.text(1.125*spot, 0.13*ymax, 'Call prices', horizontalalignment='left', color='b')


I am working on the file "Code/IPython/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  US government bond yields
People often refer to interest rates moving up and down, but in fact the yields (as we call them) move up and down differently at different maturities.  From 2010 to 2015, for examples, yields on bonds with maturities under 2 years were essentially fixed at zero, but yields on higher maturities moved up and down quite a bit.
One challenge with bond yields is getting good data.  Bonds are typically traded over the counter, and those bonds differ in maturity, coupon, and often other features.  Analysts often focus on the interest rates of pure discount "zero-coupon" bonds, which they infer from prices of coupon bonds -- which are, after all, what is generally traded.  This isn't something to go into unless you have a strong interest.  Suffice it to say that we have taken these yields from estimates supplied by the Fed.
This is still a work in progress, but our goal is to produce a movie of bond yields over time.
Can you write Python code for it?

url = 'http://pages.stern.nyu.edu/~dbackus/Data/GSW_nominal_yields.csv'
y = pd.read_csv(url, index_col=0, parse_dates=True)
y[list(range(10))].tail()

# compute mean yields
ybar = y.mean(axis=0)
ystd = y.std(axis=0)
maturities = list(range(1,122))
ybar.index = maturities
ystd.index = maturities

fig, ax = plt.subplots()
ybar.plot(ax=ax)
ystd.plot(ax=ax)
ax.set_xlabel('Maturity in Months')
ax.set_ylabel('Mean and Std Dev of Yield (Annual Percent)')
ax.set_ylim(0)
ax.legend(["Mean", "std. dev."])


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
(Your results may differ, but we really enjoyed that.)
Exercise. Use the set_xlabel() method to add an x-axis label.  What would you choose?  Or would you prefer to leave it empty?
Exercise. Enter ax.set_legend? to access the documentation for the set_legend method.  What options appeal to you?
Exercise.  Change the line width to 2 and the line colors to blue and magenta.  Hint:  Use us.plot? to get the documentation.
Exercise (challenging). Use the set_ylim() method to start the y axis at zero.  Hint:  Use ax.set_ylim? to get the documentation.
Exercise. Create a line plot for the Fama-French dataframe ff that includes both returns.  Bonus points: Add a title with the set_title method.
Multiple subplots
Same idea, but we create a multidimensional ax and apply methods to each component. Here we redo the plots of US GDP and consumption.
Can you write Python code for it?

# this creates a 2-dimensional ax
fig, ax = plt.subplots(nrows=2, ncols=1, sharex=True)
print('Object ax has dimension', len(ax))

# now add some content
fig, ax = plt.subplots(nrows=2, ncols=1, sharex=True, sharey=True)

us['gdp'].plot(ax=ax[0], color='green')   # first plot
us['pce'].plot(ax=ax[1], color='red')     # second plot


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Comment.  In the previous cell, we used the print() function to produce output.  Here we just put the name of the dataframe.  The latter displays the dataframe -- and formats it nicely -- if it's the last statement in the cell.
Can you write Python code for it?

# Fama-French
import pandas.io.data as web

# read annual data from website and rename variables
ff = web.DataReader('F-F_Research_Data_factors', 'famafrench')[1]
ff.columns = ['xsm', 'smb', 'hml', 'rf']
ff['rm'] = ff['xsm'] + ff['rf']
ff = ff[['rm', 'rf']]     # extract rm and rf (return on market, riskfree rate, percent)
ff.head(5)


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Comment.  When you run the code cell above, its output appears below it.
Exercise. Enter pd.read_csv? in the empty cell below.  Run the cell (Cell at the top, or shift-enter).  Do you see the documentation?  This is the Jupyter version of help in Spyder's IPython console.
Can you write Python code for it?

# This is an IPython command.  It puts plots here in the notebook, rather than a separate window.
%matplotlib inline


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Comment. Both of these statements must be in the same cell.
Can you write Python code for it?

# Fama-French example
fig, ax = plt.subplots()
ff.plot(ax=ax,
        kind='line',                 # line plot
        color=['blue', 'magenta'],   # line color
        title='Fama-French market and riskfree returns')


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Comment. For aficionados, the always tasteful xkcd style.
Can you write Python code for it?

plt.style.use('ggplot')
gdp_bar()

plt.xkcd()
gdp_bar()


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Comment. Ignore the seaborn styles, that's a package we don't have yet.
Exercise. Try another one by editing the code below.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Comment. Yikes!  That's horrible!  What can we do about it?
Let's make the figure taller.  The figsize argument has the form (width, height).  The default is (6, 4).  We want a tall figure, so we need to increase the height setting.
Can you write Python code for it?

fig.

# make the plot taller
fig, ax = plt.subplots(figsize=(4, 13))  # note figsize
pisa['Math'].plot(kind='barh', ax=ax)
ax.set_title('PISA Math Score', loc='left')


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Create dataframes to play with

US GDP and consumption
World Bank GDP per capita for several countries
Fama-French equity returns
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Digression:   Graphing in Excel
Remind yourself that we need to choose:

Data.  Typically a block of cells in a spreadsheet.
Chart type.  Lines, bars, scatter, or something else.
x and y variables.  What is the x axis?  What is y?

We'll see the same in Matplotlib.
Approach #1:  Apply plot() method to dataframe
Good simple approach, we use it a lot.  It comes with some useful defaults:

Data.  The whole dataframe.
Chart type.  We have options for lines, bars, or other things.
x and y variables.  By default, the x variable is the dataframe's index and the y variables are all the columns of the dataframe.

All of these things can be changed, but this is the starting point.
Let's do some examples, see how they work.
US GDP and consumption
Can you write Python code for it?

# try this with US GDP
us.plot()

# do GDP alone
us['gdp'].plot()

# bar chart
us.plot(kind='bar')


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Examples
We conclude with examples that take the data from the previous chapter and make better graphs with it.
Student test scores (PISA)
The international test scores often used to compare quality of education across countries.
Can you write Python code for it?

# data input
import pandas as pd
url = 'http://dx.doi.org/10.1787/888932937035'
pisa = pd.read_excel(url,
                     skiprows=18,     # skip the first 18 rows
                     skipfooter=7,    # skip the last 7
                     parse_cols=[0,1,9,13], # select columns
                     index_col=0,     # set index = first column
                     header=[0,1]     # set variable names
                     )
pisa = pisa.dropna()                          # drop blank lines
pisa.columns = ['Math', 'Reading', 'Science'] # simplify variable names

# bar chart of math scores
fig, ax = plt.subplots()
pisa['Math'].plot(kind='barh', ax=ax)


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise.  Create the same graph with this statement at the top:
python
plt.style.use('fivethirtyeight')
(Once we execute this statement, it stays executed.)
Comment. We can get a list of files from plt.style.available.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise.  Let's see if we can dress up the histogram a little.  Try adding, one at a time, the arguments title='Fama-French returns', grid=True, and legend=False.  What does the documentation say about them?  What do they do?
Exercise.  What do the histograms tell us about the two returns?  How do they differ?
Exercise.  Use the World Bank dataframe wbdf to create a bar chart of GDP per capita, the variable 'gdppc'. Bonus points: Create a horizontal bar chart. Which do you prefer?
Approach #2:  the plot(x,y) function
Here we plot variable y against variable x.  This comes closest to what we would do in Excel:  identify a dataset, a plot type, and the x and y variables, then press play.
Can you write Python code for it?

# import pyplot module of Matplotlib
import matplotlib.pyplot as plt

plt.plot(us.index, us['gdp'])


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise.  Let's see if we can teach ourselves the rest:

Add the argument kind='bar' to convert this into a bar chart.
Add the argument alpha=0.65 to the bar chart.  What does it do?
What would you change in the bar chart to make it look better?  Use the help facility to find options that might help.  Which ones appeal to you?

Exercise (somewhat challenging). Use the same approach to reproduce our earlier histograms of the Fama-French series.
Quick review of the bidding
Take a deep breath.  We've covered a lot of ground, let's take stock.
We looked at three ways to use Matplotlib:

Approach #1:  apply plot method to dataframe
Approach #2:  use plot(x,y) function
Approach #3:  create fig, ax objects, apply plot methods to them

Same result, different syntax.  This is what each of them looks like applied to US GDP:
```python
us['gdp'].plot()                   # Approach #1
plt.plot(us.index, us['gdp'])      # Approach #2
fig, ax = plt.subplots()           # Approach #3
ax.plot(us.index, us['gdp'])
```
Bells and whistles
Adding things to graphs
We have lots of choices.  Here's an example.
Can you write Python code for it?

fig, ax = plt.subplots()

us.plot(ax=ax)
ax.set_title('US GDP and Consumption', fontsize=14, loc='left')
ax.set_ylabel('Billions of 2013 USD')
ax.legend(['Real GDP', 'Consumption'], loc=0)  # more descriptive variable names
ax.set_xlim(2002.5, 2013.5)                    # expand x axis limits
ax.tick_params(labelcolor='red')               # change tick labels to red
ax.set_ylim(0)


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. Create the same graph for the Reading score.
World Bank data
We'll use World Bank data for GDP, GDP per capita, and life expectancy to produce a few graphs and illsutrate some methods we haven't seen yet.

Bar charts of GDP and GDP per capita
Scatter plot (bubble plot) of life expectancy v GDP per capita
Can you write Python code for it?

# load packages (redundancy is ok)
import pandas as pd                   # data management tools
from pandas.io import wb              # World Bank api
import matplotlib.pyplot as plt       # plotting tools

# variable list (GDP, GDP per capita, life expectancy)
var = ['NY.GDP.PCAP.PP.KD', 'NY.GDP.MKTP.PP.KD', 'SP.DYN.LE00.IN']
# country list (ISO codes)
iso = ['USA', 'FRA', 'JPN', 'CHN', 'IND', 'BRA', 'MEX']
year = 2013

# get data from World Bank
df = wb.download(indicator=var, country=iso, start=year, end=year)

# massage data
df = df.reset_index(level='year', drop=True)
df.columns = ['gdppc', 'gdp', 'life'] # rename variables
df['pop']  = df['gdp']/df['gdppc']    # population
df['gdp'] = df['gdp']/10**12          # convert to trillions
df['gdppc'] = df['gdppc']/10**3       # convert to thousands
df['order'] = [5, 3, 1, 4, 2, 6, 0]   # reorder countries
df = df.sort_values(by='order', ascending=False)
df

# GDP bar chart
fig, ax = plt.subplots()
df['gdp'].plot(ax=ax, kind='barh', alpha=0.5)
ax.set_title('GDP', loc='left', fontsize=14)
ax.set_xlabel('Trillions of US Dollars')
ax.set_ylabel('')

# ditto for GDP per capita (per person)
fig, ax = plt.subplots()
df['gdppc'].plot(ax=ax, kind='barh', color='m', alpha=0.50)  # 'm' == 'magenta'
ax.set_title('GDP Per Capita', loc='left', fontsize=14)
ax.set_xlabel('Thousands of US Dollars')
ax.set_ylabel('')


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. Enter us.plot(kind='bar') and us.plot.bar() in separate cells.  Show that they produce the same bar chart.
Exercise.  Add each of these arguments, one at a time, to us.plot():

kind='area'
subplots=True
sharey=True
figsize=(3,6)
ylim=(0,16000)

What do they do?
Exercise. Type us.plot? in a new cell.  Run the cell (shift-enter or click on the run cell icon).  What options do you see for the kind= argument?  Which ones have we tried?  What are the other ones?
Fama-French asset returns
Can you write Python code for it?

# now try a few things with the Fama-French data
ff.plot()

ff.plot()


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. Experiment with
python
plt.bar(us.index, us['gdp'],
        align='center',
        alpha=0.65,
        color='red',
        edgecolor='green')
Play with the arguments one by one to see what they do.  Or use plt.bar? to look them up.  Add comments to remind yourself. Bonus points:  Can you make this graph even uglier?
Can you write Python code for it?

# we can also add things to plots
plt.plot(us.index, us['gdp'])
plt.plot(us.index, us['pce'])

plt.title('US GDP', fontsize=14, loc='left') # add title
plt.ylabel('Billions of 2009 USD')           # y axis label
plt.xlim(2002.5, 2013.5)                     # shrink x axis limits
plt.tick_params(labelcolor='red')            # change tick labels to red
plt.legend(['GDP', 'Consumption'])           # more descriptive variable names


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. Make the bubble a little larger.
Styles (optional)
Graph settings you might like.
Can you write Python code for it?

# We'll look at this chart under a variety of styles.
# Let's make a function so we don't have to repeat the
# code to create
def gdp_bar():
    fig, ax = plt.subplots()
    df['gdp'].plot(ax=ax, kind='barh', alpha=0.5)
    ax.set_title('Real GDP', loc='left', fontsize=14)
    ax.set_xlabel('Trillions of US Dollars')
    ax.set_ylabel('')

gdp_bar()


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. Show that we get the output from us.plot.bar().
Can you write Python code for it?

us.plot

# scatter plot
# we need to be explicit about the x and y variables: x = 'gdp', y = 'pce'
us.plot.scatter('gdp', 'pce')


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. What do each of the arguments do in the code below?
Can you write Python code for it?

ff.plot(kind='hist', bins=20, subplots=True)

# "smoothed" histogram
ff.plot(kind='kde', subplots=True, sharex=True)    # smoothed histogram ("kernel density estimate")


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. What do we have here?  What type are fig and ax?
We say fig is a figure object and ax is an axis object.  This means:

fig is a blank canvas for creating a figure.
ax is everything in it:  axes, labels, lines or bars, and so on.

Exercise. Use tab completion to see what methods are available for fig and ax.  What do you see?  Do you feel like screaming?
Can you write Python code for it?

# let's try that again, this time with content
# create objects
fig, axe = plt.subplots()

# add things by applying methods to ax
us.plot(ax=axe)


I am working on the file "Code/IPython/bootcamp_graphics.ipynb".
The first task is:
Exercise. What is the x variable here?  The y variable?
Can you write Python code for it?

# we can do two lines together
plt.plot(us.index, us['gdp'])
plt.plot(us.index, us['pce'])

# or a bar chart
plt.bar(us.index, us['gdp'], align='center')


I am working on the file "Code/IPython/bootcamp_indicators.ipynb".
The first task is:
Business cycle indicators
We assess the state of the US economy with a collection of monthly indicators that (mostly) move up and down with the economy. We get the data from FRED, the St Louis Fed's popular data collection. There are lots of indicators to choose from, but we use

INDPRO:  industrial production
PAYEMS:  nonfarm employment
AWHMAN:  average weekly hours worked in manufacturing
PERMIT:  premits for new housing
NAPM:  purchasing managers index
RSXFS:  retail sales (excluding food services)

For each indicator the first term is the FRED code, the second a description.  You can find more about this kind of thing in our  Global Economy book, chapter 11.  Also in bank reports, which review this kind of information constantly.
Can you write Python code for it?

# get data from FRED
import pandas as pd
import pandas.io.data as web           # web interface with FRED
import datetime as dt                  # handles dates

# get data
indicators = ['INDPRO', 'PAYEMS', 'AWHMAN', 'PERMIT', 'NAPM', 'RSXFS']
start_date = dt.datetime(1970, 1, 1)
inds = web.DataReader(indicators, "fred", start_date)
end = inds.index[-1]

# yoy growth rates
g = inds.pct_change(periods=12).dropna()
# standardize
gs = (g - g.mean()) / g.std()

# correlations
gs.corr()

# plot
fig, ax = plt.subplots()
gs.plot(ax=ax)
ax.set_title('Economic indicators', fontsize=14, loc='left')
ax.set_ylabel('Standard deviations from mean')
ax.set_xlabel('')
ax.hlines(y=0, xmin=start_date, xmax=end, linestyles='dashed')
#ax.legend().set_visible(False)
ax.legend(loc='upper left', fontsize=10, handlelength=2, labelspacing=0.15)

# focus on recent past
recent_date = dt.datetime(2005, 1, 1)
grecent= gs[gs.index>=recent_date]

fig, ax = plt.subplots()
grecent.plot(ax=ax)
ax.set_title('Zoom in on recent past', fontsize=14, loc='left')
ax.set_ylabel('Standard deviations from mean')
ax.set_xlabel('')
ax.hlines(y=0, xmin=recent_date, xmax=end, linestyles='dashed')
ax.legend(loc='upper left', fontsize=10, handlelength=2, labelspacing=0.15)


I am working on the file "Code/IPython/bootcamp_indicators.ipynb".
The first task is:
Radar plot ??
Country indicators:  Opportunities in Southern Africa
Zambeef is a successful meat distributor located in Zambia.  Their leadership wonders whether their operation can be expanded to include neighboring Botswana and Tanzania.  We collect a number of economic and institutional indicators to assess the business climates in the three countries and ask:

What features of an economy are import to this business?
What indicators of these features can we find in the World Bank's data?
How do the three countries compare on these features?
What locations look the most attractive to you?

We start by looking through the World Bank's enormous collection of country indicators and using Pandas' World Bank API to access the numbers.
Can you write Python code for it?

from pandas.io import wb               # World Bank api

# read data from World Bank
iso  = ['ZMB', 'BWA', 'TZA']           # country list (ISO codes)
var  = ['NY.GDP.PCAP.PP.KD',           # GDP per person
        'SP.POP.TOTL',                 # population
        'IC.BUS.EASE.XQ',              # ease of doing business (rank of 189)
        'IS.ROD.PAVE.ZS',              # paved roads (percent of total)
        'SE.ADT.LITR.ZS']              # adult literacy (15 and up)
year = 2014
df = wb.download(indicator=var, country=iso, start=2005, end=2014)
df


I am working on the file "Code/IPython/bootcamp_links-scraping.ipynb".
The first task is:
Notice that we can choose to interact with either the json or the csv file. They both give us lots of columns (and not very many rows) though and we might want to choose a smaller set of the data. We will show that we can do this using the api described in the second link
Can you write Python code for it?

# First lets select subset of data
base_url_json += "?$select=boys_toilets,girls_toilets,geolocation,name_of_school,level_of_education"

df_json2 = pd.read_json(base_url_json)
df_json2.head()

# Now let's raise the number of observations
base_url_json += "&$limit=5000"

df_json3 = pd.read_json(base_url_json)

df_json3.head(5)

df_json.shape

df_json2.shape

df_json3.shape

baseurl = 'https://www.opendata.go.ke/resource/p452-xb7c.json'
baseurl += '?$select=boys_toilets,girls_toilets,geolocation,name_of_school,level_of_education'
baseurl += '&$limit=50000'

df_big = pd.read_json(baseurl)

df_big.shape


I am working on the file "Code/IPython/bootcamp_links-scraping.ipynb".
The first task is:
Question. What do we have here?  A list of length 10?  Whose elements are dataframes?  Evidently this reads in all the tables from the page into dataframes and collects them in a list.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_links-scraping.ipynb".
The first task is:
SODA API
https://dev.socrata.com/consumers/examples/pandas-and-jupyter-notebook.html
https://dev.socrata.com/docs/queries/
Now that we've talked about this, let's look at a dataset and try some things.
Can you write Python code for it?

base_url_json = "https://www.opendata.go.ke/resource/p452-xb7c.json"

df_json = pd.read_json(base_url_json)
df_json.head(3)

base_url_csv = "https://www.opendata.go.ke/resource/p452-xb7c.csv"

df_csv = pd.read_csv(base_url_csv)
df_csv.columns = df_csv.columns.str.replace(" ", "_").str.lower()
df_csv.head(3)


I am working on the file "Code/IPython/bootcamp_links-scraping.ipynb".
The first task is:
Scanning url's
It is important to think about what you are doing when you get data from online. It turns out to be useful to think about how you can specify the type of information that you want from a website. This is done through a site's API.
We will talk about two examples:

Yahoo Finance data (Itamar's suggestion)
Government Open data (through Socrata Open Data API SODA)

Arguments to a url are passed after a ? and multiple arguments are separated using &. You will see what we mean below.
Yahoo finance
http://finance.yahoo.com/
Can you write Python code for it?

# let's see if we're lucky
dfs = pd.read_html("http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices")

len(dfs)

dfs[9].head(10)


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:

Selecting variables and observations
The word selection refers to choosing a subset of variables or observations using their labels or index.  Similar methods are sometimes referred to as slicing, subsetting, indexing, or filtering.  We'll treat the terms as synonymous.
There are lots of ways to do this.  Mostly we do "Boolean" selection, which we address in the next section.  We review more direct options here, mostly at high speed because they're not things we use much.
In the outline below, df is a dataframe, var and varn are variable names, vlist = ['var1', 'var2'] is a list of variable names, and nlist = [0, 3, 4] is a list of numerical variable or observation indexes, n1 and n2 are integers, and bools ia a list or pandas Series of booleans (True and False).  Some of the basic selection/indexing/slicing methods have the form:

df[var] extracts a variable -- a series, in other words.
df[vlist] extracts a new dataframe consisting of the variables in vlist.
df[nlist] does the same thing.
df[bools]: extracts each row where the corresponding element in bools is true. len(bools) must be equal to df.size[0]
df[n1:n2] extracts observations n1 to n2-1, the traditional slicing syntax.

We find the last one confusing:  it extracts rows, not columns.  Pandas guru Wes McKinney notes: "This might seem inconsistent to some readers."  Yup!  We don't do it much, partly for that reason.

The Pandas docs push the loc and iloc methods.  We'll ignore them -- we don't use them much -- but if you're interested, see the docs.
Can you write Python code for it?

# we create a small dataframe to experiment with
small = weo.head()
small


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:

String methods
We can treat variables as strings in Pandas in much the same way we dealt with strings in core Python.  Run the code below to remind yourself how this works.
Can you write Python code for it?

dollars = '$123.45'
print('Type of variable dollars:', type(dollars))
num = dollars.replace('$', '')
num = float(num)
print('Type of variable num:', type(num))


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment.  Replace automatically updates the dtypes.  Here the double dots led us to label the variables as objects.  After the replace, they're now floats, as they should be.
Can you write Python code for it?

docs.dtypes.head()

docsna = docs.replace(to_replace=['..'], value=[None])
docsna.dtypes.head()


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment.  Some people prefer to use the numpy nan.  Here's an example.  The only advantage is that we avoid possible conflicts with other uses of the value None.
Can you write Python code for it?

docs.replace(to_replace=['..'], value=[np.nan]).head()

# assign back to docs
docs = docs.replace(to_replace=['..'], value=[np.nan])


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Just for fun, here's a bar graph of the result.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Now let's do the same for some of the other entries and save them in new variables.
Can you write Python code for it?

media = ['None', 'Twitter', 'Facebook', 'Blog']
oldep = ep.copy()

vnames = []
for x in media:
    newname = 'Media' + ':' + x
    vnames.append(newname)
    ep[newname] = ep['Media'].str.contains(x)*1

vnames


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. This has several issues:

The variables run across rows with observations labeled 1980, 1981, etc across the top.  We saw the same problem in the previous example.
If we run the first version of the read_csv statement, the data columns (1980, 1981, etc) have dtype object.  A little work suggests that this is because they include commas marking thousands.
The entries labeled n/a need to be marked as missing values.

We can solve the last two in the read_csv function by deleting the hash -- which is what we see in the second read_csv statement.  The other one takes some work.
Question.  Can we transpose the whole thing to get the data running down columns?
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Unlike the string methods we described earlier, this use of replace affects complete entries, not elements of string entries.  For example, suppose we tried to replace the periods in decimal numbers with an asterisk.  We could try the following, but it doesn't work:  the decimal numbers  don't change.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We can do the same thing with countries.  If we want to choose two variables and three countries, the code looks like:
Can you write Python code for it?

variables = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
countries = ['ARG', 'DEU', 'GRC']
weo_sub = weo[weo['WEO Subject Code'].isin(variables) & weo['ISO'].isin(countries)]
weo_sub


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We did everything here in one line:  replace the dollar sign with a string method, then converted to float using astype.  If you think this is too dense, you might break it into two steps.
Example. Here we strip off the numbers at the end of the indexes in the OECD docs dataframe.  This involves some experimentation:

Play with the rsplit method to see how it works.
Apply rsplit to the example country = 'United States 1'.
Use a string method to do this to all the entries of the variable Country.
Can you write Python code for it?

# try this with an example first
country = 'United States 1'

# get documentation for the rsplit method
#country.rsplit?

# an example
country.rsplit()


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We're choosing 2 variables from 45, so there are lots of Falses.
Can you write Python code for it?

# this time let's use the result of isin for selection
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
weo[weo['WEO Subject Code'].isin(vlist)].head(6)

# we've combined several things in one line
comparison = weo['WEO Subject Code'].isin(vlist)
selection  = weo[comparison]
selection.head(6)


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. You might want to think about this a minute.  Or two.
Can you write Python code for it?

# create new df of just these variables
media = ep[vnames]
media.head()

# count them with the sum method
media_counts = media.sum()
media_counts


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comments.

We've now done what we described when we applied the want operator.
This is a go-to method.  Circle it for later reference.
This is a go-to method.  Circle it for later reference.

Exercise. Use the isin method to extract Gross domestic product in US dollars for China, India, and the United States.  Assign the result to the dataframe gdp.  Hint:  You can adapt the code we just ran. The variable code is NGDPD.  The country codes are CHN, IND, and USA.
Exercise (challenging). Plot the variable gdp['2015'] as a bar chart.  What would you say it needs?

The contains method
Another useful one.  The contains string method for series identifies observations that contain a specific string.  If yes, the observation is labelled True, if no, False.  A little trick converts the True/False outcomes to ones and zeros.
We apply it to the Media variable of the Entry Poll dataframe ep.  You may recall that this variable could have more than one response.  We tease them apart with the contains method.  Our want is to have a yes/no variable for each response.
Can you write Python code for it?

# recall
ep['Media'].head(10)

# the contains method
ep['Media'].str.contains('Twitter').head(10)


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comments.

Note that we need two str's here:  one to do the split, the other to extract the first element.
For reasons that mystify us, we ran into problems when we used maxsplit=1, but it works with n=1.
This is probably more than you want to know, but file away the possibilities in case you need them.

Missing values
It's important to label missing values, so that Pandas doesn't interpret entries as strings.  Pandas is also smart enough to ignore things labeled missing when it does calculations or graphs.  If we compute, for example, the mean of a variable, the default is to ignore missing values.
We've seen that we can label certain entries as missing values in read statements:  read_csv, read_excel, and so on.  Here we do it directly, mostly to remind ourselves what's involved.
Marking missing values
Example. The docs dataframe contains a number of instances of .. (double period).  How can we mark them as missing values?
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comments.  Here we have a couple issues.

The first column includes a space and a number:  Australia 1, Chile 3, etc.  We care about this because when we plot the data across countries, the country labels are going to be country names, so we want them in a better form than this.
The ..'s in the sheet lead us to label any column that includes them as dtype object.  Here we want to label them as missing values.
If we want to plot each country against time, then we'll need to switch the rows and columns somehow, so that the x axis in the plot (the year) is the index and not the column label.

One more thing before we proceeed:  change the name of the country variable.
Can you write Python code for it?

names = list(docs)
docs = docs.rename(columns={names[0]: 'Country'})
docs.head(2)


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Comments. This is mostly text data, which means it's assigned the dtype object.  Which is fine.  But there are two things that would make the data easier to work with:

The column names are excessively verbose.  This one's easy:  We replace them with single words.  Which we do below.
The second one is harder.  Two of the questions -- social media and special topics -- say "mark all that apply."  In the spreadsheet, we have a list of every choice the person checked.  Our want is to count the number of each type of response.  For example, we might want a bar chart that gives us the number of each response.  The question is how we get there.
Can you write Python code for it?

# rename variables
newnames = ['time', 'program', 'career', 'programming', 'stats', 'media',
             'other', 'major', 'data', 'why', 'topics']
newnames = [name.title() for name in newnames]
ep.columns = newnames
ep.head()

# check multi-response question to see what we're dealing with
ep['Media'].head(20)


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Example. Let's try each of these in a different cell and see what they do:

small[['ISO', 'Units']]
small[[0, 4]]
small['2011']
small[1:3]

Can you explain the results?
Can you write Python code for it?

small[['ISO', 'Units']]

small[[0, 4]]

small['2011']

small['2011'][3]

small[1:3]

small[[False, True, True, False, False]]


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Example:  World Economic Outlook
The IMF's World Economic Outlook database contains a broad range of macroeconomic data for a large number of countries.  It's updated twice a year and is a go-to source for things like current account balances (roughly, the trade balance) and government debt and deficits.  It also has a few quirks, as we'll see.
Example. Run the following code as is, and with the thousands and na_values parameters commented out.  How do the dtypes differ?
Can you write Python code for it?

url1 = 'http://www.imf.org/external/pubs/ft/weo/2015/02/weodata/'
url2 = 'WEOOct2015all.xls'
url = url1 + url2
weo = pd.read_csv(url, sep='\t',
                  usecols=[1,2,3,4,6,40,41,42,43,44],
                  thousands=',',
                  na_values=['n/a', '--']
                 )

print('Variable dtypes:\n', weo.dtypes, sep='')
weo.head()


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Exercise.

Construct a list of countries with countries = weo[['ISO', 'Country']]; that is, without applying the drop_duplicates method.  How large is it?  How many duplicates have we dropped?
What are the country codes (ISO) for Argentina and the United States?
What are the variable codes (WEO Subject Code) for government debt (gross debt, percent of GDP) and net lending/borrowing (also percent of GDP)?

Comment. Now that we have the country and variable codes, we can be more explicit about what we want.  We want observations with those country and variable codes.
We work up to the solution one step at a time.
Comparisons for series
We can construct comparisons for series (dataframe columns) much as we did with simple variables.  The difference is that we get a complete column of True/False responses, not just one.
Mutiple comparisons have a different syntax than we saw earlier: and is replaced by &, and or is replaced by |.  And when we have more than one comparison, we need to enclose them in parentheses.
Examples.  Consider the comparisons:

small['Units'] == 'National currency'
small['2011'] >= 100
(small['Units'] == 'National currency') & (small['2011'] >= 100)
(small['Units'] == 'National currency') | (small['2011'] >= 100)

Remind yourself what the & and | do.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Exercise. Construct dataframes for which

small['Units'] does not equal 'National currency'.
small['Units'] equals 'National currency' and small['2011'] is greater than 100.

The isin method
Pay attention now, this is really useful.  Suppose we want to extract the data for which weo['ISO'] == 'ARG' (Argentina) or weo['ISO'] == 'GRC' (Greece).  We could do that by combining the comparisons:
python
(weo['ISO'] == 'ARG') | (weo['ISO'] == 'GRC')
Remind youself that | stands for "or."  (What do we use for "and"?)
A simpler approach is to apply the isin method to a variable.  This sets the comparison equal to True if the value of the observation is of weo['ISO'] equals any element in a list.  We could do the same thing using mulitple comparisons, but this is a lot easier.
Let's see how this works.
Example.  Let's apply the same logic to variable codes.  If we want to extract the observations with codes
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
we would use
Can you write Python code for it?

vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
weo['WEO Subject Code'].isin(vlist).head(45)


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Pandas string methods. We can do the same thing to all the observations of a variable with so-called string methods.  We append .str to a variable in a dataframe and then apply the string method of our choice.  If this is part of converting a number-like entry that has mistakenly been given dtype object, we then convert its dtype with the astype method.
Example.  Let's use a string method to fix the item_price variable in the Chipotle dataframe.  This has three parts:

Use the method str to identify this as a string method.
Apply the string method of our choice (here replace) to fix the string.
Use the astype method to convert the fixed-up string to a float.

We start by making a copy of the chp dataframe that we can experiment with.
Can you write Python code for it?

chp.head()

chpnum = chp.copy()
print('Original dtype:', chpnum['item_price'].dtype)
# create a copy of the df to play with

# delete dollar signs
chpnum['item_price'].str.replace('$', '').head()

# delete dollar signs, convert to float, and assign back to chpnum
chpnum['item_price'] = chpnum['item_price'].str.replace('$', '').astype(float)
print('New dtype:', chpnum['item_price'].dtype)

# assign back to chp for future use
chp = chpnum


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
What to do.  We use the replace method on the whole dataframe.  To mark something as missing, we replace it as None, which Pandas interprets as missing and labels NaN.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Working with missing values
Can you write Python code for it?

# grab a variable to play with
var = docsna[2013].head(10)
var

# which ones are missing ("null")?
var.isnull()

# which ones are not missing ("not null")?
var.notnull()

# drop the missing
var.dropna()


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:

String methods
We can treat variables as strings in Pandas in much the same way we dealt with strings in core Python.  Run the code below to remind yourself how this works.
Can you write Python code for it?

dollars = '$123.45'
print('Type of variable dollars:', type(dollars))
num = dollars.replace('$', '')
num = float(num)
print('Type of variable num:', type(num))


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment.  Replace automatically updates the dtypes.  Here the double dots led us to label the variables as objects.  After the replace, they're now floats, as they should be.
Can you write Python code for it?

docs.dtypes.head()

docsna = docs.replace(to_replace=['..'], value=[None])
docsna.dtypes.head()


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment.  Some people prefer to use the numpy nan.  Here's an example.  The only advantage is that we avoid possible conflicts with other uses of the value None.
Can you write Python code for it?

docs.replace(to_replace=['..'], value=[np.nan]).head()

# assign back to docs
docs = docs.replace(to_replace=['..'], value=[np.nan])


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. Not quite, we only want to split once.
Can you write Python code for it?

# what about this?
country.rsplit(maxsplit=1)

# one more step, we want the first component of the list
country.rsplit(maxsplit=1)[0]

docs["Country"].head()

# now do this for the variable Country
#docs['Country'].str.rsplit(maxsplit=1).str[0].head()   # explain why this doesn't work
docs['Country'].str.rsplit(n=1).str[0].head()

# some people prefer the get method to slicing
docs['Country'].str.rsplit(n=1).str.get(0).head()

# now assign it to newdocs and see what we have
newdocs = docs.copy()
newdocs['Country'] = newdocs['Country'].str.rsplit(n=1).str.get(0)
newdocs.head()

# assign it back to docs for future use
docs = newdocs


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. Note that the variable item_price has dtype object. The reason is evidently the dollar sign.  We'd prefer to have it as a number, specifically a float.
Example:  Data Bootcamp entry poll
This is the poll we did at the start of the course.  Responses were collected in a Google spreadsheet, which we converted to a csv and uploaded to our website.
Can you write Python code for it?

pd.set_option("display.width", 80)

import pandas as pd
url1 = 'http://pages.stern.nyu.edu/~dbackus/Data/'
url2 = 'Data-Bootcamp-entry-poll_s16.csv'
url = url1 + url2
ep = pd.read_csv(url, header=0)
print('Dimensions:', ep.shape)
print('\nData types:\n', ep.dtypes, sep='')
ep.head(2)


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. Now let's do the same for some of the other entries and save them in new variables.
Can you write Python code for it?

media = ['None', 'Twitter', 'Facebook', 'Blog']
oldep = ep.copy()

vnames = []
for x in media:
    newname = 'Media' + ':' + x
    vnames.append(newname)
    ep[newname] = ep['Media'].str.contains(x)*1

vnames

media = ep[vnames]
media.head()

media_counts = media.sum()
media_counts

media_counts.plot.barh()


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. This has several issues:

The variables run across rows with observations labeled 1980, 1981, etc across the top.  We saw the same problem in the previous example.
If we run the first version of the read_csv statement, the data columns (1980, 1981, etc) have dtype object.  A little work suggests that this is because they include commas marking thousands.
The entries labeled n/a need to be marked as missing values.

We can solve the last two in the read_csv function by deleting the hash -- which is what we see in the second read_csv statement.  The other one takes some work.
Question.  Can we transpose the whole thing to get the data running down columns?
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. We can do the same thing with countries.  If we want to choose two variables and three countries, the code looks like:
Can you write Python code for it?

variables = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
countries = ['ARG', 'DEU', 'GRC']
weo_sub = weo[weo['WEO Subject Code'].isin(variables) & weo['ISO'].isin(countries)]
weo_sub


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. We did everything here in one line:  replace the dollar sign with a string method, then converted to float using astype.  If you think this is too dense, you might break it into two steps.
Example. Here we strip off the numbers at the end of the indexes in the OECD docs dataframe.  This involves some experimentation:

Play with the rsplit method to see how it works.
Apply rsplit to the example country = 'United States 1'.
Use a string method to do this to all the entries of the variable Country.
Can you write Python code for it?

# try this with an example first
country = 'United States 1'

# get documentation for the rsplit method
#country.rsplit?

# an example
country.rsplit()


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comment. We usually don't have to worry about this, Pandas takes care of missing values automatically.
Comment. Let's try a picture to give us a feeling of accomplishment. What else would you say we need?  How would we get it?
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comments.

We've now done what we described when we applied the want operator.
This is a go-to method.  Circle it for later reference.
This is a go-to method.  Circle it for later reference.

Exercise. Use the isin method to extract Gross domestic product in US dollars for China, India, and the United States.  Assign the result to the dataframe gdp.
Exercise (challenging). Plot the variable gdp['2015'] as a bar chart.  What would you say it needs?

The contains method
Another useful one.  The contains string method for series identifies observations that contain a specific string.  If yes, the observation is labelled True, if no, False.  A little trick converts the True/False outcomes to ones and zeros.
We apply it to the Media variable of the Entry Poll dataframe ep.  You may recall that this variable could have more than one response.  We tease them apart with the contains method.  Our want is to have a yes/no variable for each response.
Can you write Python code for it?

# recall
ep['Media'].head(10)

# the contains method
ep['Media'].str.contains('Twitter').head(10)


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comments.

Note that we need two str's here:  one to do the split, the other to extract the first element.
For reasons that mystify us, we ran into problems when we used maxsplit=1, but it works with n=1.
This is probably more than you want to know, but file away the possibilities in case you need them.

Missing values
It's important to label missing values, so that Pandas doesn't interpret entries as strings.  Pandas is also smart enough to ignore things labeled missing when it does calculations or graphs.  If we compute, for example, the mean of a variable, the default is to ignore missing values.
We've seen that we can label certain entries as missing values in read statements:  read_csv, read_excel, and so on.  Here we do it directly, mostly to remind ourselves what's involved.
Marking missing values
Example. The docs dataframe contains a number of instances of .. (double period).  How can we mark them as missing values?
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comments.  Here we have a couple issues.

The first column includes a space and a number:  Australia 1, Chile 3, etc.  We care about this because when we plot the data across countries, the country labels are going to be country names, so we want them in a better form than this.
The ..'s in the sheet lead us to label any column that includes them as dtype object.  Here we want to label them as missing values.
If we want to plot each country against time, then we'll need to switch the rows and columns somehow, so that the x axis in the plot (the year) is the index and not the column label.

One more thing before we proceeed:  change the name of the country variable.
Can you write Python code for it?

names = list(docs)
docs = docs.rename(columns={names[0]: 'Country'})
docs.head(2)


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Comments. This is mostly text data, which means it's assigned the dtype object.  Which is fine.  But there are two things that would make the data easier to work with:

The column names are excessively verbose.  This one's easy:  We replace them with single words.  Which we do below.
The second one is harder.  Two of the questions -- social media and special topics -- say "mark all that apply."  In the spreadsheet, we have a list of every choice the person checked.  Our want is to count the number of each type of response.  For example, we might want a bar chart that gives us the number of each response.  The question is how we get there.
Can you write Python code for it?

# rename variables
newnames = ['time', 'program', 'career', 'programming', 'stats', 'media',
             'other', 'major', 'data', 'why', 'topics']
newnames = [name.title() for name in newnames]
ep.columns = newnames
ep.head()

# check multi-response question to see what we're dealing with
ep['Media'].head(20)


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Exercise.

Construct a list of countries with countries = weo[['ISO', 'Country']]; that is, without applying the drop_duplicates method.  How large is it?  How many duplicates have we dropped?
What are the country codes (ISO) for Argentina, Germany, and Greece?
What are the variable codes (WEO Subject Code) for government debt (gross debt, percent of GDP) and net lending/borrowing (also percent of GDP)?

Comment. Now that we have the country and variable codes, we can be more explicit about what we want.  We want observations with those country and variable codes.
We work up to the solution one step at a time.
Comparisons for series
We can construct comparisons for dataframe columns much as we did with simple variables.  The difference is that we get a complete column or True/False responses, not just one.
Mutiple comparisons have a different syntax than we saw earlier.  and is replaced by &, and or is replaced by |.  And when we have more than comparison, we need to enclose them in parentheses.
Here's an example.
Exercise.  Compute and explain the comparisons:

small['Units'] == 'National currency'
small['2011'] >= 100
(small['Units'] == 'National currency') & (small['2011'] >= 100)
(small['Units'] == 'National currency') | (small['2011'] >= 100)

Boolean selection
Boolean selection simply chooses those observations for which a condition is True.  Some people refer to this as filtering.
Example.  We choose obervations for which the units are 'National currency'.  We do this first in two steps, then in one.
Can you write Python code for it?

ncunits = small['Units'] == 'National currency'
small[ncunits]

small[small['Units'] == 'National currency']


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Exercise. Construct dataframes for which

small['Units'] does not equal 'National currency'.
small['Units'] equals 'National currency' and small['2011'] is greater than 100.

The isin method
Pay attention now, this is really useful.  Suppose we want to extract the data for which weo['ISO'] == 'ARG' (Argentina) or weo['ISO'] == 'GRC' (Greece).  We could do that by combining the comparisons:
python
(weo['ISO'] == 'ARG') | (weo['ISO'] == 'GRC')
Remind youself that | stands for "or."  (What do we use for "and"?)
A simpler approach is to apply the isin method to a variable.  This sets the comparison equal to True if the value of the observation is of weo['ISO'] equals any element in a list.  We could do the same thing using mulitple comparisons, but this is a lot easier.
Let's see how this works.
Example.  Let's apply the same logic to variable codes.  If we want to extract the observations with codes
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
we would use
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Exercise. What would you change in this graph?  How would you do it?  (Words are enough.)
Review
Exercise. We explore the Census's Business Dynamics Statistics, a huge collection of data about firms. We've extracted a small piece of one of their databases that includes these variables for 2013:

Size: size category of firms based on number of employees
Firms: number of firms in each size category
Emp: number of employees in each size category

Run the code cell below to load the data.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Pandas string methods. We can do the same thing to all the observations of a variable with so-called string methods.  We append .str to a variable in a dataframe and then apply the string method of our choice.  If this is part of converting a number-like entry that has mistakenly been given dtype object, we then convert its dtype with the astype method.
Example.  Let's use a string method to fix the item_price variable in the Chipotle dataframe.  This has three parts:

Use the method str to identify this as a string method.
Apply the string method of our choice (here replace) to fix the string.
Use the astype method to convert the fixed-up string to a float.

We start by making a copy of the chp dataframe that we can experiment with.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
What to do.  We use the replace method on the whole dataframe.  To mark something as missing, we replace it as None, which Pandas interprets as missing and labels NaN.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-group.ipynb".
The first task is:
Working with missing values
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-merge.ipynb".
The first task is:

MovieLens data
The data comes as a zip file that contains several csv's.  We get the details from the README inside.  (It's written in Markdown, so it's easier to read if we use a browser to format it.  Or we could cut and paste into a Markdown cell in an IPython notebook.)
The file descriptions are:

ratings.csv:  each line is an individual film rating with the rater and movie id's and the rating.  Order:  userId, movieId, rating, timestamp.
tags.csv:  each line is a tag on a specific film.  Order:  userId, movieId, tag, timestamp.
movies.csv:  each line is a movie name, its id, and its genre.  Order:  movieId, title, genres.  Multiple genres are separated by "pipes" |.
links.csv:  each line contains the movie id and corresponding id's at IMBd and TMDb.

The easy way to input this data is to download the zip file onto our computer, unzip it, and read the individual csv files using read.csv().  But anyone can do it the easy way.  We want to automate this, so we can redo it without any manual steps.  This takes some effort, but once we have it down we can apply it to lots of other data sources.

Automate file download
We're looking for an automated way, so that if we do this again, possibly with updated data, the whole process is in our code.  Automated data entry involves these steps:

Get the file.  We use the requests package, which handles internet files and comes pre-installed with Anaconda. This kind of thing was hidden behind the scenes in the Pandas read_csv function, but here we need to do it for ourselves. The package authors add:
Recreational use of other HTTP libraries may result in dangerous side-effects, including: security vulnerabilities, verbose code, reinventing the wheel, constantly reading documentation, depression, headaches, or even death.

Convert to zip.   Requests simply loads whatever's at the given url. The io module's io.Bytes reconstructs it as a file, here a zip file.
Unzip the file.  We use the zipfile module, which is...
Can you write Python code for it?

# get "response" from url
url = 'http://files.grouplens.org/datasets/movielens/ml-latest-small.zip'
r = requests.get(url)

# describe response
print('Response status code:', r.status_code)
print('Response type:', type(r))
print('Response .content:', type(r.content))
print('Response headers:\n', r.headers, sep='')

# convert bytes to zip file
mlz = zf.ZipFile(io.BytesIO(r.content))
print('Type of zipfile object:', type(mlz))

# what's in the zip file?
mlz.namelist()

# extract and read csv's
movies  = pd.read_csv(mlz.open(mlz.namelist()[2]))
ratings = pd.read_csv(mlz.open(mlz.namelist()[3]))

# what do we have?
for df in [movies, ratings]:
    print('Type:', type(df))
    print('Dimensions:', df.shape)
    print('Variables:', list(df))
    print('First few rows', df.head(3), '\n')


I am working on the file "Code/IPython/bootcamp_pandas-merge.ipynb".
The first task is:
Clean data
Pick a useable subset and fix extra column so that we can combine them.  The problem here is that until 1990, the highest age category was '80+.  From 1990 on, we have a finer breakdown.
We fix this by reassigning '80+' to '80-84' and not worrying that some of these people are 85 or older.  Note that df.fillna(0.0) replaces missing values with zeros.
Can you write Python code for it?

def cleanpop(df, countries, years):
    """
    take df as input and select countries and years
    """
    # rename first two columns
    names = list(df)
    df = df.rename(columns={names[0]: 'Country', names[1]: 'Year'})

    # select countries and years
    newdf = df[df['Country'].isin(countries) & df['Year'].isin(years)]

    return newdf

countries = ['Japan']
past      = [1950, 2000]
future    = [2050, 2100]

e = cleanpop(est, countries, past)
p = cleanpop(prj, countries, future)

# make copie sfor later use
ealt = e.copy()
palt = p.copy()

# fix top-coding in estimates
e['80-84'] = e['80-84'].fillna(0.0) + e['80+'].fillna(0.0)
e = e.drop(['80+'], axis=1)

# check dimensions again
print('Dimensions of cleaned estimates: ', e.shape)
print('Dimensions of cleaned projections: ', p.shape)

# check to see if we have the same variables
list(e) == list(p)


I am working on the file "Code/IPython/bootcamp_pandas-merge.ipynb".
The first task is:
Comment. Note that they have different numbers of columns.  Let's see where that comes from.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-merge.ipynb".
The first task is:
Shape data
We want age categories in the index (the default x axis in a plot) and the years in the columns.  The country we don't care about because there's only one.
Can you write Python code for it?

pop = pop.drop('Country', axis=1)
popi = pop.set_index('Year')
popi

popi.columns.name = 'Age'
popt = popi.T
popt.head()

ax = popt.plot(kind='bar', color='blue', alpha=0.5,
               subplots=True,
               sharey=True,
               figsize=(8,12))


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:

Review
We return to the OECD's healthcare data, specifically a subset of their table on the number of doctors per one thousand population.  This loads and cleans the data:
Can you write Python code for it?

url1 = 'http://www.oecd.org/health/health-systems/'
url2 = 'OECD-Health-Statistics-2015-Frequently-Requested-Data.xls'
docs = pd.read_excel(url1+url2,
                     skiprows=3,
                     usecols=[0, 51, 52, 53, 54, 55, 57],
                     sheetname='Physicians',
                     na_values=['..'],
                     skip_footer=21)
# rename country variable
names = list(docs)
docs = docs.rename(columns={names[0]: 'Country'})
# strip footnote numbers from country names
docs['Country'] = docs['Country'].str.rsplit(n=1).str.get(0)
docs = docs.head()
docs


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Comment. By default, reset_index pushes one or more index levels into columns.  If we want to discard that level of the index altogether, we use the parameter drop=True.
Can you write Python code for it?