I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Comment. This needs some work on the dates.
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

The following two lines reset the graphics to the pyplot standard.  If we ignore them, we get the seaborn standard, which most people think looks better.  If you don't have seaborn installed, none of this matters.
Can you write Python code for it?

# reset graphics (or not)
#mpl.rcParams.update(mpl.rcParamsDefault)
#%matplotlib inline


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Example:  Employment trends
Not long ago, the US had more people working (a higher fraction of the adult population) than many other developed countries.  Over the past 15 years things have flipped.  The deep question is why, but here we simply report what we know.
The key variables are the employment rate (fraction of people aged 25-54) and the participation rate (fraction either working or unemployed).  They're similar but the latter irons out some of the cyclical fluctuations.  We get the data from FRED, but they get it from the OECD's Main Economic Indicators, which covers mostly rich countries.
Can you write Python code for it?

# countries = AU, CA, CH, DE, ES, EU, EZ, FR, GB, JP, KR, NL, O1 (OECD), SE, US, ZA
countries = ['CA', 'DE', 'GB', 'JP', 'US']
emcodes = ['LREM25TT' + c + 'Q156S' for c in countries]
lrcodes = ['LRAC25TT' + c + 'Q156S' for c in countries]

start_date = dt.datetime(2000, 1, 1)
em = data.DataReader(emcodes, 'fred', start_date)
lr = data.DataReader(lrcodes, 'fred', start_date)
em.columns = countries
lr.columns = countries
em.head(3)

def emplot(em, title):
    fig, ax = plt.subplots()
    em.plot(ax=ax, lw=2)
    ax.set_title(title, fontsize=14, loc='left')
    ax.set_xlabel('')
    ax.set_ylabel('Percent')
    ax.legend(loc='best', fontsize=8, handlelength=2, labelspacing=0.1)

emplot(em, 'Employment rates, age 25-54')
emplot(lr, 'Participation rates, age 25-54')


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Example:  US economic conditions
We see a lot of variation in GDP growth rates in all countries.  It's not obvious why, but it's  a fact of life, and one that investors and other business analysts track closely.  Basic numbers like GDP only come out quarterly, and with at least a one month lag, so we're often in the position of not knowing how the economy is doing now, much less how it will do in the future.  To get a clearer sense of current conditions, analysts typically look at a wide range of indicators.  Many of these indicators are available more quickly than GDP.  Since they typically move up and down with GDP, we can use their movements to infer what GDP might be doing.
Of the hundreds of economic indicators, some of the most popular (with their FRED codes) are

INDPRO:  industrial production
PAYEMS:  nonfarm employment
AWHMAN:  average weekly hours worked in manufacturing
PERMIT:  permits for new housing
UMCSENT: consumer sentiment

All of them are monthly.  You can find more about this kind of thing in our book, chapter 11.
Can you write Python code for it?

# get data from FRED
import pandas as pd
from pandas_datareader import data, wb
import datetime as dt                  # handles dates

# get data
indicators = ['INDPRO', 'PAYEMS', 'AWHMAN', 'PERMIT', 'UMCSENT']
start_date = dt.datetime(1970, 1, 1)
inds = data.DataReader(indicators, "fred", start_date)
end = inds.index[-1]

# yoy growth rates
g = inds.pct_change(periods=12).dropna()
# standardize
g_std = (g - g.mean()) / g.std()
gs = g_std

# correlations
g_std.corr()

# plot
fig, ax = plt.subplots()
g_std.plot(ax=ax)
ax.set_title('Economic indicators', fontsize=14, loc='left')
ax.set_ylabel('Standard deviations from mean')
ax.set_xlabel('')
ax.hlines(y=0, xmin=start_date, xmax=end, linestyles='dashed')
ax.legend().set_visible(False)

# focus on recent past
recent_date = dt.datetime(2011, 1, 1)
g_std = g_std[g_std.index>=recent_date]

fig, ax = plt.subplots()
g_std.plot(ax=ax)
ax.set_title('Zoom in on recent past', fontsize=14, loc='left')
ax.set_ylabel('Standard deviations from mean')
ax.set_xlabel('')
ax.hlines(y=0, xmin=recent_date, xmax=end, linestyles='dashed')
ax.legend(loc='upper left', fontsize=10, handlelength=2, labelspacing=0.15)


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  Government debt
One of the traditional ways for a country to get into trouble is to issue so much debt that investors worry about getting paid back.  How much is that?  Hard to say, but we'll look at some debt numbers from the IMF's World Economic Outlook or WEO, a popular source of international data on debt, deficits, and other macroeconomic indicators.  We use numbers for the ratio of government debt to GDP, a standard indicator of public indebtedness.
Given recent events, we pay special attention to Argentina and Greece.
Can you write Python code for it?

%%time

url = 'http://www.imf.org/external/pubs/ft/weo/2016/02/weodata/WEOOct2016all.xls'
weo = pd.read_csv(url, sep='\t', thousands=',', na_values=['n/a', '--'])
weo.shape
#weo

list(weo[list(range(12))])

country_guide = weo[['ISO', 'Country']].drop_duplicates().set_index('ISO')

variable_guide = weo[['WEO Subject Code', 'Subject Descriptor', 'Subject Notes']].drop_duplicates().set_index('WEO Subject Code')
variable_guide.head(3)

variables = ['GGXWDG_NGDP']
countries = ['ARG', 'DEU', 'FRA', 'GRC', 'USA']
debt = weo[weo['ISO'].isin(countries) & weo['WEO Subject Code'].isin(variables)]
some = [3] + list(range(9,44))
debt = debt[some].set_index('Country').T.dropna()
debt.head(3)

fig, ax = plt.subplots()
debt.plot(ax=ax, lw=2)
ax.set_title('Ratio of government debt to GDP', fontsize=14, loc='left')
ax.set_ylabel('Government Debt (Percent of GDP)')
ax.legend(loc='best', fontsize=10, handlelength=2, labelspacing=0.15)


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  Options data
WARNING: This code is currently broken in pandas_datareader, so we will skip this for now
The Pandas package is spinning off the remote data access tools, including FRED, World Bank, and so on.  Here we use the new tool, the Pandas DataReader.  This requires installation:  on the command line, enter
conda install pandas-datareader html5lib
and follow instructions.
You can run commands on the command line from within jupyter by putting a ! at the beginning of the line. To run the commands above you can do enter the following in a code cell:
!conda install -y pandas-datareader html5lib
We'll use the Options tool, which reads in prices of stock options from Yahoo Finance.  The options come with a variety of contract specifications:

Type.  Calls are options to buy, puts are options to sell.
Expiration.  The date when the option expires.
Underlying.  The asset you have a right to buy or sell.  In this data, it's a stock.
Strike price.  The price at which you are able to buy (calls) or sell (puts) the underlying.

We give an illustration using options on Amazon stock.
The first version of this module was written by our very own Spencer Lyon.
```Python
from pandas_datareader.data import Options
"""
supply ticker, get option prices
"""
ticker = 'aapl'
aapl = Options(ticker, 'yahoo')
data = aapl.get_all_data()
exp = aapl.expiry_dates     # get expiration dates
exp
```
```Python
get option prices
cols = [0, 1, 2, 7]
opexp = 2
print('Options with expiry', exp[opexp])
calls = otk.get_call_data(expiry=exp[opexp])[cols]
puts  = otk.get_put_data(expiry=exp[opexp])[cols]
calls.head()
```
```Python
drop extra index levels
calls = calls.reset_index(level=[1,2,3], drop=True)
puts  = puts.reset_index(level=[1,2,3], drop=True)
cut off extremes
spot = otk.underlying_price
print('Spot price', spot)
delta = 0.25
calls  = calls[(calls.index >=...
Can you write Python code for it?

url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2017_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLSX'
url = url1 + url2 + url3

cols = [2, 4, 5] + list(range(6,28))
#est = pd.read_excel(url, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])

"""
for later:  change cols for the two sources, rename 80+ to 80-84, then concat
#pop = pd.concat([est, prj], axis=0, join='outer')
"""
pop = prj

pop.head()

# rename some variables
pop = pop.rename(columns={'Reference date (as of 1 July)': 'Year',
                          'Region, subregion, country or area *': 'Country',
                          'Country code': 'Code'})
# select Japan and years
countries = ['Japan']
years     = [2015, 2025, 2035, 2045, 2055, 2065]
pop = pop[pop['Country'].isin(countries) & pop['Year'].isin(years)]
pop = pop.drop(['Country', 'Code'], axis=1)
pop.head()

pop = pop.set_index('Year').T
pop.head()

fig, ax = plt.subplots(6, 1)
pop.plot(ax=ax[0])
pop.plot(ax=ax[1])

ax = pop.plot(kind='bar',
         alpha=0.5, subplots=True, sharey=True, figsize=(6, 8))
for axnum in range(len(ax)):
    ax[axnum].set_title('')
    ax[axnum].legend(loc='upper left', prop={'size':10})

ax[0].set_title('Japanese population by age', fontsize=14, loc='left')


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Questions

What would you change in this picture?
What did you learn?
What else would you like to know?
Where does the data come from?

Example:  US government bond yields
People often refer to interest rates moving up and down, but in fact the yields (as we call them) move up and down differently at different maturities.  From 2010 to 2015, for examples, yields on bonds with maturities under 2 years were essentially fixed at zero, but yields on higher maturities moved up and down quite a bit.
One challenge with bond yields is getting good data.  Bonds are typically traded over the counter, and those bonds differ in maturity, coupon, and often other features.  Analysts often focus on the interest rates of pure discount "zero-coupon" bonds, which they infer from prices of coupon bonds -- which are, after all, what is generally traded.  This isn't something to go into unless you have a strong interest.  Suffice it to say that we have taken these yields from estimates supplied by the Fed.
This is still a work in progress, but our goal is to produce a movie of bond yields over time.
Can you write Python code for it?

url = 'http://pages.stern.nyu.edu/~dbackus/Data/GSW_nominal_yields.csv'
y = pd.read_csv(url, index_col=0, parse_dates=True)
y.tail()

# compute mean yields
ybar = y.mean(axis=0)
ystd = y.std(axis=0)
maturities = list(range(1,122))
ybar.index = maturities
ystd.index = maturities

fig, ax = plt.subplots()
ybar.plot(ax=ax)
ystd.plot(ax=ax)
ax.set_xlabel('Maturity in Months')
ax.set_ylabel('Mean and Std Dev of Yield (Annual Percent)')
ax.set_ylim(0)
ax.legend(["Mean", "std. dev."])


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Starbucks Revenue
This data was given to us by one of our students.
Can you write Python code for it?

url1 = "https://github.com/NYUDataBootcamp/Materials/blob/master/"
url2 = "Data/Starbucks_Revenue_Worldwide.xlsx?raw=true"
url = url1 + url2
starbucks = pd.read_excel(url,
                          sheetname=1,
                          skiprows=4,
                          parse_cols=[1,2])
starbucks

starbucks.rename(columns={"Unnamed: 0": "year", "data": "revenue"}, inplace=True)
starbucks

starbucks.set_index("year", inplace=True)
starbucks

mpl.rcParams.update(mpl.rcParamsDefault)
starbucks.plot()

starbucks_growth = starbucks.pct_change()
starbucks_growth

starbucks_growth.plot()


I am working on the file "Code/notebooks/bootcamp_format_plotting.ipynb".
The first task is:
The .format() method
what is format and how does it work? Google and find a good link
Can you write Python code for it?

print('Today is {}.'.format(datetime.date.today()))

for index in snap.index:
     print('On {} Snap closed at ${} and the volume was {} million.'.format(index, snap['Close'][index], snap['Volume'][index]))

for index in snap.index:
     print('On {:.10} Snap closed at ${} and the volume was {:.1f} million.'.format(str(index), snap['Close'][index], snap['Volume'][index]))


I am working on the file "Code/notebooks/bootcamp_format_plotting.ipynb".
The first task is:
This looks aweful. We want to cut the day and express the volume in millions.
Can you write Python code for it?

# express Volume in millions
snap['Volume'] = snap['Volume']/10**6

snap


I am working on the file "Code/notebooks/bootcamp_format_plotting.ipynb".
The first task is:
We will want to run the notebook in the future with updated values. How can we do this? Make the dates automatically updated.
Can you write Python code for it?

start = datetime.datetime(2017, 3, 2) # the day Snap went public
end = datetime.date.today() # datetime.date.today

snap = web.DataReader("SNAP", 'google', start, end)

snap

snap.index.tolist()


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
And just because it's fun, here's an example of Tufte-like axes from Matplotlib examples:
Can you write Python code for it?

fig, ax = gdp_bar()

# Tufte-like axes
ax.spines['left'].set_position(('outward', 7))
ax.spines['bottom'].set_position(('outward', 7))
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Approach 1:  Apply plot methods to dataframes
The simplest way to produce graphics from a dataframe is to apply a plot method to it.
We see that a number of things are preset for us:

Data.  By default the data consists of the whole dataframe.
Chart type.  We have options for lines, bars, or other things, but the default is line
x and y variables.  By default, the x variable is the dataframe's index and the y variables are the columns of the dataframe -- all of them that can be plotted (e.g. columns with a numeric dtype).

We can change all of these things, but that's always a good starting point.
US GDP and consumption
Can you write Python code for it?

# try this with US GDP
us.plot()

# do GDP alone
us['gdp'].plot()

us.plot(y="gdp")

# bar chart
us.plot(kind='bar')

# scatter plot
# we need to be explicit about the x and y variables: x = 'gdp', y = 'pce'
us.plot.scatter('gdp', 'pce')

us.plot('gdp', 'pce', kind='scatter')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Approach 2:  plt.plot(x, y)
Next up:  the popular plot(x,y) function from the pyplot module of Matplotlib.  We never use this and will go over it at high speed -- or perhaps not at all.
This is a more explicit version of Matplotlib graphics in which we specify the x and y variables directly.
Can you write Python code for it?

plt.plot(us.index, us['gdp'])

# we can do two lines together
plt.plot(us.index, us['gdp'])
plt.plot(us.index, us['pce'])

# we can also add things to plots
plt.plot(us.index, us['gdp'])
plt.plot(us.index, us['pce'])

plt.title('US GDP', fontsize=14, loc='left')          # add title
plt.ylabel('Billions of 2009 USD')                    # y axis label
plt.xlabel('Year')                                    # y axis label
plt.tick_params(labelcolor='red')                     # change tick labels to red
#plt.legend(['GDP', 'Consumption'], loc='best')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Comment.  In the previous cell, we used the print() function to produce output.  Here we just put the name of the dataframe.  The latter displays the dataframe -- and formats it nicely -- if it's the last statement in the cell.
Fama-French.

Gene Fama and Ken French post lots of data on equity returns on Ken French’s website.
The data are zipped text files, which we can easily read into Excel OR
use the Pandas tool which is better.
Can you write Python code for it?

ff = data.DataReader('F-F_Research_Data_factors', 'famafrench')
ff


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Comment.  When you run the code cell above, its output appears below it.
Exercise. Enter pd.read_csv? in the empty cell below.  Run the cell (Cell at the top, or shift-enter).  Do you see the documentation?  This is the Jupyter version of help in Spyder's IPython console.
Create dataframes to play with

US GDP and consumption
World Bank GDP per capita for several countries
Fama-French equity returns

Data input 3:  APIs
Last time we saw how to read
* internet files using links
* files on your computer.
The pandas method was read_csv() and read_excel(). A third way is through APIs.
APIs are "application program interfaces".  A dataset with an API allows access through some method other than a spreadsheet.
The API is the set of rules for accessing the data. People have written easy-to-use code to access the APIs.
The Pandas developers have created what they call a set of Remote Data Access tools and have put them into a package called pandas_datareader.
FRED.
The St Louis Fed has put together a large collection of time series data that they refer to as FRED:  Federal Reserve Economic Data.  They started with the US, but now include data for many countries.
The Pandas docs describe how to access FRED.  Here's an example that reads in quarterly data for US real GDP and real consumption.

The variable codes  -- not to be confused with "code" -- consists of FRED variable codes.  Go to FRED, use the search box to find the series you want, and look for the variable code at the end of the url in your browser.
The variables start and end contain dates in (year, month, day) format using datetime.datetime.
Can you write Python code for it?

from pandas_datareader import data
import datetime as dt                           # package to handle dates

codes = ['GDPCA', 'PCECCA']                     # real GDP, real consumption (from FRED website)

start = dt.datetime(2003, 1, 1)                 # start date
end = dt.datetime(2013, 1, 1)                   # end date

us = data.DataReader(codes, 'fred', start, end)

us.columns = ['gdp', 'pce']                     #us.set_index (we don't need it now)

print(us.head(3))


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Comment. All of these statements must be in the same cell for this to work.
Comment. This is overkill -- it looks horrible -- but it makes the point that we control everything in the plot.  We recommend you do very little of this until you're more comfortable with the basics.

Approach 3:  Create figure objects and apply methods
This approach is probably the most mysterious, but it's the best.
The idea is to use the matplotlib.pyplot function subplots(), which creates two objects:
* fig : figure object -- blank canvas for creating a figure
* ax  : axis object -- everything in the figure: axes, labels, legend
apply methods on these objects to set the various elements of the graph.
Create objects.   We'll see this line over and over:
Can you write Python code for it?

fig, ax = plt.subplots(2, 1)             # create fig and ax objects -- nrows, ncols


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Comment. Both of these statements must be in the same cell.
Can you write Python code for it?

# Fama-French example
fig, ax = plt.subplots()
ff.plot(ax=ax,
        kind='line',                 # line plot
        color=['blue', 'magenta'],   # line color
        title='Fama-French market and riskfree returns')

# Fama-French example
fig, ax = plt.subplots(1, 2, figsize=(12, 4))

ff.plot(ax=ax[0],
        kind='hist',                 # line plot
        color=['blue', 'magenta'],   # line color
        alpha=0.65,
        bins=20,
        title='Fama-French market and riskfree returns')

ff.plot(ax=ax[1],
        kind='kde',                 # line plot
        color=['blue', 'magenta'],   # line color
        title='Fama-French market and riskfree returns',
        alpha=0.65)

fig.tight_layout()


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Comment. What if we wanted to make the US bar red?  This is far too complicated, but we used our Google fu and found a solution.
Can you write Python code for it?

fig, ax = plt.subplots()
pisa['Math'].plot(ax=ax, kind='barh', figsize=(4,13))
ax.set_title('PISA Math Score', loc='left')

us_index = pisa.index.tolist().index('United States')
ax.get_children()[us_index].set_color('r')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Examples
We conclude with examples that take data from the previous chapter and make better graphs than we did there.
Student test scores (PISA)
The international test scores often used to compare quality of education across countries.
Can you write Python code for it?

url = 'http://dx.doi.org/10.1787/888932937035'
pisa = pd.read_excel(url,
                     skiprows=18,                    # skip the first 18 rows
                     skipfooter=7,                   # skip the last 7
                     parse_cols=[0,1,9,13],          # select columns
                     index_col=0,                    # set index = first column
                     header=[0,1]                    # set variable names
                     )
pisa = pisa.dropna()                                 # drop blank lines
pisa.columns = ['Math', 'Reading', 'Science']        # simplify variable names

# bar chart of math scores
fig, ax = plt.subplots()
pisa['Math'].plot(kind='barh', ax=ax)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise (challenging).  Make the ticks point out.
Can you write Python code for it?

# scatterplot of life expectancy vs gdp per capita
fig, ax = plt.subplots()

ax.scatter(df['gdppc'], df['life'],     # x,y variables
           s=df['pop']/10**6,          # size of bubbles
           alpha=0.5)
ax.set_title('Life expectancy vs. GDP per capita', loc='left', fontsize=14)
ax.set_xlabel('GDP Per Capita')
ax.set_ylabel('Life Expectancy')
ax.text(58, 66, 'Bubble size represents population', horizontalalignment='right')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise.  Add each of these arguments, one at a time, to us.plot():

kind='area'
subplots=True
sharey=True
figsize=(3,6)
ylim=(0,16000)

What do they do?
Can you write Python code for it?

us.plot(kind='area')               # fill the area below

us.plot(subplots=True)                # make separate subplots for the variables in the dataframe

us.plot(subplots=True, sharey = True)              # make the y axis the same

us.plot(figsize = (10, 2))           # first arg: width, second: height (inches)

us.plot(ylim = (0, 16000))                     # change the range of the y axis


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise.  Try adding the arguments title='Fama-French returns', grid=True, and legend=False.
Can you write Python code for it?

ff.plot(kind='kde',
        subplots=True,
        sharex=True,
        title='Fama-French returns',
        grid=True,
        legend=False)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise. Create the same graph for the Reading score.
World Bank data
We'll use World Bank data for GDP, GDP per capita, and life expectancy to produce a few graphs and illsutrate some methods we haven't seen yet.

Bar charts of GDP and GDP per capita
Scatter plot (bubble plot) of life expectancy v GDP per capita
Can you write Python code for it?

# variable list (GDP, GDP per capita, life expectancy)
var = ['NY.GDP.PCAP.PP.KD', 'NY.GDP.MKTP.PP.KD', 'SP.DYN.LE00.IN']
# country list (ISO codes)
iso = ['USA', 'FRA', 'JPN', 'CHN', 'IND', 'BRA', 'MEX']
year = 2013

# get data from World Bank
df = wb.download(indicator=var, country=iso, start=year, end=year)

# massage data
df = df.reset_index(level='year', drop=True)
df.columns = ['gdppc', 'gdp', 'life'] # rename variables
df['pop']  = df['gdp']/df['gdppc']    # population
df['gdp'] = df['gdp']/10**12          # convert to trillions
df['gdppc'] = df['gdppc']/10**3       # convert to thousands
df['order'] = [5, 3, 1, 4, 2, 6, 0]   # reorder countries
df = df.sort_values(by='order', ascending=False)
df

# We'll use this same basic graph a few times.
# Let's make a function so we don't have to repeat the
# code to create
def gdp_bar(variable="gdp"):
    fig, ax = plt.subplots()
    df[variable].plot(ax=ax, kind='barh', alpha=0.5)
    ax.set_title('Real GDP', loc='left', fontsize=14)
    ax.set_xlabel('Trillions of US Dollars')
    ax.set_ylabel('')
    return fig, ax

gdp_bar()

# ditto for GDP per capita (per person)
fig, ax = gdp_bar("gdppc")
ax.set_title('GDP Per Capita', loc='left', fontsize=14)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise. Make the bubble a little larger.
Exercise (challenging). Add labels to the bubbles so we know which country they correspond to.
Can you write Python code for it?

# scatterplot of life expectancy vs gdp per capita
fig, ax = plt.subplots()

ax.scatter(df['gdppc'], df['life'],     # x,y variables
           s=df['pop']/10**6,          # size of bubbles
           alpha=0.5)

ax.set_title('Life expectancy vs. GDP per capita', loc='left', fontsize=14)
ax.set_xlabel('GDP Per Capita')
ax.set_ylabel('Life Expectancy')
ax.text(58, 66, 'Bubble size represents population', horizontalalignment='right')

for (x, y, country) in zip(df['gdppc'], df['life'], df.index):
    ax.text(x, y, country)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise. What do each of the arguments do in the code below?
Can you write Python code for it?

ff.plot(kind='hist',
        bins=20,
        subplots=True)

# "smoothed" histogram
ff.plot(kind='kde', subplots=True, sharex=True)    # smoothed histogram ("kernel density estimate")


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Exercise. What do we have here?  What type are fig and ax?
Can you write Python code for it?

print('fig is ', type(fig))
print('ax  is ', type(ax))

# let's try that again, this time with content
fig, axe = plt.subplots(figsize=(8, 4))

# add things to ax
us.plot(ax=axe, color = ['red', 'green'])


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Fama-French asset returns
Can you write Python code for it?

# now try a few things with the Fama-French data
ff.plot()


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Learn about the structure
Can you write Python code for it?

ff.keys()

ff['DESCR']

ff = ff[1]
ff

ff.columns = ['xsm', 'smb', 'hml', 'rf']

ff['rm'] = ff['xsm'] + ff['rf']
ff = ff[['rm', 'rf']]                    # extract rm (market) and rf (riskfree)

ff.head(5)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Quick review of the bidding
We looked at three ways to use Matplotlib:

Approach #1:  apply plot method to dataframe
Approach #2:  use plot(x,y) function
Approach #3:  create fig, ax objects, apply plot methods to them

Same result, different syntax.  This is what each of them looks like applied to US GDP:
```python
us['gdp'].plot()                   # Approach #1
plt.plot(us.index, us['gdp'])      # Approach #2
fig, ax = plt.subplots()           # Approach #3
ax.plot(us.index, us['gdp'])
Or
fig, ax = plt.subplots()           # Approach #3
us['gdp'].plot(ax=ax)
```
Each one produces the same graph.
Which one should we use?  Use Approach #3.  Really.  This is a case where choice is confusing.
We also suggest you not commit any of this to memory.  If you end up using it a lot, you'll remember it.  If you don't, it's not worth remembering.
Bells and whistles
Adding things to graphs
Axis methods offer us great flexibility. Here's an example.
Can you write Python code for it?

fig, ax = plt.subplots()

us.plot(ax=ax)

# Apply axis methods
ax.set_title('US GDP and Consumption', fontsize=14, loc='left')
ax.set_ylabel('Billions of 2013 USD')
ax.legend(['Real GDP', 'Consumption'], loc=0)  # more descriptive variable names
ax.tick_params(labelcolor='red')               # change tick labels to red
ax.set_ylim(0)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
Review
Consider the data from Randal Olson's blog post:
Can you write Python code for it?

import pandas as pd
data = {'Food': ['French Fries', 'Potato Chips', 'Bacon', 'Pizza', 'Chili Dog'],
        'Calories per 100g':  [607, 542, 533, 296, 260]}
cals = pd.DataFrame(data)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
What is this object?
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_graphics_s17_MBA.ipynb".
The first task is:
World Bank.
The World Bank's databank covers economic and social statistics for most countries in the world.
Variables include GDP, population, education, and infrastructure.  Here's an example:
Can you write Python code for it?

from pandas_datareader import wb

iso_codes = ['BRA', 'CHN', 'FRA', 'IND', 'JPN', 'MEX', 'USA']
var = ['NY.GDP.PCAP.PP.KD']
year = 2013

wbdf = wb.download(indicator=var, country=iso_codes, start=year, end=year)
print(wbdf)

# Change the index for iso codes
wbdf.index = iso_codes

# Add country variable
country = ['Brazil', 'China', 'France', 'India', 'Japan', 'Mexico', 'United States']
wbdf['country'] = country

# Rename the variables
wbdf.columns = ['gdppc', 'country']

# set the display precision in terms of decimal places
pd.set_option('precision', 2)
wbdf['gdppc'] = wbdf['gdppc']/1000

print(wbdf)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
And just because it's fun, here's an example of Tufte-like axes from Matplotlib examples:
Can you write Python code for it?

fig, ax = gdp_bar()

# Tufte-like axes
ax.spines['left'].set_position(('outward', 7))
ax.spines['bottom'].set_position(('outward', 7))
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')
#ax.tick_params(direction='out')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Approach 1:  Apply plot methods to dataframes
The simplest way to produce graphics from a dataframe is to apply a plot method to it.
We see that a number of things are preset for us:

Data.  By default the data consists of the whole dataframe.
Chart type.  We have options for lines, bars, or other things, but the default is line
x and y variables.  By default, the x variable is the dataframe's index and the y variables are the columns of the dataframe -- all of them that can be plotted (e.g. columns with a numeric dtype).

We can change all of these things, but that's always a good starting point.
US GDP and consumption
Can you write Python code for it?

# try this with US real GDP, real consumption
us.plot()

# do GDP alone
us['gdp'].plot()

us.plot(y="gdp")

# bar chart
us.plot(kind='bar')

# scatter plot
# we need to be explicit about the x and y variables: x = 'gdp', y = 'pce'
us.plot.scatter('gdp', 'pce')

us.plot('gdp', 'pce', kind='scatter')


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Approach 2:  plt.plot(x, y)
Next up:  the popular plot(x,y) function from the pyplot module of Matplotlib.  We never use this and will go over it at high speed -- or perhaps not at all.
This is a more explicit version of Matplotlib graphics in which we specify the x and y variables directly.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Comment.  In the previous cell, we used the print() function to produce output.  Here we just put the name of the dataframe.  The latter displays the dataframe -- and formats it nicely -- if it's the last statement in the cell.
Fama-French.

Gene Fama and Ken French post lots of data on equity returns on Ken French’s website.
The data are zipped text files, which we can easily read into Excel OR
use the Pandas tool which is better.
Can you write Python code for it?

ff = data.DataReader('F-F_Research_Data_factors', 'famafrench')
ff


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Comment.  When you run the code cell above, its output appears below it.
Exercise. Enter pd.read_csv? in the empty cell below.  Run the cell (Cell at the top, or shift-enter).  Do you see the documentation?  This is the Jupyter version of help in Spyder's IPython console.
Create dataframes to play with

US GDP and consumption
World Bank GDP per capita for several countries
Fama-French equity returns

Data input
Last time we saw how to read

internet files using links
files on your computer
data sets through APIs

For the first two we used pandas methods, read_csv() and read_excel().
APIs are "application program interfaces".
The API is the set of rules for accessing the data. People have written easy-to-use code to access the APIs.
The Pandas developers have created what they call a set of Remote Data Access tools and have put them into a package called pandas_datareader.
FRED.
The St Louis Fed has put together a large collection of time series data that they refer to as FRED:  Federal Reserve Economic Data.  They started with the US, but now include data for many countries.
The Pandas docs describe how to access FRED.  Here's an example that reads in quarterly data for US real GDP and real consumption.

The variables start and end contain dates in (year, month, day) format using datetime.datetime.
The variable codes  -- not to be confused with "code" -- consists of FRED variable codes.  Go to FRED, use the search box to find the series you want, and look for the variable code at the end of the url in your browser.
Can you write Python code for it?

from pandas_datareader import data
import datetime                                 # package to handle dates

start = datetime.datetime(2003, 1, 1)           # start date -- year/month/day
end = datetime.datetime(2013, 1, 1)             # end date

codes = ['GDPCA', 'PCECCA']                     # real GDP, real consumption (from FRED website)

us  = data.DataReader(codes, 'fred', start, end)

us.columns = ['gdp', 'pce']                     #us.set_index = list(range(2003, 2014)) (we don't need it now)

print(us.head(3))


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Examples
We conclude with examples that take data from the previous chapter and make better graphs than we did there.
World Bank data
We'll use World Bank data for GDP, GDP per capita, and life expectancy to produce a few graphs and illsutrate some methods we haven't seen yet.

Bar charts of GDP and GDP per capita
Scatter plot (bubble plot) of life expectancy v GDP per capita
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Exercise.  Add each of these arguments, one at a time, to us.plot():

kind='area'
subplots=True
sharey=True
figsize=(3,6)
ylim=(0,16000)

What do they do?
Can you write Python code for it?

us.plot?

us.plot(kind='area')               # fill the area below

us.plot(subplots=True)                # make separate subplots for the variables in the dataframe

us.plot(subplots=True, sharey = True)              # make the y axis the same (transparency!)

us.plot(figsize = (10, 2))           # first arg: width, second: height (inches)

us.plot(ylim = (0, 16000))                     # change the range of the y axis


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Exercise. Make the bubble a little larger.
Exercise (challenging). Add labels to the bubbles so we know which country they correspond to.
Can you write Python code for it?

# scatterplot of life expectancy vs gdp per capita
fig, ax = plt.subplots()

ax.scatter(df['gdppc'], df['life'],     # x,y variables
           s=df['pop']/10**6,          # size of bubbles
           alpha=0.4)

ax.set_title('Life expectancy vs. GDP per capita', loc='left', fontsize=14)
ax.set_xlabel('GDP Per Capita')
ax.set_ylabel('Life Expectancy')
ax.text(58, 66, 'Bubble size represents population', horizontalalignment='right')

for (x, y, country) in zip(df['gdppc'], df['life'], df.index):
    ax.text(x, y, country)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Multiple subplots
Same idea, but we create a multidimensional ax and apply methods to each component. Here we redo the plots of US GDP and consumption.
Can you write Python code for it?

fig, ax = plt.subplots(nrows=2, ncols=2, sharex=True)
print('Object ax has dimension', ax.shape)

fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(10, 6), sharex=True, sharey=True)

ax[0, 0].plot(us.index, us['pce'])
ax[0, 1].plot(us.index, us['gdp'])
ax[1, 0].plot(us.index, us['gdp'])
ax[1, 1].plot(us.index, us['pce'])
#plt.setp(ax[1, 0].xaxis.get_majorticklabels(), rotation=40)
#plt.setp(ax[1, 1].xaxis.get_majorticklabels(), rotation=40)
plt.show()

# now add some content
fig, ax = plt.subplots(nrows=2, ncols=1, sharex=True, sharey=True)

us['gdp'].plot(ax=ax[0], color='green')   # first plot
us['pce'].plot(ax=ax[1], color='red')     # second plot


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
Review
Consider the data from Randal Olson's blog post:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
World Bank.
The World Bank's databank covers economic and social statistics for most countries in the world.  Variables include GDP, population, education, and infrastructure.  Here's an example:
Can you write Python code for it?

from pandas_datareader import wb

iso_codes = ['BRA', 'CHN', 'FRA', 'IND', 'JPN', 'MEX', 'USA']
var = ['NY.GDP.PCAP.PP.KD']
year = 2013

wbdf = wb.download(indicator=var, country=iso_codes, start=year, end=year)
print(wbdf)

# Change the index for iso codes
wbdf.index = iso_codes

print(wbdf)

# Add country variable
country = ['Brazil', 'China', 'France', 'India', 'Japan', 'Mexico', 'United States']
wbdf['country'] = country

# Rename the variables
wbdf.columns = ['gdppc', 'country']

# set the display precision in terms of decimal places
pd.set_option('precision', 2)
wbdf['gdppc'] = wbdf['gdppc']/1000

print(wbdf)


I am working on the file "Code/notebooks/bootcamp_graphics_s17_UG.ipynb".
The first task is:
try plt. + tab completion -- a lot of options
Can you write Python code for it?

# we can do two lines together
plt.plot(us.index, us['gdp'])
plt.plot(us.index, us['pce'])

# we can also add things to plots
plt.plot(us.index, us['gdp'])
plt.plot(us.index, us['pce'])

plt.title('US GDP', fontsize=14, loc='left')          # add title
plt.ylabel('Billions of 2009 USD')                    # y axis label
plt.xlabel('Year')                                    # y axis label
plt.tick_params(labelcolor='red')                     # change tick labels to red
#plt.legend(['GDP', 'Consumption'], loc='best')


I am working on the file "Code/notebooks/bootcamp_indicators.ipynb".
The first task is:
Radar plot ??
Country indicators:  Opportunities in Southern Africa
Zambeef is a successful meat distributor located in Zambia.  Their leadership wonders whether their operation can be expanded to include neighboring Botswana and Tanzania.  We collect a number of economic and institutional indicators to assess the business climates in the three countries and ask:

What features of an economy are import to this business?
What indicators of these features can we find in the World Bank's data?
How do the three countries compare on these features?
What locations look the most attractive to you?

We start by looking through the World Bank's enormous collection of country indicators and using Pandas' World Bank API to access the numbers.
Can you write Python code for it?

from pandas.io import wb               # World Bank api

# read data from World Bank
iso  = ['ZMB', 'BWA', 'TZA']           # country list (ISO codes)
var  = ['NY.GDP.PCAP.PP.KD',           # GDP per person
        'SP.POP.TOTL',                 # population
        'IC.BUS.EASE.XQ',              # ease of doing business (rank of 189)
        'IS.ROD.PAVE.ZS',              # paved roads (percent of total)
        'SE.ADT.LITR.ZS']              # adult literacy (15 and up)
year = 2014
df = wb.download(indicator=var, country=iso, start=2005, end=2014)
df


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:

Boolean selection
This is mostly what we do:  we choose observations that satisfy one or more conditions.  Boolean selection consists of two steps that we typically combine in one statement:

Use a comparison to construct a Boolean variable consisting of True and False.
Compute df[comparison], where df is a dataframe and comparison is a comparison.  This will select the observations (rows) for which comparison is true and throw away the others.

We work through this one step at a time:

Example:  apply the want operator
Comparisons for dataframes
Boolean selection:  select observations for which the comparison is True
The isin method

This is easier to describe with an example.
Example:  Apply the want operator to WEO
Our want here is to take the weo dataframe and extract government debt and deficits for a given set of countries.  Putting this to work involves several steps.
Here's the head of the dataframe to remind us what we're dealing with.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:

The want operator
We need to know what we're trying to do -- what we want the data to look like. To borrow a phrase from our friend Tom Sargent, we say that we apply the want operator.
Some problems we've run across that ask to be solved:

Numerical data is contaminated by commas (marking thousands) or dollar signs.
Row and column labels are contaminated.
Missing values are marked erratically.
We have too much data, would prefer to choose a subset.
Variables run across rows rather than down columns.

What we want in each case is the opposite of what we have:  we want nicely formatted numbers, clean row and column labels, and so on.
We'll solve the first four problems here, the last one in the next notebook.
Example:  Chipotle data
This data comes from a New York Times story about the number of calories in a typical order at Chipotle.  The topic doesn't particularly excite us, but the data raises a number of issues that come up repeatedly. We adapt some code written by Daniel Forsyth.
Can you write Python code for it?

url = 'https://raw.githubusercontent.com/TheUpshot/chipotle/master/orders.tsv'
chp = pd.read_csv(url, sep='\t')   # tab (\t) delimited
print('Variable dtypes:\n', chp.dtypes, sep='')
chp.head()


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Boolean selection
Boolean selection simply chooses those observations for which a condition is True.  Some people refer to this as filtering.  The syntax is
python
df[comparison]
The result is a new dataframe of observations in which comparison is true.
Example.  We choose obervations for which the units are 'National currency'.  We do this first in two steps, then in one.
Can you write Python code for it?

# remind ourslves what we're starting with
small

# two steps:  comparison, then selection
ncunits = small['Units'] == 'National currency'   # comparison
print(ncunits)
small[ncunits]                                    # selection

# put the steps together in one line
small[small['Units'] == 'National currency']


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment.  Note the commas separating answers with more than one choice.  We want to unpack them somehow.
Example:  OECD healthcare statistics
The OECD collects healthcare data on lots of (mostly rich) countries, which is helpful in producing comparisons.  Here we use a spreadsheet linked in one of their documents.
Can you write Python code for it?

url1 = 'http://www.oecd.org/health/health-systems/'
url2 = 'OECD-Health-Statistics-2015-Frequently-Requested-Data.xls'
docs = pd.read_excel(url1+url2,
                     skiprows=3,
                     usecols=[0, 51, 52, 53, 54, 55, 57],
                     sheetname='Physicians',
#                     na_values=['..'],
                     skip_footer=21)
print('Dimensions:', docs.shape)
print('\nIndex', docs.index.tolist(), sep='')
print('\nVariable dtypes:\n', docs.dtypes.tail(8), sep='')
docs.head()


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Just for fun, here's a bar graph of the result.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Not quite, we only want to split once.
Can you write Python code for it?

# what about this?
country.rsplit(maxsplit=1)

# one more step, we want the first component of the list
country.rsplit(maxsplit=1)[0]

docs["Country"].head()

# now do this for the variable Country
#docs['Country'].str.rsplit(maxsplit=1).str[0].head()   # explain why this doesn't work
docs['Country'].str.rsplit(n=1).str[0].head()

# Spencer prefers the get method to slicing
docs['Country'].str.rsplit(n=1).str.get(0).head()

# now assign it to newdocs and see what we have
newdocs = docs.copy()
newdocs['Country'] = newdocs['Country'].str.rsplit(n=1).str.get(0)
newdocs.head()

# assign it back to docs for future use
docs = newdocs


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Note that the variable item_price has dtype object. The reason is evidently the dollar sign.  We want to have it as a number, specifically a float.
Example:  Data Bootcamp entry poll
This is the poll we did at the start of the course.  Responses were collected in a Google spreadsheet, which we converted to a csv and uploaded to our website.
Can you write Python code for it?

import pandas as pd
# url1 = 'http://pages.stern.nyu.edu/~dbackus/Data/'
# url2 = 'Data-Bootcamp-entry-poll_s16.csv'
url1 = "https://raw.githubusercontent.com/NYUDataBootcamp/"
url2 = "Materials/master/Data/entry_poll_fall16.csv"
url = url1 + url2
ep = pd.read_csv(url, header=0)
print('Dimensions:', ep.shape)
print('\nData types:\n', ep.dtypes, sep='')
ep.tail(2)


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. Now let's do the same for some of the other entries and save them in new variables.
Can you write Python code for it?

media = ['None', 'Twitter', 'FB', 'Blog']
oldep = ep.copy()

vnames = []
for x in media:
    newname = 'Media' + '_' + x
    vnames.append(newname)
    ep[newname] = ep['Media'].str.contains(x)*1

vnames


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. This has several issues. Here's what we want:

We would like the variables to be in columns and have observations labeled 1980, 1981, etc. In other words, we want the years in the index.
We would like to make sure the data columns (1980, 1981, etc.) have dtype float64

Here's what we might to do achieve what we want:

We could try to transpose the dataframe to get the years in the index. This is close, but not quite what we want (see below). We'll come back to this in the next notebook.
To make the data columns have dtype float64 we need to warn pandas about the , thousand separators and n/a for missing data.

Question.  Can we transpose the whole thing to get the data running down columns?
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We can do the same thing with countries.  If we want to choose two variables and three countries, the code looks like:
Can you write Python code for it?

variables = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
countries = ['ARG', 'DEU', 'GRC']
weo_sub = weo[weo['WEO Subject Code'].isin(variables) & weo['ISO'].isin(countries)]
weo_sub


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We did everything here in one line:  replace the dollar sign with a string method, then converted to float using astype.  If you think this is too dense, you might break it into two steps.
Example. Here we strip off the numbers at the end of the indexes in the OECD docs dataframe.  This involves some experimentation:

Play with the rsplit method to see how it works.
Apply rsplit to the example country = 'United States 1'.
Use a string method to do this to all the entries of the variable Country.
Can you write Python code for it?

# try this with an example first
country = 'United States 1'

# get documentation for the rsplit method
#country.rsplit?

# an example
country.rsplit()


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We usually don't have to worry about this, Pandas takes care of missing values automatically.
Comment. Let's try a picture to give us a feeling of accomplishment. What else would you say we need?  How would we get it?
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comment. We're choosing 2 variables from 45, so there are lots of Falses.
Can you write Python code for it?

# this time let's use the result of isin for selection
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
weo[weo['WEO Subject Code'].isin(vlist)].head(6)

# we've combined several things in one line
comparison = weo['WEO Subject Code'].isin(vlist)
selection  = weo[comparison]
selection.head(6)


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comments.

We've now done what we described when we applied the want operator.
This is a go-to method.  Circle it for later reference.
This is a go-to method.  Circle it for later reference.

Exercise. Use the isin method to extract Gross domestic product in US dollars for China, India, and the United States.  Assign the result to the dataframe gdp.  Hint:  You can adapt the code we just ran. The variable code is NGDPD.  The country codes are CHN, IND, and USA.
Exercise (challenging). Plot the variable gdp['2015'] as a bar chart.  What would you say it needs?

The contains method
Another useful one.  The contains string method for series identifies observations that contain a specific string.  If yes, the observation is labelled True, if no, False.  A little trick converts the True/False outcomes to ones and zeros.
We apply it to the Media variable of the Entry Poll dataframe ep.  You may recall that this variable could have more than one response.  We tease them apart with the contains method.  Our want is to have a yes/no variable for each response.
Can you write Python code for it?

# recall
ep['Media'].head(10)

# the contains method
ep['Media'].str.contains('Twitter').head(10)


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comments.  Here we have a couple issues.

The first column includes a space and a number:  Australia 1, Chile 3, etc.  We care about this because when we plot the data across countries, the country labels are going to be country names, so we want them in a better form than this.
The ..'s in the sheet lead us to label any column that includes them as dtype object.  Here we want to label them as missing values.
If we want to plot each country against time, then we'll need to switch the rows and columns somehow, so that the x axis in the plot (the year) is the index and not the column label.

One more thing before we proceeed:  we want to change the name of the country variable.
Can you write Python code for it?

names = list(docs)
docs = docs.rename(columns={names[0]: 'Country'})
docs.head(2)


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Comments. This is mostly text data, which means it's assigned the dtype object.  Which is fine.  But there are two things that would make the data easier to work with:

The column names are excessively verbose.  This one's easy:  We replace them with single words.  Which we do below.
The second one is harder.  Two of the questions -- social media and special topics -- say "mark all that apply."  In the spreadsheet, we have a list of every choice the person checked.  Our want is to count the number of each type of response.  For example, we might want a bar chart that gives us the number of each response.  The question is how we get there.
Can you write Python code for it?

# rename variables
newnames = ['time', "why", 'program', 'programming', "prob_stats", 'major',
            "career", "media", "data", "topics"]
newnames = [name.title() for name in newnames]
ep.columns = newnames
ep.head()

# check multi-response question to see what we're dealing with
ep['Media'].head(20)


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Example. Let's try each of these in a different cell and see what they do:

small[['ISO', 'Units']]
small[[0, 4]]
small['2011']
small[1:3]

Can you explain the results?
Can you write Python code for it?

small[['ISO', 'Units']]

small[[0, 4]]

small['2011']

small['2011'][3]

small[1:3]

small[[False, True, True, False, False]]


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Example:  World Economic Outlook
The IMF's World Economic Outlook database contains a broad range of macroeconomic data for a large number of countries.  It's updated twice a year and is a go-to source for things like current account balances (roughly, the trade balance) and government debt and deficits.  It also has a few quirks, as we'll see.
Example. Run the following code as is, and with the thousands and na_values parameters commented out.  How do the dtypes differ?
Can you write Python code for it?

url1 = 'http://www.imf.org/external/pubs/ft/weo/2015/02/weodata/'
url2 = 'WEOOct2015all.xls'
url = url1 + url2
weo = pd.read_csv(url, sep='\t',
                  usecols=[1,2,3,4,6,40,41,42,43,44],
                  thousands=',',
                  na_values=['n/a', '--']
                 )

print('Variable dtypes:\n', weo.dtypes, sep='')
weo.head()


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Exercise. Construct dataframes for which

small['Units'] does not equal 'National currency'.
small['Units'] equals 'National currency' and small['2011'] is greater than 100.

The isin method
Pay attention now, this is really useful.  Suppose we want to extract the data for which weo['ISO'] == 'ARG' (Argentina) or weo['ISO'] == 'GRC' (Greece).  We could do that by combining the comparisons:
python
(weo['ISO'] == 'ARG') | (weo['ISO'] == 'GRC')
Remind youself that | stands for "or."  (What do we use for "and"?)
A simpler approach is to apply the isin method to a variable.  This sets the comparison equal to True if the value of the observation is of weo['ISO'] equals any element in a list.  We could do the same thing using mulitple comparisons, but this is a lot easier.
Let's see how this works.
Example.  Let's apply the same logic to variable codes.  If we want to extract the observations with codes
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
we would use
Can you write Python code for it?

vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
weo['WEO Subject Code'].isin(vlist).head(45)


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Exercise. What would you change in this graph?  How would you do it?  (Words are enough.)
Review
Let's remind ourselves what we've learned.
Exercise. We explore the Census's Business Dynamics Statistics, a huge collection of data about firms. We've extracted a small piece of one of their databases that includes these variables for 2013:

Size: size category of firms based on number of employees
Firms: number of firms in each size category
Emp: number of employees in each size category

Run the code cell below to load the data.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
NOTE: go back to the previous call to pd.read_csv to get the data columns to have dtype float64 -- we'll need that below

String methods
We can treat variables as strings in Pandas in much the same way we dealt with strings in core Python.  Run the code below to remind yourself how this works.
Can you write Python code for it?

dollars = '$123.45'
print('Type of variable dollars:', type(dollars))
num = dollars.replace('$', '')
num = float(num)
print('Type of variable num:', type(num))


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Pandas string methods. We can do the same thing to all the observations of a variable with so-called string methods.  We append .str to a variable in a dataframe and then apply the string method of our choice.  If this is part of converting a number-like entry that has mistakenly been given dtype object, we then convert its dtype with the astype method.
Example.  Let's use a string method to fix the item_price variable in the Chipotle dataframe.  This has three parts:

Use the method str to identify this as a string method.
Apply the string method of our choice (here replace) to fix the string.
Use the astype method to convert the fixed-up string to a float.

We start by making a copy of the chp dataframe that we can experiment with.
Can you write Python code for it?

chp.head()

chpnum = chp.copy()
print('Original dtype:', chpnum['item_price'].dtype)
# create a copy of the df to play with

# delete dollar signs
chpnum['item_price'].str.replace('$', '').head()

# delete dollar signs, convert to float, and assign back to chpnum
chpnum['item_price'] = chpnum['item_price'].str.replace('$', '').astype(float)
print('New dtype:', chpnum['item_price'].dtype)

# assign back to chp for future use
chp = chpnum


I am working on the file "Code/notebooks/bootcamp_pandas-clean.ipynb".
The first task is:
Series indexing
Indexing a Series is a little different because we only have one column, so all indexing operations interact with rows.
The rules here are a little subtle, so we'll show examples and add comments that explain what each example does
In the list below s is a Series, n is an integer, nlist = [0, 3] is a list of integers, and i is a string, and is is a list of strings

s[n]: if the index has dtype int, this extracts the row with index n. Otherwise extracts the nth row (starting at zero)
s[nlist]: if the index has dtype int, this extracts rows with indices in nlist returning NaN if they don't appear. Otherwise extracts the rows at positions in nlist, filling with NaN for invalid positions
s[i]: if the index has dtype object, this extracts the row with index i, otherwise it is an error
s[is]:
Can you write Python code for it?

s1 = pd.Series([5, 6, 7, 8], index=["a", "b", "c", "d"])
s1

s2 = pd.Series([50, 60, 70, 80], index=[0, 4, 2, 999])
s2

# index has dtype object, so using an int returns the value in that row (starting at 0)
s1[1]

# index has dtype int, so using an integer tries to find the that int in the
# index and return the corresponding value and throws an error if it can't find it
s2[1]

s2[0]  # no error, 0 is in the index

# index has dtype object, so a list of ints extracts those rows
s1[[0, 3]]

# index has dtype int, so a list of ints tries to match each int to the index
# it returns NaN where it can't find the index. Notice it **did not** return
# `80` for 3
s2[[0, 3, 999]]

# index has type object, so a string finds row with matching index
s1["c"]

# index has dtype int, so using a string causes an error
s2["c"]

# similar behavior for lists of strings
s1[["a", "b", "penguin"]]

# index has dtype int, so list of strings returns NaN's everywhere
s2[["a", "b"]]

# lists of True/False work the same for any dtype of index
bools = [True, False, False, True]
s1[bools]

s2[bools]


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:

Merging ratings and movie titles
The movie ratings in the dataframe ratings give us individual opinions about movies, but they don't include the name of the movie.   Why not?  Rather than include the name every time a movie is rated, the MovieLens data associates each rating with a movie code, than stores the names of movies associatd with each movie code in the dataframe movies.  We run across this a lot:  some information is in one data table, other information is in another.
Our want is therefore to add the movie name to the ratings dataframe.  We say we merge the two dataferames.  There are lots of ways to merge.  Here we do one as an illustration.
Let's start by reminding ourselves what we have.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:

Population "estimates" and "projections"
We look (again) at the UN's  population data, specifically the age distribution of the population.  The data comes in two sheets:  estimates that cover the period 1950-2015 and projections that cover 2016-2100.  Our mission is to combine them.
Load data
We start, as usual, by loading the data.  This takes a minute or so.
Can you write Python code for it?

url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2017_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLSX'
url = url1 + url2 + url3

cols = [2, 5] + list(range(6,28))
est = pd.read_excel(url, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])

print('Dimensions and dtypes of estimates: ', est.shape, '\n', est.dtypes.head(), sep='')
print('\nDimensions and dtypes of projections: ', prj.shape, '\n', prj.dtypes.head(), sep='')


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Clean data
Pick a useable subset and fix extra column so that we can combine them.  The problem here is that until 1990, the highest age category was '80+.  From 1990 on, we have a finer breakdown.
We fix this by reassigning '80+' to '80-84' and not worrying that some of these people are 85 or older.  Note that df.fillna(0.0) replaces missing values with zeros.
Can you write Python code for it?

def cleanpop(df, countries, years):
    """
    take df as input and select countries and years
    """
    # rename first two columns
    names = list(df)
    df = df.rename(columns={names[0]: 'Country', names[1]: 'Year'})

    # select countries and years
    newdf = df[df['Country'].isin(countries) & df['Year'].isin(years)]

    return newdf

countries = ['Japan']
past      = [1950, 2000]
future    = [2050, 2100]

e = cleanpop(est, countries, past)
p = cleanpop(prj, countries, future)

# make copie sfor later use
ealt = e.copy()
palt = p.copy()

# fix top-coding in estimates
e['80-84'] = e['80-84'].fillna(0.0) + e['80+'].fillna(0.0)
e = e.drop(['80+'], axis=1)

# check dimensions again
print('Dimensions of cleaned estimates: ', e.shape)
print('Dimensions of cleaned projections: ', p.shape)

# check to see if we have the same variables
list(e) == list(p)


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Comment. Note that they have different numbers of columns.  Let's see where that comes from.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Exercise. Something to do together.  suppose we wanted to save the files on our computer.  How would we do it? Would we prefer individual csv's or a single zip?
Can you write Python code for it?

# experiment via http://stackoverflow.com/a/18043472/804513
with open('test.zip', 'wb') as out_file:
    shutil.copyfileobj(io.BytesIO(r.content), out_file)


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Exercise. What happens if we try to merge the original dataframes, including the one with the extra 80+ column?  Run the code below and comment on what you get.
Can you write Python code for it?

"""
popalt = pd.concat([ealt, palt], axis=0)
popalt
"""


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Merge estimates and projections
If we have two blocks of data, and want just want to put them on top of each other, we use the Pandas' concatenate function.  Ditto two blocks next to each other.
But first we need to fix the difference in the columns of the two dataframes.
Can you write Python code for it?

pop = pd.concat([e, p], axis=0).fillna(0.0)
pop


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Merging
Here's roughly what's involved in what we're doing.  We take the movieId variable from ratings and look it up in movies.  When we find it, we look up the title and add it as a column in ratings.  The variable movieId is common, so we can use it to link the two dataframes.
Can you write Python code for it?

combo = pd.merge(ratings, movies,   # left and right df's
                 how='left',        # add to left
                 on='movieId'       # link with this variable/column
                )

print('Dimensions of ratings:', ratings.shape)
print('Dimensions of movies:', movies.shape)
print('Dimensions of new df:', combo.shape)

combo.head(20)

# save as csv file for future use
combo.to_csv('mlcombined.csv')

print('Current directory:\n', os.getcwd(), sep='')
print('List of files:', os.listdir(), sep='\n')


I am working on the file "Code/notebooks/bootcamp_pandas-merge.ipynb".
The first task is:
Shape data
We want age categories in the index (the default x axis in a plot) and the years in the columns.  The country we don't care about because there's only one.
Can you write Python code for it?

pop = pop.drop('Country', axis=1)
popi = pop.set_index('Year')
popi

popi.columns.name = 'Age'
popt = popi.T
popt.head()

ax = popt.plot(kind='bar', color='blue', alpha=0.5,
               subplots=True,
               sharey=True,
               figsize=(8,12))


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:

Review
We return to the OECD's healthcare data, specifically a subset of their table on the number of doctors per one thousand population.  This loads and cleans the data:
Can you write Python code for it?

url1 = 'http://www.oecd.org/health/health-systems/'
url2 = 'OECD-Health-Statistics-2015-Frequently-Requested-Data.xls'
docs = pd.read_excel(url1+url2,
                     skiprows=3,
                     usecols=[0, 51, 52, 53, 54, 55, 57],
                     sheetname='Physicians',
                     na_values=['..'],
                     skip_footer=21)
# rename country variable
names = list(docs)
docs = docs.rename(columns={names[0]: 'Country'})
# strip footnote numbers from country names
docs['Country'] = docs['Country'].str.rsplit(n=1).str.get(0)
docs = docs.head()
docs


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:

Example:  WEO debt and deficits
We spend most of our time on one of the examples from the previous notebook. The problem in this example is that variables run across rows, rather than down columns.  Our want is to flip some of the rows and columns so that we can plot the data against time.  The question is how.
We use a small subset of the IMF's World Economic Outlook database that contains two variables and three countries.
Can you write Python code for it?

url1 = 'http://www.imf.org/external/pubs/ft/weo/2015/02/weodata/'
url2 = 'WEOOct2015all.xls'
url = url1 + url2
weo = pd.read_csv(url, sep='\t',
                  usecols=[1,2,3,4,6,40,41,42],
                  thousands=',',
                  na_values=['n/a', '--'])
print('Variable dtypes:\n', weo.dtypes, sep='')

# create debt and deficits dataframe:  two variables and three countries
variables = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
countries = ['ARG', 'DEU', 'GRC']
dd = weo[weo['WEO Subject Code'].isin(variables) & weo['ISO'].isin(countries)]

# change column labels to something more intuitive
dd = dd.rename(columns={'WEO Subject Code': 'Variable',
                        'Subject Descriptor': 'Description'})

# rename variables
dd['Variable'] = dd['Variable'].replace(to_replace=['GGXWDG_NGDP', 'GGXCNL_NGDP'],
                                        value=['Debt', 'Surplus'])
dd


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Comment. By default, reset_index pushes one or more index levels into columns.  If we want to discard that level of the index altogether, we use the parameter drop=True.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Comment. Note that all the parameters here are columns.  That's not a choice, it's the way the the pivot method is written.
We do a plot for fun:
Can you write Python code for it?

# plot 2013 as bar chart
fig, ax = plt.subplots()
bdsp[2013].plot.barh(ax=ax)
ax.set_ylabel('')
ax.set_xlabel('Number of Employees (millions)')


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Comment. We see here that the multi-index for the rows has been turned into a multi-index for the columns.  Works the same way.
The only problem here is that the column labels are more complicated than we might want.  Here, for example, is what we get with the plot method.  As usual, .plot() plots all the columns of the dataframe, but here that means we're mixing variables.  And the legend contains all the levels of the column labels.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Comments. The problem here is that the columns include both the numbers (which we want to plot) and some descriptive information (which we don't).

Setting and resetting the index
We start by setting and resetting the index.  That may sound like a step backwards -- haven't we done this already? -- but it reminds us of some things that will be handy later.
Take the dataframe dd.  What would we like in the index?  Evenutally we'd like the dates [2011, 2012, 2013], but right now the row labels are more naturally the variable or country.  Here are some varriants.
Setting the index
Can you write Python code for it?

dd.set_index('Country')

# we can do the same thing with a list, which will be meaningful soon...
dd.set_index(['Country'])


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise (challenging).  Take the unstacked dataframe dds.  Use some combination of stack, unstack, and plot to plot the variable Surplus against Year for all three countries.   Challenging mostly because you need to work out the steps by yourself.

Pivoting
The pivot method:  a short cut to some kinds of unstacking.  In rough terms, it takes a wide dataframe and constructs a long one.  The inputs are columns, not index levels.
Example:  BDS data
The Census's Business Dynamnics Statistics collects annual information about the hiring decisions of firms by size and age.  This table list the number of firms and total employment by employment size categories:  1 to 4 employees, 5 to 9, and so on.
Apply want operator. Our want is to plot total employment (the variable Emp) against size (variable fsize).  Both are columns in the original data.
Here we construct a subset of the data, where we look at two years rather than the whole 1976-2013 period.
Can you write Python code for it?

url = 'http://www2.census.gov/ces/bds/firm/bds_f_sz_release.csv'
raw = pd.read_csv(url)
raw.head()

sizes = ['a) 1 to 4', 'b) 5 to 9', 'c) 10 to 19', 'd) 20 to 49']
bds = raw[(raw['year2']>=2012) & raw['fsize'].isin(sizes)][['year2', 'fsize', 'Firms', 'Emp']]
bds


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise.  For the dataframe ddi do the following in separate code cells:

Use the reset_index method to move the Units level of the index to a column of the dataframe.
Use the drop parameter of reset_index to delete Units from the dataframe.

Switching rows and columns
If we take the dataframe ddi, we see that the everything's been put into the index but the data itself.  Perhaps we can get what we want if we just flip the rows and columns.  Roughly speaking, we refer to this as pivoting.
First look at switching rows and columns
The simplest way to flip rows and columns is to use the T or transpose property.  When we do that, we end up with a lot of stuff in the column labels, as the multi-index for the rows gets rotated into the columns.  Other than that, we're good. We can even do a plot.  The only problem is all the stuff we've pushed into the column labels -- it's kind of a mess.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. Run the code below and explain what each line of code does.
Can you write Python code for it?

# stacked dataframe
ds.head(8)

du1 = ds.unstack()

du2 = du1.unstack()


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. Set Variable as the index.
Comment. Note that the new index brought its name along:  Country in the two examples, Variable in the exercise.  That's incredibly useful because we can refer to index levels by name.  If we happen to have an index without a name, we can set it with
python
df.index.name = 'Whatever name we like'
Multi-indexes
We can put more than one variable in an index, which gives us a multi-index.  This is sometimes called a hierarchical index because the levels of the index (as they're called) are ordered.
Multi-indexes are more common than you might think.  One reason is that data itself is often multi-dimensional.  A typical spreadsheet has two dimensions:  the variable and the observation.  The WEO data is naturally three dimensional:  the variable, the year, and the country.  (Think about that for a minute, it's deeper than it sounds.)
The problem we're having is fitting this nicely into two dimensions.  A multi-index allows us to manage that.  A two-dimensional index would work here -- the country and the variable code -- but right now we have some redundancy.
Example. We push all the descriptive, non-numerical columns into the index, leaving the dataframe itself with only numbers, which seems like a step in thee right direction.
Can you write Python code for it?

ddi = dd.set_index(['Variable', 'Country', 'ISO', 'Description', 'Units'])
ddi


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. With the dataframe ddt:

What type of object is ddt["Debt"]?
Construct a line plot of Debt over time with one line for each country.

Example. Let's do this together.  How would we fix up the legend?  What approaches cross your mind?  (No code, just the general approach.)
Swapping levels
Since variables refer to the first level of the column index, it's not clear how we would group data by country.  Suppose, for example, we wanted to plot Debt and Surplus for a specific country.  What would we do?
One way to do that is to make the country the top level with the swaplevel method.  Note the axis parameter.  With axis=1 we swap column levels, with axis=0 (the default) we swap row levels.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Knowing the order of the index components and being able to inspect their values and names is fundamental to working with a multi-index.
Exercise: What would happen if we had switched the order of the strings in the list when we called dd.set_index? Try it with this list to find out: ['ISO', 'Country', 'Variable', 'Description', 'Units']
Resetting the index
We've seen that set_index pushes columns into the index.  Here we see that reset_index does the reverse:  it pushes components of the index back to the columns.
Example.
Can you write Python code for it?

ddi.head(2)

ddi.reset_index()

# or we can reset the index by level
ddi.reset_index(level=1).head(2)

# or by name
ddi.reset_index(level='Country').head(2)

# or do more than one at a time
ddi.reset_index(level=[1,3]).head(2)


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Let's get a smaller subset of this data to work with so we can see things a bit more clearly
Can you write Python code for it?

# drop some of the index levels (think s for small)
dds = ddi.reset_index(level=[1,3,4], drop=True)
# give a name to the column labels
dds.columns.name = 'Year'
dds


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Let's remind ourselves what we want. We want to

move the column index (Year) into the row index
move the Variable and ISO levels the other way, into the column labels.

The first one uses stack, the second one unstack.
Stacking
We stack our data, one variable on top of another, with a multi-index to keep track of what's what.  In simple terms, we change the data from a wide format to a long format.  The stack method takes the inner most column level and makes it the lowest row level.
Can you write Python code for it?

# convert to long format. Notice printing is different... what `type` is ds?
ds = dds.stack()
ds

# same thing with explicit reference to column name
dds.stack(level='Year').head(8)

# or with level number
dds.stack(level=0).head(8)


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Let's take a closer look at the index
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Pivoting the data
Let's think specifically about what we want.  We want to graph Emp against fsize for (say) 2013.  This calls for:

The index should be the size categories fsize.
The column labels should be the entries of year2, namely 2012 and 2013.
The data should come from the variable Emp.

These inputs translate directly into the following pivot method:
Can you write Python code for it?

# pivot and divide by a million (dividing so bars aren't too long)
bdsp = bds.pivot(index='fsize', columns='year2', values='Emp')/10**6
bdsp


I am working on the file "Code/notebooks/bootcamp_pandas-shape.ipynb".
The first task is:
Wants
We might imagine doing several different things with this data:

Plot a specific variable (debt or surplus) for a given date.
Time series plots for a specific country.
Time series plots for a specific variable.

Depending on which we want, we might organize the data differently.  We'll focus on the last two.
Here's a brute force approach to the problem:  simply transpose the data.  This is where that leads:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:

Grouping data
Next up:  group data by some variable.  As an example, how would we compute the average rating of each movie?  If you think for a minute, you might think of these steps:

Group the data by movie:  Put all the "Pulp Fiction" ratings in one bin, all the "Shawshank" ratings in another.  We do that with the groupby method.
Compute a statistic (the mean, for example) for each group.

Pandas has tools that make that relatively easy.
Can you write Python code for it?

# group
g = ml[['title', 'rating']].groupby('title')
type(g)


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:

WEO data on government debt
We use the IMF's data on government debt again, specifically its World Economic Outlook database, commonly referred to as the WEO.  We focus on government debt expressed as a percentage of GDP,  variable code GGXWDG_NGDP.
The central question here is how the debt of Argentina, which defaulted in 2001, compared to other countries.  Was it a matter of too much debt or something else?
Load data
First step:  load the data and extract a single variable:  government debt (code GGXWDG_NGDP) expressed as a percentage of GDP.
Can you write Python code for it?

url1 = "http://www.imf.org/external/pubs/ft/weo/2016/02/weodata/"
url2 = "WEOOct2016all.xls"
url = url1 + url2
weo = pd.read_csv(url, sep='\t',
                  usecols=[1,2] + list(range(19,46)),
                  thousands=',',
                  na_values=['n/a', '--'])
print('Variable dtypes:\n', weo.dtypes.head(6), sep='')


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Comment. If we add enough quantiles, we might as well plot the whole distribution.  The easiest way to do this is with a histogram.
Can you write Python code for it?

fig, ax = plt.subplots()
db01.hist(bins=15, ax=ax, alpha=0.35)
ax.set_xlabel('Government Debt (Percent of GDP)')
ax.set_ylabel('Number of Countries')

ymin, ymax = ax.get_ylim()
ax.vlines(db01['ARG'], ymin, ymax, color='blue', lw=2)


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Example. Let's add the mean to our graph.  We make it a dashed line with linestyle='dashed'.
Can you write Python code for it?

fig, ax = plt.subplots()
dbt.plot(ax=ax,
         legend=False, color='blue', alpha=0.2,
         ylim=(0,200)
        )
dbt['ARG'].plot(ax=ax, color='black', linewidth=1.5)
ax.set_ylabel('Percent of GDP')
ax.set_xlabel('')
ax.set_title('Government debt', fontsize=14, loc='left')
dbt.mean(axis=1).plot(ax=ax, color='black', linewidth=2, linestyle='dashed')


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Example. Let's try a simple graph of the dataframe dbt.  The goal is to put Argentina in perspective by plotting it along with many other countries.
Can you write Python code for it?

fig, ax = plt.subplots()
dbt.plot(ax=ax,
         legend=False, color='blue', alpha=0.3,
         ylim=(0,150)
        )
ax.set_ylabel('Percent of GDP')
ax.set_xlabel('')
ax.set_title('Government debt', fontsize=14, loc='left')
dbt['ARG'].plot(ax=ax, color='black', linewidth=1.5)


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Exercise.

What do you take away from this graph?
What would you change to make it look better?
To make it mnore informative?
To put Argentina's debt in context?

Exercise. Do the same graph with Greece (GRC) as the country of interest.  How does it differ?  Why do you think that is?

Describing numerical data
Let's step back a minute.  What we're trying to do is compare Argentina to other countries.  What's the best way to do that?  This isn't a question with an obvious best answer, but we can try some things, see how they look.  One thing we could do is compare Argentina to the mean or median.  Or to some other feature of the distribution.
We work up to this by looking first at some features of the distribution of government debt numbers across countries.  Some of this we've seen, some is new.
What's (not) there?
Let's check out the data first.  How many non-missing values do we have at each date?   We can do that with the count method. The argument axis=1 says to do this by date, counting across columns (axis number 1).
Can you write Python code for it?

dbt.shape

# count non-missing values
dbt.count(axis=1).plot()


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Exercise. Which do we like better?
Exercise. Replace the (constant) mean with the (constant) median?  Which do you prefer?

Describing categorical data
A categorical variable is one that takes on a small number of values.  States take on one of fifty values.  University students are either grad or undergrad.  Students select majors and concentrations.
We're going to do two things with categorical data:

In this section, we count the number of observations in each category using the value_counts method. This is a series method, we apply it to one series/variable at a time.
In the next section, we go on to describe how other variables differ across catagories.  How do students who major in finance differ from those who major in English?  And so on.

We start with the combined MovieLens data we constructed in the previous notebook.
Can you write Python code for it?

url = 'http://pages.stern.nyu.edu/~dbackus/Data/mlcombined.csv'
ml  = pd.read_csv(url, index_col=0,encoding = "ISO-8859-1")
print('Dimensions:', ml.shape)

# fix up the dates
ml["timestamp"] = pd.to_datetime(ml["timestamp"], unit="s")
ml.head(10)

# which movies have the most ratings?
ml['title'].value_counts().head(10)

ml['title'].value_counts().head(10).plot.barh(alpha=0.5)

# which people have rated the most movies?
ml['userId'].value_counts().head(10)


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Now that we have a groupby object, what can we do with it?
Can you write Python code for it?

# the number in each category
g.count().head(10)

# what type of object have we created?
type(g.count())


I am working on the file "Code/notebooks/bootcamp_pandas-summarize.ipynb".
The first task is:
Question. Do you think this looks better when the mean varies with time, or when we use a constant mean?  Let's try it and see.
Can you write Python code for it?

dbar = dbt.mean().mean()
dbar

fig, ax = plt.subplots()
dbt.plot(ax=ax,
         legend=False, color='blue', alpha=0.3,
         ylim=(0,150)
        )
dbt['ARG'].plot(ax=ax, color='black', linewidth=1.5)
ax.set_ylabel('Percent of GDP')
ax.set_xlabel('')
ax.set_title('Government debt', fontsize=14, loc='left')
xmin, xmax = ax.get_xlim()
ax.hlines(dbar, xmin, xmax, linewidth=2, linestyle='dashed')


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:

Boolean selection
We choose observations that satisfy one or more conditions.  Boolean selection consists of two steps that we typically combine in one statement:

Use a comparison to construct a Boolean variable consisting of True and False.
Compute df[comparison], where df is a dataframe and comparison is a comparison.  This will select the observations (rows) for which comparison is true and throw away the others.

We work through this one step at a time:

Example:  apply the want operator
Comparisons for dataframes
Boolean selection:  select observations for which the comparison is True
The isin method

This is easier to describe with an example.
Example:  Apply the want operator to WEO
Our want here is to take the weo dataframe and extract government debt and deficits for a given set of countries.  Putting this to work involves several steps.
Here's the head of the dataframe to remind us what we're dealing with.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:

The contains method
Another useful one.  The contains string method for series identifies observations that contain a specific string.  If yes, the observation is labelled True, if no, False.  A little trick converts the True/False outcomes to ones and zeros.
We apply it to the Topics variable of the Entry Poll dataframe entry_poll.  You may recall that this variable could have more than one response.  We tease them apart with the contains method.  Our want is to have a yes/no variable for each response.
Can you write Python code for it?

# recall
entry_poll['Topics'].head(10)

# the contains method
entry_poll['Topics'].str.contains('Machine Learning')


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:

Selecting variables and observations
The word selection refers to choosing a subset of variables or observations using their labels or index.  Similar methods are sometimes referred to as slicing, subsetting, indexing, querying, or filtering.  We'll treat the terms as synonymous.
There are lots of ways to do this.  Mostly we do "Boolean" selection, which we address in the next section.  We review more direct options here, mostly at high speed because they're not things we use much.
In the outline below, df is a dataframe, var and varn are variable names, n1 and n2 are integers,
- vlist = ['var1', 'var2'] is a list of variable names, and
- nlist = [0, 3, 4] is a list of numerical variable or observation indexes and
- bools is a list or pandas Series of booleans (True and False).
Some of the basic selection/indexing/slicing methods have the form:

df[var] extracts a variable -- a series, in other words.
df[vlist] extracts a new dataframe consisting of the variables in vlist.
df[nlist] does the same thing.
df[bools]: extracts each row where the corresponding element in bools is true. len(bools) must be equal to df.size[0]
df[n1:n2] extracts observations n1 to n2-1, the traditional slicing syntax.
Can you write Python code for it?

# we create a small dataframe to experiment with
small = weo.head()
small


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:

String methods
We can treat variables as strings in Pandas in much the same way we dealt with strings in core Python.  Run the code below to remind yourself how this works.
Can you write Python code for it?

dollars = '$123.45'
print('Type of variable dollars:', type(dollars))
num = dollars.replace('$', '')
num = float(num)
print('Type of variable num:', type(num))


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:

The want operator
We need to know what we're trying to do -- what we want the data to look like. To borrow a phrase from our friend Tom Sargent, we say that we apply the want operator.
Some problems we've run across that ask to be solved:

Numerical data is contaminated by commas (marking thousands) or dollar signs.
Row and column labels are contaminated.
Missing values are marked erratically.
We have too much data, would prefer to choose a subset.
Variables run across rows rather than down columns.

What we want in each case is the opposite of what we have:  we want nicely formatted numbers, clean row and column labels, and so on.
We'll solve the first four problems here, the last one in the next notebook.
Example:  Chipotle data
This data comes from a New York Times story about the number of calories in a typical order at Chipotle.  The topic doesn't particularly excite us, but the data raises a number of issues that come up repeatedly. We adapt some code written by Daniel Forsyth.
Note: The file is a tsv (Tab Separated Values) file, so we need to set the separator accordingly when we call pandas' read_csv method. Remember that the default value of sep is sep=',' (see the docstring). We can change it to tabular by wrinting sep='\t'.
Can you write Python code for it?

url = 'https://raw.githubusercontent.com/TheUpshot/chipotle/master/orders.tsv'
chipotle = pd.read_csv(url, sep='\t')               # tab (\t) separated values
print('Variable dtypes:\n', chipotle.dtypes, sep='')
chipotle.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment.  Note the commas separating answers with more than one choice.  We want to unpack them somehow.
Example:  OECD healthcare statistics
The OECD collects healthcare data on lots of (mostly rich) countries, which is helpful in producing comparisons.  Here we use a spreadsheet that can be found under Frequently Requested Data.
Can you write Python code for it?

url1 = 'http://www.oecd.org/health/health-systems/'
url2 = 'OECD-Health-Statistics-2016-Frequently-Requested-Data.xls'
oecd = pd.read_excel(url1 + url2)
oecd.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. Just for fun, here's a bar graph of the result.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. Not quite, we only want to split once.
Can you write Python code for it?

# what about this?
country.rsplit(maxsplit=1)

# one more step, we want the first component of the list
country.rsplit(maxsplit=1)[0]

oecd.index

oecd.index.str.rsplit(maxsplit=1)[0]

#try
oecd.index.str.rsplit?

# Note the TWO str's
oecd.index.str.rsplit(n=1).str[0]

#or use the str.get() method
oecd.index.str.rsplit(n=1).str.get(0)

oecd.index = oecd.index.str.rsplit(n=1).str.get(0)

oecd.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. Note that the variable item_price has dtype object. The reason is evidently the dollar sign.  We want to have it as a number, specifically a float.
Example:  Data Bootcamp entry poll
This is the poll we did at the start of the course.  Responses were collected in a Google spreadsheet, which we converted to a csv and uploaded to our website.
Can you write Python code for it?

url1 = "https://raw.githubusercontent.com/NYUDataBootcamp/"
url2 = "Materials/master/Data/entry_poll_spring17.csv"
url = url1 + url2
entry_poll = pd.read_csv(url)

entry_poll.head()

print('Dimensions:', entry_poll.shape)

print('Data types:\n\n', entry_poll.dtypes, sep='')


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. Unlike the string methods we described earlier, this use of replace affects complete entries, not elements of string entries.  For example, suppose we tried to replace the periods in decimal numbers with an asterisk.  We could try the following, but it doesn't work:  the decimal numbers  don't change.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. We can do the same thing with countries.  If we want to choose two variables and three countries, the code looks like:
Can you write Python code for it?

variables = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
countries = ['Argentina', 'Greece']
weo_sub = weo[weo['WEO Subject Code'].isin(variables) & weo['Country'].isin(countries)]
weo_sub


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. We did everything here in one line:  replace the dollar sign with a string method, then converted to float using astype.  If you think this is too dense, you might break it into two steps.
Example. Here we use the astype method again to convert the dtypes of weo into float
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. We usually don't have to worry about this, Pandas takes care of missing values automatically.
Comment. Let's try a picture to give us a feeling of accomplishment. What else would you say we need?  How would we get it?
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. We're choosing 2 variables from 45, so there are lots of Falses.
Can you write Python code for it?

weo.tail(4)

# this time let's use the result of isin for selection
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
weo[weo['WEO Subject Code'].isin(vlist)].head(6)

# we've combined several things in one line
comparison = weo['WEO Subject Code'].isin(vlist)
selection  = weo[comparison]
selection.head(6)


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comment. You might want to think about this a minute.  Or two.
Can you write Python code for it?

# create new df of just these variables
student_topics = entry_poll[vnames]
student_topics

# count them with the sum method
topics_counts = student_topics.sum()
topics_counts


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comments.

Note that we need two str's here:  one to do the split, the other to extract the first element.
For reasons that mystify us, we ran into problems when we used maxsplit=1, but it works with n=1.
This is probably more than you want to know, but file away the possibilities in case you need them.

Missing values
It's important to label missing values, so that Pandas doesn't interpret entries as strings.  Pandas is also smart enough to ignore things labeled missing when it does calculations or graphs.  If we compute, for example, the mean of a variable, the default is to ignore missing values.
We've seen that we can label certain entries as missing values in read statements:  read_csv, read_excel, and so on.  Here we do it directly, mostly to remind ourselves what's involved.
Marking missing values
Example. The oecd dataframe contains a number of instances of .. (double period).  How can we mark them as missing values?
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Comments. This is mostly text data, which means it's assigned the dtype object. There are two things that would make the data easier to work with:

First: The column names are excessively verbose.  This one's easy:  We replace them with single words.  Which we do below.
Can you write Python code for it?

# (1) create list of strings with the new varnames
newnames = ['time', 'why', 'program', 'programming', 'prob_stats', 'major', 'career', 'data', 'topics']
newnames

# (2) Use the str.title() string method to make the varnames prettier
newnames = [name.title() for name in newnames]


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Example. Let's try each of these in a different cell and see what they do:

small[['Country', 'Units']]
small[[0, 4]]
small['2011']
small[1:3]

Can you explain the results?
Can you write Python code for it?

small[['Country', 'Units']]

small[[0, 4]]

small['2011']

small[1:3]

small[[False, True, True, False, False]]


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Exercise (challenging). Plot the variable gdp['2015'] as a bar chart.  What would you say it needs?
Can you write Python code for it?