I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Comment. Note that all the parameters here are columns.  That's not a choice, it's the way the the pivot method is written.
We do a plot for fun:
Can you write Python code for it?

# plot 2013 as bar chart
fig, ax = plt.subplots()
bdsp[2013].plot.barh(ax=ax)
ax.set_ylabel('')
ax.set_xlabel('Number of Employees (millions)')


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Comment. We see here that the multi-index for the rows has been turned into a multi-index for the columns.  Works the same way.
The only problem here is that the column labels are more complicated than we might want.  Here, for example, is what we get with the plot method.  As usual, .plot() plots all the columns of the dataframe, but here that means we're mixing variables.  And the legend contains all the levels of the column labels.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise (challenging).  Take the unstacked dataframe dds.  Use some combination of stack, unstack, and plot to plot the variable Surplus against Year for all three countries.   Challenging mostly because you need to work out the steps by yourself.

Pivoting
The pivot method:  a short cut to some kinds of unstacking.  In rough terms, it takes a wide dataframe and constructs a long one.  The inputs are columns, not index levels.
Example:  BDS data
The Census's Business Dynamnics Statistics collects annual information about the hiring decisions of firms by size and age.  This table list the number of firms and total employment by employment size categories:  1 to 4 employees, 5 to 9, and so on.
Apply want operator. Our want is to plot total employment (the variable Emp) against size (variable fsize).  Both are columns in the original data.
Here we construct a subset of the data, where we look at two years rather than the whole 1976-2013 period.
Can you write Python code for it?

url = 'http://www2.census.gov/ces/bds/firm/bds_f_sz_release.csv'
raw = pd.read_csv(url)
raw.head()

sizes = ['a) 1 to 4', 'b) 5 to 9', 'c) 10 to 19', 'd) 20 to 49']
bds = raw[(raw['year2']>=2012) & raw['fsize'].isin(sizes)][['year2', 'fsize', 'Firms', 'Emp']]
bds


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise.  For the dataframe ddi:

Use the reset_index method to move the Units level of the index to a column of the dataframe.
Use the drop parameter of reset_index to delete Units from the dataframe.

Switching rows and columns
If we take the dataframe ddi, we see that the everything's been put into the index but the data itself.  Perhaps we can get what we want if we just flip the rows and columns.  Roughly speaking, we refer to this as pivoting.
First look at switching rows and columns
The simplest way to flip rows and columns is to use the T or transpose property.  When we do that, we end up with a lot of stuff in the column labels, as the multi-index for the rows gets rotated into the columns.  Other than that, we're good. We can even do a plot.  The only problem is all the stuff we've pushed into the column labels -- it's kind of a mess.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. Run the code below and explain what each line of code does.
Can you write Python code for it?

# stacked dataframe
ds.head(8)

du1 = ds.unstack()
du2 = du1.unstack()


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. Set Variable as the index.
Comment. Note that the new index brought its name along:  Country in the two examples, Variable in the exercise.  That's incredibly useful because we can refer to index levels by name.  If we happen to have an index without a name, we can set it with
python
df.index.name = 'Whatever name we like'
Multi-indexes
We can put more than one variable in an index, which gives us a multi-index.  This is sometimes called a hierarchical index because the levels of the index (as they're called) are ordered.
Multi-indexes are more common than you might think.  One reason is that data itself is often multi-dimensional.  A typical spreadsheet has two dimensions:  the variable and the observation.  The WEO data is naturally three dimensional:  the variable, the year, and the country.  (Think about that for a minute, it's deeper than it sounds.)
The problem we're having is fitting this nicely into two dimensions.  A multi-index allows us to manage that.  A two-dimensional index would work here -- the country and the variable code -- but right now we have some redundancy.
Example. We push all the descriptive, non-numerical columns into the index, leaving the dataframe itself with only numbers, which seems like a step in thee right direction.
Can you write Python code for it?

ddi = dd.set_index(['Variable', 'Country', 'ISO', 'Description', 'Units'])
ddi


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. Use a combination of xs and standard slicing with [...] to extract the variable Debt for Greece.
Exercise. Use the dataframe ddt -- and the xs method -- to plot Debt and Surplus across time for Argentina.

Stacking and unstacking
The set_index and reset_index methods work on the row labels -- the index.  They move columns to the index and the reverse.  The stack and unstack methods move index levels to and from column levels:

stack stacks the data up, moving the columns to the index and creating a long dataframe.
unstack does the reverse, moving columns or index levels into the column labels and creating a wide dataframe.

We use both to shape (or reshape) our data.  We use set_index to push things into the index.  And then use reset_index to push some of them back to the columns.  That gives us pretty fine-grainded control over the shape of our data.
We start by simplifying our initial dataframe.
Can you write Python code for it?

# drop some of the index levels (think s for small)
dds = ddi.reset_index(level=[1,3,4], drop=True)
# give a name to the column labels
dds.columns.name = 'Year'
dds


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Exercise. Use the dataframe ddts to plot Debt and Surplus across time for Argentina.  Hint: In the plot method, set subplots=True so that each variable is in a separate subplot.
The xs method
Another approach to extracting data that cuts across levels of the row or column index:  the xs method. This is recent addition tpo Pandas and an extremely good method once you get the hang of it.
The basic syntax is df.xs(item, axis=X, level=N), where N is the name or number of an index level and X describes if we are extracting from the index or column names. Setting X=0 (so axis=0) will slice up the data along the index, X=1 extracts data for column labels.
Here's how we could use xs to get the Argentina data without swapping the level of the column labels
Can you write Python code for it?

#ddt.xs?

ddt.xs("Argentina", axis=1, level="Country")

ddt.xs("Argentina", axis=1, level="Country")["Debt"]


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Let's remind ourselves what we want. We want to

move the column index (Year) into the row index
move the Variable and ISO levels the other way, into the column labels.

The first one uses stack, the second one unstack.
Stacking
We stack our data, one variable on top of another, with a multi-index to keep track of what's what.  In simple terms, we change the data from a wide format to a long format.  The stack method takes the lowest column level and makes it the lowest row level.
Can you write Python code for it?

# convert to long format. Notice printing is different... what `type` is ds?
ds = dds.stack()
ds

# same thing with explicit reference to column name
dds.stack(level='Year').head(8)

# or with level number
dds.stack(level=0).head(8)


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Let's take a closer look at the index
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Pivoting the data
Let's think specifically about what we want.  We want to graph Emp against fsize for (say) 2013.  This calls for:

The index should be the size categories fsize.
The column labels should be the entries of year2, namely 2012 and 2013.
The data should come from the variable Emp.

These inputs translate directly into the following pivot method:
Can you write Python code for it?

# pivot and divide by a million (dividing so bars aren't too long)
bdsp = bds.pivot(index='fsize', columns='year2', values='Emp')/10**6
bdsp


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Reminders
What kind of object does each of the following produce?
Can you write Python code for it?

dd.index

dd.columns

dd['ISO']

dd[['ISO', 'Variable']]

dd[dd['ISO'] == 'ARG']


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
That's a lot to process, so we break it into pieces.

ddi.index.names contains a list of level names.  (Remind yourself that lists are ordered, so this tracks levels.)
ddi.index.levels contains the values in each level.

Here's what they like like here:
Can you write Python code for it?

# Chase and Spencer like double quotes
print("The level names are:\n", ddi.index.names, "\n", sep="")
print("The levels (aka level values) are:\n", ddi.index.levels, sep="")


I am working on the file "Code/IPython/bootcamp_pandas-shape.ipynb".
The first task is:
Unstacking
Stacking moves columns into the index, "stacking" the data up into longer columns.  Unstacking does the reverse, taking levels of the row index and turning them into column labels.  Roughly speaking we're rotating or pivoting the data.
Can you write Python code for it?

# now go long to wide
ds.unstack()  # default is lowest value level='ISO'

# different level
ds.unstack(level='Variable')

# or two at once
ds.unstack(level=['Variable', 'ISO'])


I am working on the file "Code/IPython/bootcamp_pandas-summarize.ipynb".
The first task is:

Grouping data
Next up:  group data by some variable.  As an example, how would we compute the average rating of each movie?  If you think for a minute, you might think of these steps:

Group the data by movie:  Put all the "Pulp Fiction" ratings in one bin, all the "Shawshank" ratings in another.  We do that with the groupby method.
Compute a statistic (the mean, for example) for each group.

Pandas has tools that make that relatively easy.
Can you write Python code for it?

# group
g = ml[['title', 'rating']].groupby('title')
type(g)


I am working on the file "Code/IPython/bootcamp_pandas-summarize.ipynb".
The first task is:

WEO data on government debt
We use the IMF's data on government debt again, specifically its World Economic Outlook database, commonly referred to as the WEO.  We focus on government debt expressed as a percentage of GDP,  variable code GGXWDG_NGDP.
The central question here is how the debt of Argentina, which defaulted in 2001, compared to other countries.  Was it a matter of too much debt or something else?
Load data
First step:  load the data and extract a single variable:  government debt (code GGXWDG_NGDP) expressed as a percentage of GDP.
Can you write Python code for it?

url1 = 'http://www.imf.org/external/pubs/ft/weo/2015/02/weodata/'
url2 = 'WEOOct2015all.xls'
url = url1 + url2
weo = pd.read_csv(url, sep='\t',
                  usecols=[1,2] + list(range(19,45)),
                  thousands=',',
                  na_values=['n/a', '--'])
print('Variable dtypes:\n', weo.dtypes.head(6), sep='')


I am working on the file "Code/IPython/bootcamp_pandas-summarize.ipynb".
The first task is:
Comment. If we add enough quantiles, we might as well plot the whole distribution.  The easiest way to do this is with a histogram.
Can you write Python code for it?

fig, ax = plt.subplots()
db01.hist(bins=15, ax=ax, alpha=0.35)
ax.set_xlabel('Government Debt (Percent of GDP)')
ax.set_ylabel('Number of Countries')

darg = db01['ARG']
ymin, ymax = ax.get_ylim()
ax.vlines(darg, ymin, ymax, color='blue', lw=2)


I am working on the file "Code/IPython/bootcamp_pandas-summarize.ipynb".
The first task is:
Describing dataframes
We can compute the same statistics for dataframes.  Here we hve a choice:  we can compute (say) the mean down rows (axis=0) or across columns (axis=1).  If we use the dataframe dbt, computing the mean across countries (columns) calls for axis=1.
Can you write Python code for it?

# here we compute the mean across countries at every date
dbt.mean(axis=1).head()

# or we could do the median
dbt.median(axis=1).head()

# or a bunch of stats at once
# NB:  db not dbt (there's no axix argument here)
db.describe()

# the other way
dbt.describe()


I am working on the file "Code/IPython/bootcamp_pandas-summarize.ipynb".
The first task is:
Exercise.

What do you take away from this graph?
What would you change to make it look better?
To make it mnore informative?
To put Argentina's debt in context?

Exercise. Do the same graph with Greece (GRC) as the country of interest.  How does it differ?  Why do you think that is?

Describing numerical data
Let's step back a minute.  What we're trying to do is compare Argentina to other countries.  What's the best way to do that?  This isn't a question with an obvious best answer, but we can try some things, see how they look.  One thing we could do is compare Argentina to the mean or median.  Or to some other feature of the distribution.
We work up to this by looking first at some features of the distribution of government debt numbers across countries.  Some of this we've seen, so is new.
What's (not) there?
Let's check out the data first.  How many non-missing values do we have at each date?   We can do that with the count method. The argument axis=1 says to do this by date, counting across columns (axis number 1).
Can you write Python code for it?

dbt.shape

# count non-missing values
dbt.count(axis=1).plot()


I am working on the file "Code/IPython/bootcamp_pandas-summarize.ipynb".
The first task is:
Now that we have a groupby object, what can we do with it?
Can you write Python code for it?

# the number in each category
g.count().head(10)

# what type of object have we created?
type(g.count())


I am working on the file "Code/IPython/bootcamp_practice_a.ipynb".
The first task is:
Comment. Now we have the number of people in any five-year age group running down columns.  The column labels are the years.
With the dataframe df:

Plot the current age distribution with pop[[2015]].plot().  Note that 2015 here does not have quotes around it:  it's an unusual case of integer column labels.
Plot the current age distribution as a bar chart.  Which do you think looks better?
Create figure and axis objects
Use the axis object to plot the age distribution for all the years in the dataframe.
Add titles and axis labels.
Plot the age distribution for each date in a separate subplot.  What argument parameter does this?  Bonus points: Change the size of the figure to accomodate the subplots.

Question 5.  Dynamics of the yield curve
One of our favorite topics is the yield curve:  a plot of the yield to maturity on a bond against the bond's maturity.  The foundation here is yields on zero coupon bonds, which are simpler objects than yields on coupon bonds.
We often refer to bond yields rising or falling, but in fact the yield curve often does different things at different maturities.  We will see that here.  For several years, short yields have been stuck at zero, yet yields for bond with maturities of two years and above have varied quite a bit.
We use the Fed's well-known Gurkaynak, Sack, and Wright data, which provides daily data on US Treasury yields from 1961 to the present.  The Fed posts the data, but it's in an unfriendly format.  So we saved it as a csv file, which we read in below.  The variables are yields:  SVENYnn is the yield for maturity nn years.
Can you write Python code for it?

# data input (takes about 20 seconds on a wireless network)
url = 'http://pages.stern.nyu.edu/~dbackus/Data/feds200628.csv'
gsw = pd.read_csv(url, skiprows=9, index_col=0, usecols=list(range(11)), parse_dates=True)
print('Dimensions: ', gsw.shape)
print('Column labels: ', gsw.columns)
print('Row labels: ', gsw.index)

# grab recent data
df = gsw[gsw.index >= dt.datetime(2010,1,1)]
# convert to annual, last day of year
df = df.resample('A', how='last').sort_index()
df.head()

df.columns = list(range(1,11))
ylds = df.T
ylds.head(3)


I am working on the file "Code/IPython/bootcamp_practice_a.ipynb".
The first task is:
If you get something like "Pandas version:  0.17.1" you're fine.  If you get an error, bring your computer by and ask for help.  If you're unusually brave, go to StackOverflow and read the instructions.  Then come ask for help.  (This has to do with how your computer processes unicode.  When you hear that word -- unicode -- you should run away at high speed.)
Question 1. Setup
Import packages, arrange for graphs to display in the notebook.
Can you write Python code for it?

import pandas as pd
import matplotlib.pyplot as plt
import datetime as dt
%matplotlib inline


I am working on the file "Code/IPython/bootcamp_practice_a.ipynb".
The first task is:
Question. Can you see consolidation here?
Can you write Python code for it?

# a basic plot
fig, ax = plt.subplots()

pdf[1947].plot(ax=ax, logy=True)
pdf[1967].plot(ax=ax, logy=True)
pdf[1987].plot(ax=ax, logy=True)
pdf[2004].plot(ax=ax, logy=True)
ax.legend()


I am working on the file "Code/IPython/bootcamp_practice_a.ipynb".
The first task is:
Remind yourself:

What does the pandas package do?
What does the matplotlib package do?
What does %matplotlib inline do?

Question 2.  Jupyter basics

We refer to the cell that's highlighted as the current cell.
Clicking once on any cell makes it the current cell.  Clicking again allows you to edit it.
The + in the toolbar at the top creates a new cell below the current cell.
Change a cell from Code to Markdown (in other words, text) with the dropdown menu in the toolbar.
To run a cell, hit shift-enter or click on the run-cell icon in the tooolbar (sideways triangle and vertical line).
For more information, click on Help at the top.  User Interface Tour is a good place to start.

Practice with the following:

Make this cell the current cell.
Add an empty cell below it.
Add text to the new cell:  your name and the date, for example.
Optional:  Add a link to your LinkedIn or Facebook page.  Hint: Look at the text in the top cell to find an example of a link.
Run the cell.

Question 3. Winner take all and the long tail in the US beer industry
The internet has produced some interesting market behavior, music being a great example.  Among them:

Winner take all.  The large producers (Beyonce, for example) take larger shares of the market than they had in the past.
The long tail.  At the same time, small producers in aggregate increase their share.

Curiously enough, we see the same thing in the US beer industry:

Scale economies and a reduction in transportation costs (the interstate highway system was built in the 1950s and 60s) led to consolidation, with the large firms getting larger, and the small ones either sellingout or going bankrupt.  (How many beer brands can you think of that no longer exist?)
Starting in the 1980s, we saw a significant increase in the market share of small firms ("craft brewers") overall, even though each of them remains small.

We illustrate this with data from Victor and Carol Tremblay that describe the output of the top 100 US beer producers...
Can you write Python code for it?

url = 'http://pages.stern.nyu.edu/~dbackus/Data/beer_production_1947-2004.xlsx'
beer = pd.read_excel(url, skiprows=12, index_col=0)

print('Dimensions:', beer.shape)
beer[list(range(1,11))].head(3)

vars = list(range(1,101))   # extract top 100 firms
pdf = beer[vars].T          # transpose (flip rows and columns)
pdf[[1947, 1967, 1987, 2004]].head()


I am working on the file "Code/IPython/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Answer these questions below. Code is sufficient, but it's often helpful to add comments to remind yourself what you did, and why.

Get help for the set.title method by typing ax.set_title? in a new cell and running it.  Note that you can open the documentation this produces in a separate tab with the icon in the upper right (hover text = "Open the pager in an external window").
Add a title with ax.set_title('Your title').
Change the fontsize of the title to 14.
What happens if we add the argument/parameter lw=2 to the ax.plot() statements?
Add a label to the x axis with ax.set_xlabel().
Add a label to the y axis.
Why did we use a log scale (logy=True)?  What happens if we don't?
Use the color argument/parameter to choose a more effective set of colors.
In what sense do you see "winner takes all"?  A "long tail"?

Put each answer in a separate code cell.
Can you write Python code for it?

# for help
ax.set_title?

# this is easier if we put the basic plot in a function
def make_plot():
    fig, ax = plt.subplots()
    pdf[1947].plot(ax=ax, logy=True)
    pdf[1967].plot(ax=ax, logy=True)
    pdf[1987].plot(ax=ax, logy=True)
    pdf[2004].plot(ax=ax, logy=True)
    ax.legend()
    return ax

ax = make_plot()
ax.set_title('Beer sales by industry rank', fontsize=14)

# line width:  put lw=2 in each of the plot statements

ax = make_plot()
ax.set_xlabel('Industry Rank')
ax.set_ylabel('Sales (log scale)')

# log scale:  otherwise the differences are too large
# we can't show the alternative because some of the numbers are zero

# color:  we add color='somecolor' in each of the plot statements


I am working on the file "Code/IPython/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Comment. Now we have the number of people in any five-year age group running down columns.  The column labels are the years.
With the dataframe df:

Plot the current age distribution with pop[[2015]].plot().  Note that 2015 here does not have quotes around it:  it's an unusual case of integer column labels.
Plot the current age distribution as a bar chart.  Which do you think looks better?
Create figure and axis objects
Use the axis object to plot the age distribution for all the years in the dataframe.
Add titles and axis labels.
Plot the age distribution for each date in a separate subplot.  What argument parameter does this?  Bonus points: Change the size of the figure to accomodate the subplots.
Can you write Python code for it?

pop[[2015]].plot()

pop[[2015]].plot(kind='bar')

# my fav
pop[[2015]].plot(kind='barh')

fig, ax = plt.subplots(figsize=(10,6))
pop.plot(ax=ax)
ax.set_title('Population by age')
ax.set_xlabel('Population (millions)')
ax.set_ylabel('Age Range')

pop.plot(kind='bar', subplots=True, figsize=(6,8), sharey=True)


I am working on the file "Code/IPython/bootcamp_practice_a_answerkey.ipynb".
The first task is:
If you get something like "Pandas version:  0.17.1" you're fine.  If you get an error, bring your computer by and ask for help.  If you're unusually brave, go to StackOverflow and read the instructions.  Then come ask for help.  (This has to do with how your computer processes unicode.  When you hear that word -- unicode -- you should run away at high speed.)
Question 1. Setup
Import packages, arrange for graphs to display in the notebook.
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Question. Can you see consolidation here?
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Remind yourself:

What does the pandas package do?  [data management]
What does the matplotlib package do?  [graphics]
What does %matplotlib inline do?  [displays plots in the notebook]

Question 2.  Jupyter basics

We refer to the cell that's highlighted as the current cell.
Clicking once on any cell makes it the current cell.  Clicking again allows you to edit it.
The + in the toolbar at the top creates a new cell below the current cell.
Change a cell from Code to Markdown (in other words, text) with the dropdown menu in the toolbar.
To run a cell, hit shift-enter or click on the run-cell icon in the tooolbar (sideways triangle and vertical line).
For more information, click on Help at the top.  User Interface Tour is a good place to start.

Practice with the following:

Make this cell the current cell.
Add an empty cell below it.
Add text to the new cell:  your name and the date, for example.
Optional:  Add a link to your LinkedIn or Facebook page.  Hint: Look at the text in the top cell to find an example of a link.
Run the cell.

Question 3. Winner take all and the long tail in the US beer industry
The internet has produced some interesting market behavior, music being a great example.  Among them:

Winner take all.  The large producers (Beyonce, for example) take larger shares of the market than they had in the past.
The long tail.  At the same time, small producers in aggregate increase their share.

Curiously enough, we see the same thing in the US beer industry:

Scale economies and a reduction in transportation costs (the interstate highway system was built in the 1950s and 60s) led to consolidation, with the large firms getting larger, and the small ones either sellingout or going bankrupt.  (How many beer brands can you think of that no longer exist?)
Starting in the 1980s, we saw a significant increase in the market share of small firms ("craft brewers") overall, even though each of them remains small.

We illustrate this with data from Victor and Carol Tremblay...
Can you write Python code for it?


I am working on the file "Code/IPython/bootcamp_test.ipynb".
The first task is:
Now the test:

If you get the message below "Program halted, old version of Python, etc," you need to go back and install Anaconda again, this time following the instructions EXACTLY!  Yes, we know that's discouraging, but better to know now than run into problems later.

If you get the message "Congratulations etc," you're all set.  Pat yourself on the back. You can close out the program and go back to whatever you were doing.
Can you write Python code for it?

import sys

print('What version of Python are we running? \n', sys.version, '\n', sep='')

if float(sys.version_info[0]) < 3.0 :
    raise Exception('Program halted, old version of Python. \n',
                    'Sorry, you need to install Anaconda again.')
else:
    print('Congratulations, Python is up to date!')


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
A function to estimate the emissions intensity of each fuel over time, making sure that they add up to the total emissions intensity.
Can you write Python code for it?

def generation_index(gen_df, index_df, group_by='year'):
    """
    Calculate the emissions intensity of each fuel in each time period. Use the
    adjusted total emissions from the index dataframe to ensure that the weighted
    sum of fuel emission intensities will equal the total index value.
    """
    final_adj_co2 = index_df.loc[:,'final CO2 (kg)'].copy()

    calc_total_co2 = gen_df.groupby(group_by)['elec fuel CO2 (kg)'].sum().values
#     calc_total_co2 = calc_total_co2.reset_index()

    for fuel in gen_df['fuel category'].unique():
        gen_df.loc[gen_df['fuel category']==fuel, 'adjusted CO2 (kg)'] = (gen_df.loc[gen_df['fuel category']==fuel,
                                                                                'elec fuel CO2 (kg)'] /
                                                                           calc_total_co2 *
                                                                           final_adj_co2.values)

    gen_df['adjusted index (g/kWh)'] = gen_df['adjusted CO2 (kg)']  /  gen_df['generation (MWh)']
    gen_df['adjusted index (lb/MWh)'] = gen_df['adjusted index (g/kWh)'] * 2.2046


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Adjust CO2 emissions where we have a CO2 ratio value
Because of the inner merge above, all rows should have a valid CO2 ratio
Can you write Python code for it?

# Calaculated with an "inner" merge of the dataframes
for year in range(2001, 2017):
    total_co2 = epa_adj.loc[epa_adj['YEAR']==year, 'CO2_MASS (kg)'].sum()
    union_co2 = epa_adj.loc[(epa_adj['YEAR']==year) &
                            ~(epa_adj['CO2 ratio'].isnull()), 'CO2_MASS (kg)'].sum()
    missing = total_co2 - union_co2

    print year, '{:.3%}'.format(union_co2/total_co2), 'accounted for', \
           missing/1000, 'metric tons missing'


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Apply the function above to each generation dataframe
Can you write Python code for it?

generation_index(eia_gen_annual, annual_index, 'year')

generation_index(eia_gen_monthly, monthly_index, ['year', 'month'])

generation_index(eia_gen_quarterly, quarterly_index, 'year_quarter')

eia_gen_annual.head()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Calculate extra electric fuel CO2 emissions
Can you write Python code for it?

path = os.path.join('Final emission factors.csv')
ef = pd.read_csv(path, index_col=0)


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Consolidate total EIA to monthly gen and emissions
Only keep non-overlapping fuel categories so that my totals are correct (e.g. don't keep utility-scale photovoltaic, because it's already counted in utility-scale solar [SUN]).
Can you write Python code for it?

keep_types = [u'WWW', u'WND', u'WAS', u'SUN', 'DPV', u'NUC', u'NG',
       u'PEL', u'PC', u'OTH', u'COW', u'OOG', u'HPS', u'HYC', u'GEO']
keep_cols = ['generation (MWh)', 'total fuel (mmbtu)', 'elec fuel (mmbtu)',
             'all fuel CO2 (kg)', 'elec fuel CO2 (kg)']
eia_total_monthly = eia_total.loc[(eia_total['type'].isin(keep_types))].groupby(['type', 'year', 'month'])[keep_cols].sum()

eia_total_monthly.head()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Contents

Import data
EIA facility data
Total EIA gen & emissions
EPA emissions data

Check EIA facility against EIA total
Adjust EPA facility emissions
Difference between EIA facility and total
Combine all data (monthly)
Create plots
Monthly Index
Quarterly Index
Annual Index

Import cleaned data

Facility generation and CO2 emissions
Total generation and CO2 emissions by fuel
EPA CO2 emissions

Create some helper functions to add datetime and quarter columns
Can you write Python code for it?

def add_datetime(df, year='year', month='month'):
    df['datetime'] = pd.to_datetime(df[year].astype(str) + '-' + df[month].astype(str),
                                    format='%Y-%m')

def add_quarter(df, year='year', month='month'):
    add_datetime(df, year, month)
    df['quarter'] = df['datetime'].dt.quarter


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Correct EPA facility emissions
Use the EIA facility adjustment factors to correct for CHP and biomass emissions
Use an inner merge rather than left
Justification: a left merge will retain CO2 emissions from facilities that aren't included in 923. But the generation and emissions for those facilities are included in the state-level estimates.
Can you write Python code for it?

eia_keep = ['month', 'year', 'all fuel total CO2 (kg)', 'CO2 ratio', 'plant id']

epa_adj = epa.merge(eia_facility_grouped[eia_keep], left_on=['ORISPL_CODE', 'YEAR', 'MONTH'],
                right_on=['plant id', 'year', 'month'], how='inner') # how='left

epa_adj.drop(['month', 'year', 'plant id'], axis=1, inplace=True)
epa_adj['epa index'] = epa_adj.loc[:,'CO2_MASS (kg)'] / epa_adj.loc[:,'GLOAD (MW)']
epa_adj.head()

sns.jointplot('CO2_MASS (kg)', 'all fuel total CO2 (kg)', epa_adj, marker='.')


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Create a new df that groups the facility data into more general fuel types that match up with the EIA generation and fuel use totals.
Can you write Python code for it?

eia_facility_fuel = eia_facility.copy()
for key in facility_fuel_cats.keys():
    eia_facility_fuel.loc[eia_facility_fuel['fuel'].isin(facility_fuel_cats[key]),'type'] = key
eia_facility_fuel = eia_facility_fuel.groupby(['type', 'year', 'month']).sum()
# eia_facility_fuel.reset_index(inplace=True)

eia_facility_fuel.head()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
EIA Facility level emissions (consolidate fuels/prime movers)
Because EIA tracks all fuel consumption at facilities that might produce both electricity and useful thermal output (CHP), CO2 emissions can be from one 4 categories:
1. Total fuel consumption for all uses (fossil & non-fossil, electricity & thermal output)
2. Fossil fuel consumption for all uses (electricity only or CHP)
3. Total fuel consumption for electricity only
4. Fossil fuel consumption for electricity only
We are interested in Category 4. EPA reports total emissions (Category 1), which need to be adjusted. To do this, we calculate a ratio
$$CO_2 \ Ratio = \frac{Category \ 4}{Category \ 1}$$
Will will apply the CO2 ratio factors to EPA data later in this notebook.
Can you write Python code for it?

cols = ['all fuel fossil CO2 (kg)','elec fuel fossil CO2 (kg)',
        'all fuel total CO2 (kg)','elec fuel total CO2 (kg)', 'generation (MWh)']
eia_facility_grouped = eia_facility.groupby(['year', 'month', 'plant id'])[cols].sum()
eia_facility_grouped.reset_index(inplace=True)
eia_facility_grouped['CO2 ratio'] = eia_facility_grouped['elec fuel fossil CO2 (kg)'] / eia_facility_grouped['all fuel total CO2 (kg)']
eia_facility_grouped['CO2 ratio'].fillna(0, inplace=True)
eia_facility_grouped.head()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Export to Excel file
Can you write Python code for it?

path = os.path.join('..', 'Calculated values', 'US Generation By Fuel Type.xlsx')
writer = pd.ExcelWriter(path, engine='xlsxwriter')

eia_gen_monthly.to_excel(writer, sheet_name='Monthly', index=False)

eia_gen_quarterly.to_excel(writer, sheet_name='Quarterly', index=False)

eia_gen_annual.to_excel(writer, sheet_name='Annual', index=False)
writer.save()


'US POWER SECTOR CO2 EMISSIONS INTENSITY'.title()

path = os.path.join('..', 'Calculated values', 'US Power Sector CO2 Emissions Intensity.xlsx')
writer = pd.ExcelWriter(path)

monthly_index.to_excel(writer, sheet_name='Monthly', index=False)
quarterly_index.to_excel(writer, sheet_name='Quarterly', index=False)
annual_index.to_excel(writer, sheet_name='Annual', index=False)
writer.save()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Extra generation and fuel use
Can you write Python code for it?

eia_total_monthly.head()

iterables = [eia_total_monthly.index.levels[0], range(2001, 2017), range(1, 13)]
index = pd.MultiIndex.from_product(iterables=iterables, names=['type', 'year', 'month'])
eia_extra = pd.DataFrame(index=index, columns=['total fuel (mmbtu)', 'generation (MWh)',
                                               'elec fuel (mmbtu)'])

idx = pd.IndexSlice

use_columns=['total fuel (mmbtu)', 'generation (MWh)',
                                               'elec fuel (mmbtu)']
eia_extra = (eia_total_monthly.loc[idx[:,:,:], use_columns] -
             eia_facility_fuel.loc[idx[:,:,:], use_columns])

# I have lumped hydro pumped storage in with conventional hydro in the facility data.
# Because of this, I need to add HPS rows so that the totals will add up correctly.
# Also need to add DPV because it won't show up otherwise
eia_extra.loc[idx[['HPS', 'DPV'],:,:], use_columns] = eia_total_monthly.loc[idx[['HPS', 'DPV'],:,:], use_columns]
# eia_extra = eia_extra.loc[idx[:, 2003:, :],:]

eia_extra.head()

eia_extra.loc[idx['DPV',:,:]]


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Fill nan's with 0
Can you write Python code for it?


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Generation by fuel
Can you write Python code for it?

fuel_cats = {'Coal' : [u'COW'],
             'Natural Gas' : [u'NG'],
             'Nuclear' : ['NUC'],
             'Renewables' : [u'GEO', u'HYC', u'SUN', 'DPV',
                             u'WAS', u'WND', u'WWW'],
             'Other' : [u'OOG', u'PC', u'PEL', u'OTH', u'HPS']
             }
keep_types = [u'WWW', u'WND', u'WAS', u'SUN', 'DPV', u'NUC', u'NG',
       u'PEL', u'PC', u'OTH', u'COW', u'OOG', u'HPS', u'HYC', u'GEO']

eia_gen_monthly = eia_total.loc[eia_total['type'].isin(keep_types)].groupby(['type', 'year', 'month']).sum()
eia_gen_monthly.reset_index(inplace=True)
eia_gen_monthly.drop(['end', 'sector', 'start'], inplace=True, axis=1)

for key, values in fuel_cats.iteritems():
    eia_gen_monthly.loc[eia_gen_monthly['type'].isin(values),'fuel category'] = key

eia_gen_monthly = eia_gen_monthly.groupby(['fuel category', 'year', 'month']).sum()
eia_gen_monthly.reset_index(inplace=True)

add_quarter(eia_gen_monthly)

eia_gen_quarterly = eia_gen_monthly.groupby(['fuel category', 'year', 'quarter']).sum()
eia_gen_quarterly.reset_index(inplace=True)
eia_gen_quarterly['year_quarter'] = (eia_gen_quarterly['year'].astype(str) +
                                     ' Q' + eia_gen_quarterly['quarter'].astype(str))
eia_gen_quarterly.drop('month', axis=1, inplace=True)

eia_gen_annual = eia_gen_monthly.groupby(['fuel category', 'year']).sum()
eia_gen_annual.reset_index(inplace=True)
eia_gen_annual.drop(['month', 'quarter'], axis=1, inplace=True)


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
If CEMS reported CO2 emissions are 0 but heat inputs are >0 and calculated CO2 emissions are >0, change the adjusted CO2 to NaN. These NaN values will be replaced by the calculated value later. Do the same for low index records (<300 g/kWh). If there is a valid CO2 ratio, multiply the adjusted CO2 column by the CO2 ratio.
Can you write Python code for it?

epa_adj.loc[~(epa_adj['CO2_MASS (kg)']>0) &
            (epa_adj['HEAT_INPUT (mmBtu)']>0) &
            (epa_adj['all fuel total CO2 (kg)']>0), 'adj CO2 (kg)'] = np.nan
epa_adj.loc[(epa_adj['epa index']<300) &
            (epa_adj['HEAT_INPUT (mmBtu)']>0) &
            (epa_adj['all fuel total CO2 (kg)']>0), 'adj CO2 (kg)'] = np.nan

epa_adj.loc[epa_adj['CO2 ratio'].notnull(), 'adj CO2 (kg)'] *= epa_adj.loc[epa_adj['CO2 ratio'].notnull(), 'CO2 ratio']

for year in range(2001,2017):
    num_missing = len(epa_adj.loc[(epa_adj['adj CO2 (kg)'].isnull()) &
                                  (epa_adj['YEAR']==year), 'ORISPL_CODE'].unique())
    total = len(epa_adj.loc[epa_adj['YEAR']==year, 'ORISPL_CODE'].unique())

    print 'In', str(year) + ',', num_missing, 'plants missing some data out of', total


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Load EPA data
Check to see if there are multiple rows per facility for a single month
Can you write Python code for it?

path = os.path.join('Monthly EPA emissions.csv')
epa = pd.read_csv(path)

add_quarter(epa, year='YEAR', month='MONTH')
epa.head()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Look back at this to ensure that I'm correctly accounting for edge cases
- Emissions reported to CEMS under a different code than EIA
- Emissions reported to CEMS but not EIA monthly
- Incorrect 0 value reported to CEMS
Start by setting all adjusted CO2 (adj CO2 (kg)) values to the reported CO2 value
Can you write Python code for it?

epa_adj['adj CO2 (kg)'] = epa_adj.loc[:,'CO2_MASS (kg)']


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Monthly Index
Adding generation and emissions not captured in the facility-level data
Can you write Python code for it?

monthly_index = final_co2_gen.groupby(['year', 'month'])['generation (MWh)', 'final CO2 (kg)'].sum()
monthly_index.reset_index(inplace=True)

# Add extra generation and emissions not captured by facility-level data
monthly_index.loc[:,'final CO2 (kg)'] += eia_extra.reset_index().groupby(['year', 'month'])['elec fuel CO2 (kg)'].sum().values
monthly_index.loc[:,'generation (MWh)'] += eia_extra.reset_index().groupby(['year', 'month'])['generation (MWh)'].sum().values
add_quarter(monthly_index)
monthly_index['index (g/kWh)'] = monthly_index.loc[:, 'final CO2 (kg)'] / monthly_index.loc[:, 'generation (MWh)']

change_since_2005(monthly_index)
g2lb(monthly_index)
monthly_index.dropna(inplace=True)

monthly_index.tail()

path = os.path.join('Data for plots', 'Monthly index.csv')
monthly_index.to_csv(path, index=False)


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
Total EIA generation and CO2 emissions
Can you write Python code for it?

path = os.path.join('EIA country-wide gen fuel CO2.csv')
eia_total = pd.read_csv(path, parse_dates=['datetime'], low_memory=False)

eia_total['type'].unique()


I am working on the file "Code/Index and Generation Calculations.ipynb".
The first task is:
We need to approximate some of the emission factors because the state-level EIA data is only available in the bulk download at an aggregated level. Natural gas usually makes up the bulk of this extra electric generation/fuel use (consumption not reported by facilities, estimated by EIA), and it is still a single fuel here.
Can you write Python code for it?

fuel_factors = {'NG' : ef.loc['NG', 'Fossil Factor'],
                   'PEL': ef.loc[['DFO', 'RFO'], 'Fossil Factor'].mean(),
                   'PC' : ef.loc['PC', 'Fossil Factor'],
                   'COW' : ef.loc[['BIT', 'SUB'], 'Fossil Factor'].mean(),
                   'OOG' : ef.loc['OG', 'Fossil Factor']}

# Start with 0 emissions in all rows
# For fuels where we have an emission factor, replace the 0 with the calculated value
eia_extra['all fuel CO2 (kg)'] = 0
eia_extra['elec fuel CO2 (kg)'] = 0
for fuel in fuel_factors.keys():
    eia_extra.loc[idx[fuel,:,:],'all fuel CO2 (kg)'] = \
        eia_extra.loc[idx[fuel,:,:],'total fuel (mmbtu)'] * fuel_factors[fuel]

    eia_extra.loc[idx[fuel,:,:],'elec fuel CO2 (kg)'] = \
        eia_extra.loc[idx[fuel,:,:],'elec fuel (mmbtu)'] * fuel_factors[fuel]

# eia_extra.reset_index(inplace=True)
# add_quarter(eia_extra)

eia_extra.loc[idx['NG',:,:],].tail()


I am working on the file "Code/Introduction To Python/1 - Strings, Numbers and Booleans.ipynb".
The first task is:
Floats (Floating-Point Numbers)
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/1 - Strings, Numbers and Booleans.ipynb".
The first task is:
Floats support the same operations as integers.
Gotchas
One thing to look out for is division involving integers.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/1 - Strings, Numbers and Booleans.ipynb".
The first task is:
Integers support all of the operations you would expect:
* + Addition
* - Subtraction
* * Multiplication
* / Division
* % Modulo
Can you write Python code for it?

5 + 3

5 - 3

5 * 3

5 / 3 # This is weird. More on this later.

5 % 3 # The remainder of 5 / 3


I am working on the file "Code/Introduction To Python/1 - Strings, Numbers and Booleans.ipynb".
The first task is:
Mathematically this should be 0.75, but as the computation is using integers the result rounds down. Using floats on the other hand yields more accurate result.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/1 - Strings, Numbers and Booleans.ipynb".
The first task is:
On a computer, floating-point numbers are represented using a finite number of bits (ie. memory). This means that not every real number has a floating-point number associated with it, meaning that computers need to perform rounding to represent floating-point numbers and the results of computations on floating-point numbers. This is a source of error that needs to be taken into account when performing numerical computations.
For more details look here.
Computations with Mixed Number Types
In computations involving integers and floats, the results will be "upgraded" to floats.
Can you write Python code for it?

x = 1
y = 2.
z = x + y
print(type(x))
print(type(y))
print(type(z))

3./4


I am working on the file "Code/Introduction To Python/1 - Strings, Numbers and Booleans.ipynb".
The first task is:
They support the kinds of operations you would expect from boolean logic.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
As expected lis_copy now has 9 at the end of it.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
Extracting Elements from Lists
Given a list it is possible to extract its elements in much the same way you would a tuple.
Can you write Python code for it?

# Creating a list and assigning it to the variable x.
lis = [1, 2, 3, 4, 5]
lis

lis[0] # Extract the first element.

lis[1] # Extract the second element.

lis[-1] # Extract the last element.

lis[-2] # Extract the second to last element.


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
Extracting Elements from Tuples
Given a tuple it is possible to extract the elements from it in various ways. Note that Python uses 0-based indexing, meaning that the first element of a tuple is at position 0, the second element at position 1, and so on.
Can you write Python code for it?

# Assigning a tuple to the variable tup
tup = ('first', 'second', 'third')

tup[0] # Extract the first element.

tup[1] # Extract the second element.

tup[2] # Extract the third element.

tup[3] # Extracting a non-existent element.


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
However now lis also has 9 at the end of it. The line
lis_copy = lis
makes lis_copy point to the same underlying list as lis. What's needed here is a copy of the list. There are many ways of copying a list in Python, one of which follows.
Can you write Python code for it?

lis_copy = lis[:]
lis_copy.pop()
print(lis)
print(lis_copy)


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
List Slicing
It's also possible to slice out chunks of a list.
Can you write Python code for it?

lis[:3] # Extract the first three elements or equivalently
        # extract elements up to (but not including) the fourth element.

lis[3:] # Drop the first three elements and return the rest or equivalently
        # extract elements from the fourth element onwards.

lis[1:4] # Extract elements from the second element up to
         # (but not including the fifth).


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
Mutability
Lists are mutable, so let's mutate (ie. change) them.
Can you write Python code for it?

lis

# Adding an element to the end of a list.
lis.append(6)
lis

# Adding a list to the end of a list.
lis.extend([7,8,9])
lis

# Removing an element from the end of a list.
element = lis.pop()
(element, lis)

# Changing an element in a list.
lis[3] = 42
lis


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
Mutability Gotchas
Can you write Python code for it?

lis

lis_copy = lis
lis_copy.append(9)
lis_copy


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
Notice how the first elements of x and y are "zipped" together into a tuple in the new list, as are the second elements, and the third elements.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
The immutable aspect of tuples will be explained in a bit.
Lists
A Python List is a mutable sequence. Unlike tuples they don't have a fixed size. They are created using square brackets [] with commas to separate the elements.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
Tuples and Mutability
Compare this behaviour to that of tuples.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
enumerate
The enumerate functions generates a list of of pairs (two element tuples) in which the first element is the index/position of the element and the second element is the element in the original list.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/2 - Tuples and Lists.ipynb".
The first task is:
zip
The zip function takes two or more lists and zips them together. This is easier to understand with an example.
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/3 - Dictionaries.ipynb".
The first task is:
Adding and Removing Elements
Dictionaries are mutable so just like lists you can add and remove new mappings.
Can you write Python code for it?

# Adding a mapping.
elements["Au"] = "Gold"
elements

# Removing a mapping.
elements.pop("H")

elements


I am working on the file "Code/Introduction To Python/3 - Dictionaries.ipynb".
The first task is:
Creating Dictionaries and Extracting Elements
The power of dictionary comes from the fact that keys don't have to be numbers. They can be any immutable data structure.
Can you write Python code for it?

elements = {
    "H": "Hydrogen",
    "He": "Helium",
    "Li": "Lithium",
}
elements

elements["H"]

houses = {
    ("Harry", "Potter"): "Gryffindor",
    ("Hermione", "Granger"): "Gryffindor",
    ("Draco", "Malfoy"): "Slytherin",
    ("Cho", "Chang"): "Ravenclaw"
}
houses

houses[("Harry", "Potter")]


I am working on the file "Code/Introduction To Python/3 - Dictionaries.ipynb".
The first task is:
It is possible to check if a key is in a dictionary.
Can you write Python code for it?

("Harry", "Potter") in houses

("Ronald", "Weasley") in houses


I am working on the file "Code/Introduction To Python/4 - Conditionals and Looping.ipynb".
The first task is:
Note how only the first conditional branch that matches gets executed. The else conditions is a catch all condition that would have executed if none of the other branches had been chosen.
Bonus: Loop Comprehensions
You can use the for keyword to generate new lists with data using list comprehensions.
Can you write Python code for it?

r1 = []
for i in range(10):
    r1.append(i**2)

# Equivalent to the loop above.
r2 = [i**2 for i in range(10)]

print(r1)
print(r2)

r1 = []
for i in range(30):
    if (i%2 == 0 and i%3 == 0):
        r1.append(i)

# Equivalent to the loop above.
r2 = [i for i in range(30) if i%2==0 and i%3==0]

print(r1)
print(r2)


I am working on the file "Code/Introduction To Python/4 - Conditionals and Looping.ipynb".
The first task is:
Sometimes you need the index of an element and the element itself while you iterate through a list. This can be achived with the enumerate function.
Can you write Python code for it?

for i, e in enumerate(greek):
    print(e + " is at index: " + str(i))


I am working on the file "Code/Introduction To Python/4 - Conditionals and Looping.ipynb".
The first task is:
The above is taking advantage of tuple decomposition, because enumerate generates a list of tuples. Let's look at this more explicitly.
Can you write Python code for it?

list_of_tuples = [(1,2,3),(4,5,6), (7,8,9)]
for (a, b, c) in list_of_tuples:
    print(a + b + c)


I am working on the file "Code/Introduction To Python/4 - Conditionals and Looping.ipynb".
The first task is:
You can also iterate over tuples
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/4 - Conditionals and Looping.ipynb".
The first task is:
You can have more than two conditions.
Can you write Python code for it?

if x % 4 == 0:
    print("x divisible by 4")
elif x % 3 == 0 and x % 5:
    print("x divisible by 3 and 5")
elif x % 1 == 0:
    print("x divisible by 1")
else:
    print("I give up")


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
Function Gotchas
If you forget to include the return keyword your function will return None.
Can you write Python code for it?

def no_return(x):
    x + x

a_thing = no_return(42)
print(a_thing)


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
Functions can also return functions
Can you write Python code for it?

def print_it_twice(number):
    def x():
        return number + " " + number
    return x

a_fun = print_it_twice("Double")
a_fun


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
Functions can have lots of arguments/parameters
Can you write Python code for it?

def all_the_params(a, b, c, d):
    return a + b + c + d

all_the_params(1, 2, 3, 4)


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
Functions can take pretty much anything as arguments, including other functions.
Can you write Python code for it?

def call_it_twice(fun):
    return fun() + " " + fun()

call_it_twice(toop)


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
If you wanted to change only z but not y you can used a keyword argument.
Can you write Python code for it?

with_defaults(1, z = 4) # Use the default value for y but not z.


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
This is useful for dealing with functions with lots of default values when you only want to change some of the defaults.
Recursions
Functions can also call themselves. This is known as recursion.
Can you write Python code for it?

def deeper():
    return deeper()

deeper()


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
Though unless your function stops calling itself you're going to have issues.
Can you write Python code for it?

def sum_up_to(x):
    print(x)
    if x <= 0: return 0
    else: return x + sum_up_to(x - 1)

sum_up_to(4)


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
or complex
Can you write Python code for it?

def do_math(x):
    return (x * 2.)/(x - 1.)

do_math(2)

do_math(42)


I am working on the file "Code/Introduction To Python/5 - Functions.ipynb".
The first task is:
or no arguments
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/6 - Classes.ipynb".
The first task is:
Calling the tweet method works as follows
Can you write Python code for it?


I am working on the file "Code/Introduction To Python/6 - Classes.ipynb".
The first task is:
Notice how tweet doesn't take an argument even though it is defined as
def tweet(self)
In fact notice how the first parameter of every method in the class starts with self. This is an implicit paramenter that represent the instance of the class itself. Notice how in the tweet method, the custom method is called using self.custom.
Accessing the fields bird_name and time_of_day works much the same way. Accessing them outside of the class works as follows:
Can you write Python code for it?


I am working on the file "Code/KIA_LCA_evelynegroen.ipynb".
The first task is:
Step 2: Determine output variance
Can you write Python code for it?

CV = 0.05                             #Coefficient of variation set to 5% (CV = sigma/mu)
var_A = np.power(abs(CV*A_det),2)     #Variance of the A-matrix (var =sigma^2)
var_B = np.power(abs(CV*B_det),2)     #Variance of the B-matrix

P = np.concatenate((np.reshape(dgdA, 4), dgdB), axis=1)       #P contains partial derivatives of both A and B
var_P = np.concatenate((np.reshape(var_A, 4), var_B), axis=1) #var_P contains all variances of each parameter in A and B

var_g = np.sum(np.multiply(np.power(P, 2), var_P))               #Total output variance (first order Taylor)

print("The total output variance equals:", var_g)


I am working on the file "Code/KIA_LCA_evelynegroen.ipynb".
The first task is:
Step 3: Calculate the contribution to the output variance by the indivudial input parameters
Can you write Python code for it?

KIA = np.multiply(np.power(P, 2), var_P)/var_g
KIA_procent = [KIA[0][0,k]*100 for k in range(0,6)]
print("The contribution to the output variance of each intput parameter equals (in %):")
print(KIA_procent)

#Visualize: make a bar plot
import matplotlib.pyplot as plt
import matplotlib.pyplot as plt

x_label=[ 'A(1,1)','A(1,2)', 'A(2,1)', 'A(2,2)', 'B(1,1)', 'B(1,2)']
x_pos = range(6)
plt.bar(x_pos, KIA_procent, align='center')
plt.xticks(x_pos, x_label)
plt.title('Global sensitivity analysis: squared standardized regression coefficients')
plt.ylabel('KIA (%)')
plt.xlabel('Parameter')
plt.show()


I am working on the file "Code/Lab/Airbnb.ipynb".
The first task is:
Airbnb Data
First we read in the data
Can you write Python code for it?

url1 = "http://data.insideairbnb.com/united-states/"
url2 = "ny/new-york-city/2016-02-02/data/listings.csv.gz"
full_df = pd.read_csv(url1+url2, compression="gzip")

full_df.head()


I am working on the file "Code/Lab/Airbnb.ipynb".
The first task is:
Better reviews also are correlated with higher prices
Can you write Python code for it?

df.plot.scatter(x="review_scores_rating", y="price", figsize=(10, 8), alpha=0.2)


I am working on the file "Code/Lab/Airbnb.ipynb".
The first task is:
We don't want all data, so let's focus on a few variables.
Can you write Python code for it?

df = full_df[["id", "price", "number_of_reviews", "review_scores_rating"]]

df.head()


I am working on the file "Code/Lab/Airbnb_experiments_Chase.ipynb".
The first task is:
Need to convert prices to floats
Can you write Python code for it?

df.replace({'price': {'\$': ''}}, regex=True, inplace=True)
df.replace({'price': {'\,': ''}}, regex=True, inplace=True)
df['price'] = df['price'].astype('float64', copy=False)

df.dtypes


I am working on the file "Code/Lab/Airbnb_experiments_Chase.ipynb".
The first task is:
We don't want all data, so let's focus on a few variables.
Can you write Python code for it?

full_df.dtypes

full_df.columns

df = full_df[["id", "price", "number_of_reviews", "review_scores_rating", "bedrooms", "city", "neighbourhood"]]

df.tail(10)

print(df["id"].nunique())
print(df["id"].shape)

df.dtypes


I am working on the file "Code/Lab/Airbnb_experiments_Chase.ipynb".
The first task is:
We might think that better apartments get rented more often, let's plot a scatter (or multiple boxes?) plot of the number of reviews vs the review score
Can you write Python code for it?

df.plot.scatter(x="number_of_reviews", y="review_scores_rating", figsize=(10, 8), alpha=0.2)

bins = [0, 5, 10, 25, 50, 100, 350]
boxplot_vecs = []

fig, ax = plt.subplots(figsize=(10, 8))

for i in range(1, 7):
    lb = bins[i-1]
    ub = bins[i]
    foo = df["review_scores_rating"][df["number_of_reviews"].apply(lambda x: lb <= x <= ub)].dropna()
    boxplot_vecs.append(foo.values)

ax.boxplot(boxplot_vecs, labels=bins[:-1])
ax.set_xlabel("Number of Reviews")
ax.set_ylabel("Review Score")
plt.show()


I am working on the file "Code/Lab/Detroit_Crime_data_entry.ipynb".
The first task is:
Crime Graphs
Can you write Python code for it?


I am working on the file "Code/Lab/Detroit_Crime_data_entry.ipynb".
The first task is:
Fire Station Locations
Can you write Python code for it?

url2 = 'https://data.detroitmi.gov/resource/hz79-58xh.json'
fire = pd.read_json(url2)
fire = fire.rename(columns={'station':'Station',
                            'full_address_address':'Address',
                            'full_address_zip':'Zip Code'})

fire.insert(1, 'Longitude', 0.0)
fire.insert(2, 'Latitude', 0.0)

for (i, fs) in fire.iterrows():
    # Pull out dictionary
    curr_dict = fs['full_address']
    # Pull out coordinates
    coord = curr_dict['coordinates']

    # Set value just sets
    fire.set_value(i, 'Longitude', coord[0])
    fire.set_value(i, 'Latitude', coord[1])

fire = fire[['Station','Longitude','Latitude','Address','Zip Code']].set_index('Station')

fire.head(2)


I am working on the file "Code/Lab/Detroit_Crime_data_entry.ipynb".
The first task is:
School Locations
Can you write Python code for it?

url3 = 'https://data.detroitmi.gov/resource/8xpr-6ij9.json'
school = pd.read_json(url3)
school = school.rename(columns={'entityoffi':'School',
                                'the_geom':'Location',
                                'entityphys':'Address',
                                'entityph_4':'Zip Code'})

school.insert(1, 'Longitude', 0.0)
school.insert(2, 'Latitude', 0.0)

for (i, s) in school.iterrows():
    # Pull out dictionary
    curr_dict = s['Location']
    # Pull out coordinates
    coord = curr_dict['coordinates']

    # Set value just sets
    school.set_value(i, 'Longitude', coord[0])
    school.set_value(i, 'Latitude', coord[1])

school = school[['School', 'Longitude', 'Latitude', 'Address', 'Zip Code']].set_index('School')
school.head(2)


I am working on the file "Code/Lab/FilesPython.ipynb".
The first task is:
The output of each iteration of the for-loop is a string which contains the entire line of the file.
Writing
We gave two letters for writing access: a and w. The difference between these two lies in where anything we write to the file will be placed. If we use w then anything in the file will be deleted when we write new material to the file. If we use a (for append) then anything written will be placed at the end of the file. We can illustrate this below
Can you write Python code for it?

f = open('filespython.txt', 'w')
f.write('This is another line\n')
f.close()

f = open('filespython.txt', 'r')
print(f.read())
f.close()


I am working on the file "Code/Lab/GetStarted_Python_in_Finance.ipynb".
The first task is:
Arithmetic vs Geometric mean returns
The geometric mean is a good indicator of past performance, especially when we are interested in holding period returns. But if we are interested in expected future returns, the arithmetic mean is the relevant statistic.
Suppose a stock will rise by 30% or fall by 20% with equal probability. If it did one each in consecutive years then the geometric mean return is:
Can you write Python code for it?

g_ret = ((1+0.30)*(1-0.20))**0.5-1
print "Geometric mean return = ", g_ret

a_ret = 0.5*(0.30+(-0.20))
print "Arithmetic mean return per year = ",a_ret


I am working on the file "Code/Lab/GetStarted_Python_in_Finance.ipynb".
The first task is:
BRINGING IT ALL TOGETHER
Can you write Python code for it?

import pandas as pd
import pandas.io.data as web
aapl = web.DataReader('AAPL',data_source='google',start='1/1/2104',end='4/1/2015')
aapl.head()

aapl.tail()

aapl['cont_ret'] = log(aapl['Close']/aapl['Close'].shift(1))
aapl.head()

aapl['Vols'] = pd.rolling_std(aapl['cont_ret'],window=5)*sqrt(252.0)
aapl.tail()

aapl.head(10)

aapl[['Close','Vols']].plot(subplots=True,color='blue',figsize=(8,6))


I am working on the file "Code/Lab/GetStarted_Python_in_Finance.ipynb".
The first task is:
SHARING VARIABLES FROM PYTHON TO R
Can you write Python code for it?

#SENDING DATA VARIABLES TO R
%load_ext rpy2.ipython
#THIS ALLOWS US TO USE R INSIDE THE NOTEBOOK

#SENDS DATA FROM PYTHON TO R
%Rpush stkp

#PREFIX NEEDED TO CALL R INSTEAD OF PYTHON
%R plot(stkp,type="l",col="red",lwd=2)

#GETTING DATA BACK FROM R TO PYTHON
%R ret = diff(log(stkp))
#GET DATA BACK FROM R TO PYTHON
ret = %Rget ret
plot(ret)
%R print(summary(ret))


I am working on the file "Code/Lab/GetStarted_Python_in_Finance.ipynb".
The first task is:
Which one is more realistic in predicting future expected returns over the next two years? Note that there are 4 cases to consider for outcomes, all with equal probability $1/4$.
Can you write Python code for it?

ret = zeros(4)
ret[0] = (1+0.3)*(1+0.3)
ret[1] = (1+0.3)*(1-0.2)
ret[2] = (1-0.2)*(1+0.3)
ret[3] = (1-0.2)*(1-0.2)
two_year_return = 0.25*sum(ret)
print "Expected two year return = ", two_year_return
print "Expected two year return (annualized) = ", two_year_return**0.5


I am working on the file "Code/Lab/MEPS_data_experiment_Brian.ipynb".
The first task is:
Parsing the Data in the .DAT file
Most of the time, when you open a data file in Python or Stata or Matlab or Excel (etc) the data is already pre-configured with variable names and (maybe) descriptions. This isn't the case here. The .DAT file that we unzipped above contains all the relevent data, but as of now its just a bunch of unorganized datapoints that are each a "fixed width" apart. We'll have to tell Python what data corresponds to what variable and write code to label it ourselves.
All the info we need can be found in the codebook. The first two columns in the codebook, "Start:" and "End:", are numbers that tell us where in the .DAT file that particular variable is located.
Note that if the range of a variable in the codebook is (105,107), it needs to be coded as (104, 107). That is, subtract 1 from the first element in the tuple and leave the other unchanged.
Can you write Python code for it?

"""
code format:  tuples are (a, b, c)

a = varable name, b =  column range (start-1,end), x = description
"""

codes = [('DUPERSID',(8,16),'Person identifier'),
         ('TTLP12X',(1501,1507),"Person's total income"),
         ('FAMINC12',(1507,1513),"Family's total income"),
         ('DOBYY',(204,208),"Date of Birth: Year"),
         ('EDUYRDEG',(248,250),'Year of Education or Highest Degree'),
         ('TOTEXP12',(2676,2682),'Total health care expenditure, 2012'),
         ('PERWT12F',(5302,5314),'FINAL PERSON WEIGHT, 2012'),
         ('SEX',(208,209),'Sex'),
         ('POVLEV12',(1514,1525),'Family Income as % of US Poverty Line')]

colnumbs = [x[1] for x in codes]
colnames = [x[0] for x in codes]

df = pd.read_fwf(url_unzipped,
                colspecs = colnumbs ,names = colnames, header = None)


I am working on the file "Code/Lab/MEPS_data_experiment_Brian.ipynb".
The first task is:
The goal is to re-create Figure 1 from here. It paints a pretty vivid picture about the distribution of personal health care spending in the US -- the top 5% of spenders account for nearly 50% of total spending and the top 1% account for more than 20% of the total.
Each observation in the MEPS dataset recieves a weight (PERWT12F) that tells us how many people that observation is supposed to represent. This weighting structure is supposed to make the ~30,000 observations in our sample more representative of the actual ~300 million people in the U.S. The following function (which I took mostly from here but added a bit more to it) allows us to calculate percentiles that take into account the sample weights given by the variable PERWT12F.
Can you write Python code for it?

df['TOTEXP12_WT'] = df['TOTEXP12']*df['PERWT12F']
df['age'] = 2015-df['DOBYY']
percentiles = [i * 0.1 for i in range(10)] + [0.95, 0.99, 1.00]

def weighted_percentiles(subgroup = None,  limits = []):
    import numpy
    a, b  = list(), list()
    if subgroup is None:
        tt = df
    else:
        tt = df[df[subgroup].astype(int).isin(range(limits[0],limits[1]+1))]
    values, sample_weight = tt['TOTEXP12'], tt['PERWT12F']
    for index in percentiles:
        values = numpy.array(values)
        index = numpy.array(index)
        sample_weight = numpy.array(sample_weight)

        sorter = numpy.argsort(values)
        values = values[sorter]
        sample_weight = sample_weight[sorter]

        weighted_percentiles = numpy.cumsum(sample_weight) - 0.5 * sample_weight
        weighted_percentiles /= numpy.sum(sample_weight)
        a.append(numpy.interp(index, weighted_percentiles, values))

    d = [tt.loc[tt['TOTEXP12']<=a[x],'TOTEXP12_WT'].sum()/tt['TOTEXP12_WT'].sum() for x in range(len(percentiles))]
    return d

pd.options.display.float_format = '{:.3f}'.format
age_names = [' <18','18-24','25-44','45-64','65+', 'Total']
income_names = [' < 100%','100%-200%','200%-300%','300%+', 'Total']
age_range = [[0,17],[18,24],[25,45],[45,64],[65,max(df['age'])],[min(df['age'])-1,max(df['age'])+1]]
income_names = [' < 100%','100%-200%','200%-300%','300%+', 'Total']
income_range = [[int(min(df['POVLEV12']))-1,100],[100,200],[200,300],[300,int(max(df['POVLEV12']))+1],
                [int(min(df['POVLEV12']))-1,int(max(df['POVLEV12']))+1]]

df_age = pd.DataFrame({age_names[x] : weighted_percentiles('age',age_range[x]) for x in range(len(age_range))})
df_income = pd.DataFrame({income_names[x] : weighted_percentiles('POVLEV12',income_range[x])
                          for x in range(len(income_range))})

[df_age,df_income] = [df_age.rename(index={x: '{:.2f}'.format(percentiles[x]) for x in range(13)}),
                      df_income.rename(index={x: '{:.2f}'.format(percentiles[x]) for x in...


I am working on the file "Code/Lab/MEPS_data_experiment_Brian.ipynb".
The first task is:
The tables above show that personal health care spending is very highly concentrated in the United States. The top 1% of spenders account for over 20% of total spending and the top 5% accounts for nearly 50%.
It's also clear that the distribution of health care spending is pretty similar within different income groupings. This is not the case for the different age subgroups we consider. The distribution of health care spending is the most equal for elderly people (65+): The top 1% of spenders over 65 account for ~15% of total spending amongst people over 65. The least equal subgroup is children (under 18). Almost 30% of the total health care spending on children comes from the top 1% of spenders.
The graph below makes this more clear. The dotted line in the figure reprents what health care spending would look like if it was perfectly distributed across individuals. That is, if the top 10% of spenders account for 10% of total spending, the top 20% account for 20% of total spending, etc. etc.
The more bowed the actual curve is away from the dotted line, the more unequally distributed is health care spending.
Can you write Python code for it?

%pylab inline
import matplotlib.pyplot as plt
from IPython.html.widgets import *

def figureprefs(values, Group, Title = 'text', Labels = False):
    """
    values = variable(s)
    labels = variable labels (appears in legend)
    plot_text = If True, then the text will appear in the graph like in Figure 1
            from http://www.nihcm.org/pdf/DataBrief3%20Final.pdf.

    """
    fig, ax = plt.subplots(figsize=(10,6))
    [ax.spines[x].set_visible(False) for x in ['right','top']]
    [ax.yaxis.set_ticks_position('left'), ax.xaxis.set_ticks_position('bottom')]
    vals = [i * 0.1 for i in range(11)]
    [ax.set_xticks(vals), ax.set_yticks(vals)]
    ax.set_yticklabels(['{:3.0f}%'.format(x*100) for x in vals])
    ax.set_xticklabels(['{:3.0f}%'.format(x*100) for x in vals])
    ax.set_title(Title,fontsize=14, loc='left')
    ax.set_ylabel('Cumulative Percent of Total Spending')
    ax.set_xlabel('Percent of Civilian Non-Institutionalized Population Ordered by Health Care Spending')

    plt.plot(percentiles,percentiles, linestyle =  '--', color='k',
         label='Perfect Equality')
    plt.plot(percentiles,values[Group], linewidth=1.5, marker='s')
    plt.plot(percentiles,values["Total"], linewidth=1.5, marker='s')

    legend(loc = 2)

    [ax.axvline(x=percentiles[y], ymin=0, ymax=percentiles[y], linestyle = '--',
                color = 'k') for y in [10, 11]]
    [ax.annotate(['95','99'][y-10], xy=(percentiles[y], 0.025),horizontalalignment='right',
                color = 'k', fontsize = 10) for y in [10,11]]
    ax.annotate('{:3.1f}%'.format(values['Total'][12]*100),
                    xy=(percentiles[12]+0.015, values['Total'][12]-0.015),
                    horizontalalignment='left',annotation_clip=False)
    if Labels is True:
        for num in range(7,12):
            ax.annotate('{:3.1f}%'.format(values[Group][num]*100),
                    xy=(percentiles[num]-.01, values[Group][num]),
                    color = 'b', ha = 'right', va = 'center', fontsize = 12)
...


I am working on the file "Code/Lab/Regression_statsmodels_LeBlanc.ipynb".
The first task is:
Calling only select elements of the model (coefficents, standard errors, etc.):
If you type in,
model.

and then hit the 'tab' key, a box will pop up showing all the different items you can call from your regression.
(Remember that 'model' is what we saved our regression as, so it will different depending on what name you give it. i.e. if we saved our regression as 'regression_results' then we would type 'regression_results'.)
Can you write Python code for it?

"""
Below are a few examples of calling speficic elements of our regression
"""
model.params          #Produces all coefficient estimates
model.params['P_CAP']  #Produces coefficient estimate for the regressor 'P_CAP'

model.bse               #Standard Errors for all regressors
model.bse['P_CAP']      #Standard Errors for regressor 'P_CAP'

model.pvalues           #P-values for all regressors
model.pvalues['P_CAP']  #P-values for regressor 'P_CAP'

r_sqr = model.rsquared          #R-squared

print('The R^2 for our regression is',r_sqr)


I am working on the file "Code/Lab/Regression_statsmodels_LeBlanc.ipynb".
The first task is:
Iterating over multiple regression models and storing the results:
Loops in Python are pretty standard if you're familiar with running loop arguments in other programming languages.
Below are two examples of different ways you can loop over multiple regression models.
Can you write Python code for it?

# Method 1:  [action(x) for x in range(a,b)]
fitted_models = [smf.ols(formula='GSP ~ UNEMP + P_CAP + PC',
                         data=df[df['YR']==x]).fit() for x in range(1970,1987)]

# Method 2:  for item in range(a,b):
fitted_models = list()
for item in range(1970,1987):
    fitted_models.append(smf.ols(formula='GSP ~ UNEMP + P_CAP + PC',
                         data=df[df['YR']==item]).fit())

#We can then call/store the coefficients/standard errors/etc in the same way:
P_CAPcoefs = [fitted_models[x].params['P_CAP'] for x in range(17)]
P_CAPpval = [fitted_models[x].pvalues['P_CAP'] for x in range(17)]
P_CAPse = [fitted_models[x].bse['P_CAP'] for x in range(17)]

df_PCAPResults = pd.DataFrame({'P_Cap':P_CAPcoefs,'Std. Errors':P_CAPse}, index = list(range(1970,1987)))

df_PCAPResults


I am working on the file "Code/Lab/SCF_data_experiment_Brian.ipynb".
The first task is:
Distribution of Income in the United States
How equal, or unequal, is the distribution of income across households in the US? On way to visualize this is by plotting what's known as a Lorenz curve.
The Lorenz curve plots the proportion of overall income earned (y-axis) against the bottom x% of earners (x-axis). If total income in the US was uniformly distributed across all households, than we would expect the bottom 50% of earners to account for 50% of the total income. Likewise, the bottom 80% of earners should account for 80% of total income, etc. etc. The Lorenz curve in this "perfect equality" case would just be a straight line going from (x,y) = (0%,0%) to (x,y) = (100%,100%).
In reality, income in most countries is far from being uniformly distributed. As we will see, the bottom 50% of earners in the U.S. accounted for XX% of total income during 2013 while the bottom 80% accounted for just XX%. This means the Lorenz curve for the U.S. won't be linear, but instead "bowed" towards the bottom right-hand corner. The heralded Gini coefficient is actually estimated by measuring the area between a country's actual Lorenz curve and the linear Lorenz curve we'd see in the "perfect equality" case.
Note: It would make our lives a lot easier if the income percentile groupings of households in our sample accurately reflected the actual income percentile groupings in the US. This isn't the case, unfortunately, and we'll have to weight the observations in order to get a more accurate picture of the income distribution.
The function below produces accurate, weighted percentiles for income, net worth, or whatever other variable in the SCF dataset we're interest in. About 99% of code was taken from here, but I modified it a bit to suit our needs better.
Can you write Python code for it?

def weighted_percentiles(data, variable, weights, percentiles = [],
                         dollar_amt = False, subgroup = None, limits = []):
    """
    data               specifies what dataframe we're working with

    variable           specifies the variable name (e.g. income, networth, etc.) in the dataframe

    percentiles = []   indicates what percentile(s) to return (e.g. 90th percentile = .90)

    weights            corresponds to the weighting variable in the dataframe

    dollar_amt = False returns the percentage of total income earned by that percentile
                       group (i.e. bottom 80% of earners earned XX% of total)

    dollar_amt = True  returns the $ amount earned by that percentile (i.e. 90th percentile
                       earned $X)

    subgroup = ''      isolates the analysis to a particular subgroup in the dataset. For example
                       subgroup = 'age' would return the income distribution of the age group
                       determined by the limits argument

    limits = []        Corresponds to the subgroup argument. For example, if you were interesting in
                       looking at the distribution of income across heads of household aged 18-24,
                       then you would input "subgroup = 'age', limits = [18,24]"

    """
    import numpy
    a  = list()
    data[variable+weights] = data[variable]*data[weights]
    if subgroup is None:
        tt = data
    else:
        tt = data[data[subgroup].astype(int).isin(range(limits[0],limits[1]+1))]
    values, sample_weight = tt[variable], tt[weights]

    for index in percentiles:
        values = numpy.array(values)
        index = numpy.array(index)
        sample_weight = numpy.array(sample_weight)

        sorter = numpy.argsort(values)
        values = values[sorter]
        sample_weight = sample_weight[sorter]

        weighted_percentiles =...


I am working on the file "Code/Lab/SCF_data_experiment_Brian.ipynb".
The first task is:
The figure above is pretty revealing, but the picture might be even more stark for the period 1992-2001, which corresponds roughly to President Clinton's two terms in office (or as close as we can get with the SCF, which is conducted every three years).
This isn't to say that the other decades aren't interesting, but this timeframe seems to be one with the most rapid growth in top incomes.
Real income growth exploded for top earners in the 1990s. The top 1% of earners saw their income grow by 8.35% per year on average in real terms between 1992 and 2001. The top 0.1% faired even better, with average real income growth of 18.70%.
That being said, just about every income percentile saw better-than-average growth between 1992 and 2001. Median income (50th percentile) grew 2.15 per year on average in real terms, which is much better than the 0.046% per year average real income growth the same percentile group experienced over the period 1983-2013.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(10,6))
plt.plot(percentiles,(incomes['2001']-incomes['1992'])/incomes['1992']/(2001-1992+1),
         linewidth = 2.0, label = '1992-2001');
plt.plot(percentiles,(incomes['2013']-incomes['1983'])/incomes['1983']/(2013-1983+1),
         linewidth = 2.0, label = '1983-2013');
yvals = ax.get_yticks()
ax.set_xticks([i * 0.1 for i in range(11)])
xvals = ax.get_xticks()
ax.set_yticklabels(['{:3.2f}%'.format(x*100) for x in yvals]);
ax.set_xticklabels(['{:3.0f}'.format(x*100) for x in xvals]);
ax.set_title('Annual real income growth by income percentile',  #Axes titles
                  fontsize=18, loc='center');
ax.axhline(y=0,xmin = 0, xmax = 1, linestyle = '--', color = 'k');
ax.set_ylabel('Average annual growth rate of real income');
ax.set_xlabel('Income percentile');
legend(loc=2);


I am working on the file "Code/Lab/UN_demography.ipynb".
The first task is:
Comment. At this point, we need to pivot the data.  That's not something we've done before, so take it as simply something we can do easily if we have to.  We're going to do this twice to produce different graphs:

Compare countries for the same period.
Compare different periods for the same country.
Can you write Python code for it?

# compare countries for most recent period
countries = ['China', 'Japan', 'Germany', 'United States of America']
mt = m[m['Country'].isin(countries) & m['Interval'].isin([5]) & m['Period'].isin(['2010-2015'])]
print('Dimensions:', mt.shape)

mp = mt.pivot(index='Age', columns='Country', values='Mortality')
mp.head(3)

fig, ax = plt.subplots()
mp.plot(ax=ax, kind='line', alpha=0.5, linewidth=3,
#        logy=True,
        figsize=(6, 4))
ax.set_title('Mortality by age', fontsize=14, loc='left')
ax.set_ylabel('Mortality Rate (log scale)')
ax.legend(loc='best', fontsize=10, handlelength=2, labelspacing=0.15)


I am working on the file "Code/Lab/UN_demography.ipynb".
The first task is:
Exercise. What do you see here?  What else would you like to know?
Exercise. Adapt the preceeding code to do the same thing for China.  Or some other country that sparks your interest.
Fertility:  aka birth rates
We might wonder, why is the population falling in Japan?  Other countries?  Well, one reason is that birth rates are falling.  Demographers call this fertility.  Here we look at the fertility using the same UN source as the previous example.  We look at two variables:  total fertility and fertility by age of mother.  In both cases we explore the numbers to date, but the same files contain projections of future fertility.
Can you write Python code for it?

# fertility overall
uft  = 'http://esa.un.org/unpd/wpp/DVD/Files/'
uft += '1_Indicators%20(Standard)/EXCEL_FILES/'
uft += '2_Fertility/WPP2015_FERT_F04_TOTAL_FERTILITY.XLS'

cols = [2] + list(range(5,18))
ftot = pd.read_excel(uft, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])

ftot.head(3)[list(range(6))]

# rename some variables
names = list(ftot)
f = ftot.rename(columns={names[0]: 'Country'})

# select countries
countries = ['China', 'Japan', 'Germany', 'United States of America']
f = f[f['Country'].isin(countries)]

# shape
f = f.set_index('Country').T
f = f.rename(columns={'United States of America': 'United States'})
f.tail(3)

fig, ax = plt.subplots()
f.plot(ax=ax, kind='line', alpha=0.5, lw=3, figsize=(6.5, 4))
ax.set_title('Fertility (births per woman, lifetime)', fontsize=14, loc='left')
ax.legend(loc='best', fontsize=10, handlelength=2, labelspacing=0.15)
ax.set_ylim(ymin=0)
ax.hlines(2.1, -1, 13, linestyles='dashed')
ax.text(8.5, 2.4, 'Replacement = 2.1')


I am working on the file "Code/Lab/UN_demography.ipynb".
The first task is:
Exercise. What do you see here?  What else would you like to know?
Exercise. Add Canada to the figure.  How does it compare to the others?  What other countries would you be interested in?
Life expectancy
One of the bottom line summary numbers for mortality is life expectancy:  if mortaility rates fall, people live longer, on average.  Here we look at life expectancy at birth.  There are also numbers for life expectancy given than you live to some specific age; for example, life expectancy given that you survive to age 60.
Can you write Python code for it?

# life expectancy at birth, both sexes
ule  = 'http://esa.un.org/unpd/wpp/DVD/Files/1_Indicators%20(Standard)/EXCEL_FILES/3_Mortality/'
ule += 'WPP2015_MORT_F07_1_LIFE_EXPECTANCY_0_BOTH_SEXES.XLS'

cols = [2] + list(range(5,34))
le  = pd.read_excel(ule, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])

le.head(3)[list(range(10))]

# rename some variables
oldname = list(le)[0]
l = le.rename(columns={oldname: 'Country'})
l.head(3)[list(range(8))]

# select countries
countries = ['China', 'Japan', 'Germany', 'United States of America']
l = l[l['Country'].isin(countries)]

# shape
l = l.set_index('Country').T
l = l.rename(columns={'United States of America': 'United States'})
l.tail()

fig, ax = plt.subplots()
l.plot(ax=ax, kind='line', alpha=0.5, lw=3, figsize=(6, 8), grid=True)
ax.set_title('Life expectancy at birth', fontsize=14, loc='left')
ax.set_ylabel('Life expectancy in years')
ax.legend(loc='best', fontsize=10, handlelength=2, labelspacing=0.15)
ax.set_ylim(ymin=0)


I am working on the file "Code/Lab/UN_demography.ipynb".
The first task is:
Exercise. What other countries would you like to see?  Can you add them?  The code below generates a list.
Can you write Python code for it?

countries = le.rename(columns={oldname: 'Country'})['Country']


I am working on the file "Code/Lab/UN_demography.ipynb".
The first task is:
Exercises.

What country's old people have the lowest mortality?
What do you see here for the US?  Why is our life expectancy shorter?
What other countries would you like to see?  Can you adapt the code to show them?
Anything else cross your mind?
Can you write Python code for it?

# compare periods for the one country -- countries[0] is China
mt = m[m['Country'].isin([countries[0]]) & m['Interval'].isin([5])]
print('Dimensions:', mt.shape)

mp = mt.pivot(index='Age', columns='Period', values='Mortality')
mp = mp[[0, 6, 12]]
mp.head(3)

fig, ax = plt.subplots()
mp.plot(ax=ax, kind='line', alpha=0.5, linewidth=3,
#        logy=True,
        figsize=(6, 4))
ax.set_title('Mortality over time', fontsize=14, loc='left')
ax.set_ylabel('Mortality Rate (log scale)')
ax.legend(loc='best', fontsize=10, handlelength=2, labelspacing=0.15)


I am working on the file "Code/Lab/UN_demography.ipynb".
The first task is:
Population by age
We have both "estimates" of the past (1950-2015) and "projections" of the future (out to 2100).  Here we focus on the latter, specifically what the UN refers to as the medium variant:  their middle of the road projection.  It gives us a sense of how Japan's population might change over the next century.
It takes a few seconds to read the data.
What are the numbers?  Thousands of people in various 5-year age categories.
Can you write Python code for it?

url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2015_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLS'
url = url1 + url2 + url3

cols = [2, 5] + list(range(6,28))
#est = pd.read_excel(url, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])

prj.head(3)[list(range(6))]

# rename some variables
pop = prj
names = list(pop)
pop = pop.rename(columns={names[0]: 'Country',
                          names[1]: 'Year'})
# select country and years
country = ['Japan']
years     = [2015, 2055, 2095]
pop = pop[pop['Country'].isin(country) & pop['Year'].isin(years)]
pop = pop.drop(['Country'], axis=1)

# set index = Year
# divide by 1000 to convert numbers from thousands to millions
pop = pop.set_index('Year')/1000

pop.head()[list(range(8))]

# transpose (T) so that index = age
pop = pop.T
pop.head(3)

ax = pop.plot(kind='bar',
              color='blue',
              alpha=0.5, subplots=True, sharey=True, figsize=(8,6))

for axnum in range(len(ax)):
    ax[axnum].set_title('')
    ax[axnum].set_ylabel('Millions')

ax[0].set_title('Population by age', fontsize=14, loc='left')


I am working on the file "Code/Numerical Computing with Numpy/1 - Introduction to Numpy.ipynb".
The first task is:
Both of the functions perform the same operation, one using a Python for loop and the other taking advantage of Numpy arrays.
Can you write Python code for it?

%timeit normal_multiply(x, y)

%timeit numpy_multiply(xNumpy, yNumpy)


I am working on the file "Code/Numerical Computing with Numpy/1 - Introduction to Numpy.ipynb".
The first task is:
Introduction to NumPy
Numpy is a library that provides multi-dimensional array objects. You can think of these somewhat like normal Python lists, except they have a number of qualities that make them better for numeric computations.
Let's try adding two lists together
Can you write Python code for it?


I am working on the file "Code/Numerical Computing with Numpy/1 - Introduction to Numpy.ipynb".
The first task is:
The  + operator applied to Numpy arrays performs elementwise addition. -, * and / also apply elementwise. Using these operators makes it a lot easier to understand what's happening in the code.
The other advantaged of Numpy arrays has to do with performance. Let's perform elementwise multiplication of the first 1 million numbers divided by 3 and the first 1 million numbers divided by 7, that is:
[1/3, 2/3, ..., 999999/3, 1000000/3] *
[1/7, 2/7, ..., 999999/7, 1000000/7]
Can you write Python code for it?

def normal_multiply(x, y):
    return [i * j for i, j in zip(x, y)]

def numpy_multiply(x, y):
    return x * y

x = [i/3. for i in range(1,1000001)]
y = [i/7. for i in range(1,1000001)]
xNumpy = np.array(x)
yNumpy = np.array(y)


I am working on the file "Code/Numerical Computing with Numpy/1 - Introduction to Numpy.ipynb".
The first task is:
With Numpy arrays this isn't the case
Can you write Python code for it?

xNumpy = np.array([1, 2, 3])
yNumpy = np.array([4, 5, 6])
xNumpy + yNumpy


I am working on the file "Code/Numerical Computing with Numpy/1 - Introduction to Numpy.ipynb".
The first task is:
With Python lists the + operator appends them together. If we wanted to add these two lists elementwise we'd have to use a loop
Can you write Python code for it?

z = [0]*len(x) # Generates a list of zeroes the same length as x.
for i in range(len(x)):
    z[i] = x[i] + y[i]
z


I am working on the file "Code/Numerical Computing with Numpy/1 - Introduction to Numpy.ipynb".
The first task is:
or a for comprehension
Can you write Python code for it?


I am working on the file "Code/Phone Position Classification Exercise.ipynb".
The first task is:
88% accuracy on the training set and 60% accuracy on the test set means we're overfitting the data.  Accelerometer data should be the key here, since a phone on the driver side will experience a different centripetal acceleration than a phone on the passenger side during turns.  But accelerometer data is super noisy!
Can you write Python code for it?

# Inspect feature importances
for i, ifeature in enumerate(features):
    print(ifeature + ': %6.4f' % clf.feature_importances_[i])


I am working on the file "Code/Phone Position Classification Exercise.ipynb".
The first task is:
Load the processed sensor data for car trip (see "Process Smartphone Sensor Data" jupyter notebook).  On this trip, I drove my car from home to Censio and back and used SensorLog on my iPhone to track the trip.  The total time for the trip was about 15 minutes.
Can you write Python code for it?

df = pd.read_csv('../Data/shaneiphone_exp2_processed.csv', index_col='DateTime')

# Use only userAcceleration and gyroscope data, since these features are expected to generalize well.
xyz = ['X', 'Y', 'Z']
measures = ['userAcceleration', 'gyroscope']
basefeatures = [i + j for i in measures for j in xyz]
features = [i + j for i in measures for j in xyz]

# Add Gaussian smoothed features
smoothfeatures = []
for i in features:
    df[i + 'sm'] = gaussian_filter(df[i], 3)
    df[i + '2sm'] = gaussian_filter(df[i], 100)
    smoothfeatures.append(i + 'sm')
    smoothfeatures.append(i + '2sm')
features.extend(smoothfeatures)

# Generate Jerk signal
jerkfeatures = []
for i in features:
    diffsignal = np.diff(df[i])
    df[i + 'jerk'] = np.append(0, diffsignal)
    jerkfeatures.append(i + 'jerk')
features.extend(jerkfeatures)

# assign class labels
class0 = (df.index > '2015-08-25 14:35:00') & \
        (df.index < '2015-08-25 14:42:00')

class1 = (df.index > '2015-08-25 14:43:00') & \
        (df.index < '2015-08-25 14:48:00')

df['class'] = -1
df['class'][class0] = 0
df['class'][class1] = 1

# separate into quarters for train and validation
q1 = df[(df.index <= '2015-08-25 14:38:30') &
        (df.index > '2015-08-25 14:33:00')]
q2 = df[(df.index > '2015-08-25 14:38:30') &
        (df.index <= '2015-08-25 14:42:00')]
q3 = df[(df.index > '2015-08-25 14:43:00') &
        (df.index <= '2015-08-25 14:45:30')]
q4 = df[(df.index > '2015-08-25 14:45:30') &
        (df.index <= '2015-08-25 14:48:00')]
traindf = pd.concat([q1, q3])
validationdf = pd.concat([q2, q4])

# check for NaNs in the dataframes
print(traindf.isnull().sum().sum())
print(validationdf.isnull().sum().sum())

# drop NaNs
traindf = traindf.dropna()
validationdf = validationdf.dropna()

# Make the training and validation sets
X_train = traindf[features].values
y_train = traindf['class'].values
X_test = validationdf[features].values
y_test = validationdf['class'].values

# train a random forest
clf = RandomForestClassifier(n_estimators=200)

# get the...


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
A whole bunch of stuff just happened. Let's take a minute to go through it:
* np.pi: The numpy library has a value called pi, which conveniently stores the value of pi.
* np.linspace: This function generates a list of 10 points between -pi and pi.
* np.sin: sin function provided by the numpy library.
An important thing to wrap your head around is that plotting in Python is all about points. To plot a function you evaluate the function at many points and then plot those points. The plot above isn't particularly smooth. This can be improved by sampling more points.
Can you write Python code for it?

x = np.linspace(-np.pi, np.pi, 100, endpoint=True) # Switch from 10 to 100 points.
y = np.sin(x)

plt.plot(x, y)
plt.xlabel("x")
plt.ylabel("sin(x)")
plt.title("Sine Function")
plt.grid()

plt.show()
plt.clf()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
And that's the general gist of generating plots.
Semilog and Log-Log Plots
Sometimes you want to have either the x-axis, y-axis or both as a log scale.
Can you write Python code for it?

x = np.linspace(-100, 10, 100, True)

plt.plot(x, 10**x)
plt.xlabel("x")
plt.ylabel("y")
plt.title("Power Function")

plt.grid()
plt.show()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
If you wanted the log scale on the x-axis you would use the semilogx function.
For log-log plots use the loglog function.
Can you write Python code for it?

x = np.linspace(1, 10000, 50, True)

plt.loglog(x, x) # Log-log Plot
plt.xlabel("x")
plt.ylabel("y")
plt.title("Log-Log Plot")

plt.grid()
plt.show()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
Plotting Datapoints
Let's start by plotting four points:
* (1, 4)
* (2, 3)
* (3, 6)
* (5, 2)
Can you write Python code for it?

x = [1, 2, 3, 5]    # List of x coordinates.
y = [4, 3, 6, 2]    # List of y coordinates.
y_error = [0.1, 0.5, .2, 0.25] # Errors associated with the y readings.

plt.scatter(x, y)
plt.show()       # Displays the plot.
plt.clf()        # Clears the plot.


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
Saving Plots to a File
To including plots in reports, it's useful to be able to save them.
Can you write Python code for it?

x = np.linspace(0, 10, 50)

plt.xlabel("x")
plt.ylabel("y")
plt.title("A Line")
plt.grid()
plt.plot(x, x)

plt.savefig("Line.pdf") # Save the plot.
plt.clf()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
That looks much smoother. Now let's plot multiple figures in a single plot.
Can you write Python code for it?

x = np.linspace(-np.pi, np.pi, 100, True)

plt.plot(x, np.sin(x))
plt.plot(x, np.cos(x))
plt.xlabel("x")
plt.ylabel("y")
plt.title("Sine and Cosine Function")
plt.grid()

plt.show()
plt.clf()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
This is as simple as making two calls to plot. However with two different things on the plot it's a good idea to label them.
Can you write Python code for it?

x = np.linspace(-np.pi, np.pi, 100, True)

plt.plot(x, np.sin(x), label="Sin(x)") # label associates a name with the plot.
plt.plot(x, np.cos(x), label="Cos(x)")
plt.xlabel("x")
plt.ylabel("y")
plt.title("Sine and Cosine Function")
plt.legend(loc='upper left')    # Display the legend in the upper right.
plt.grid()
plt.show()
plt.clf()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
This plot looks pretty good. Too good. Let's add in error bars using the errorbar function.
Can you write Python code for it?

plt.plot(x, y, marker="x") # x is for x
plt.errorbar(x, y, yerr=y_error) # Adding in error bars.
plt.xlabel("x")
plt.ylabel("y")
plt.title("Realistic Plot")
plt.grid()

plt.show()
plt.clf()


I am working on the file "Code/Plotting/1 - Plotting with Matplotlib.ipynb".
The first task is:
Three functions were called during the generation of this plot:
* scatter: takes a list of x-coordinates and a list of y-coordinates and generates a scatterplot.
* show: makes Python display the generated plot.
* clf: clears the generated figure.
This plot is missing a couple of important features though:
* Axes labels.
* A title.
* A background grid.
Let's fix that.
Can you write Python code for it?

plt.scatter(x, y)
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.title("Basic Scatterplot")
plt.grid()

plt.show()
plt.clf()


I am working on the file "Code/Plotting/2 - Plotting Matrices.ipynb".
The first task is:
It's possible to smooth the plot by utilizing interpolation. This isn't something that I would recommend though as it hides the structure of your data. Note however that this USED to be the default behaviour of the implot function in earlier versions of Matplotlib.
Increasing the sampling density is a better way of generating a nicer plot.
Can you write Python code for it?

mat = create_matrix(60)
plt.imshow(mat)
plt.colorbar()
plt.xlabel("x")
plt.ylabel("y")
plt.title("Denser Matrix Plot")


I am working on the file "Code/Plotting/2 - Plotting Matrices.ipynb".
The first task is:
The color scale used to represent the data can also be modified using the cmap keyword argument.
Can you write Python code for it?

import matplotlib.cm as cm

plt.imshow(mat, cmap=cm.Reds)
plt.colorbar()
plt.xlabel("x")
plt.ylabel("y")
plt.title("Switching Color Scale ")


I am working on the file "Code/Plotting/2 - Plotting Matrices.ipynb".
The first task is:
There are lots of different colormaps to choose from.
Can you write Python code for it?

plt.imshow(mat, cmap=cm.winter_r)
plt.colorbar()
plt.xlabel("x")
plt.ylabel("y")
plt.title("Switching Color Scale ")