I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Exercise. Construct dataframes for which

small['Units'] does not equal 'National currency'.
small['Units'] equals 'National currency' and small['2011'] is greater than 100.

The isin method
Pay attention now, this is really useful.  Suppose we want to extract the data for which weo['Country'] == 'Argentina' or weo['Country'] == 'Greece' (Greece).  We could do that by combining the comparisons:
python
(weo['Country'] == 'Aregentina') | (weo['Country'] == 'Greece')
Remind youself that | stands for "or."  (What do we use for "and"?)
A simpler approach is to apply the isin method to a variable.  This sets the comparison equal to True if the value of the observation is of weo['Country'] equals any element in a list.  We could do the same thing using mulitple comparisons, but this is a lot easier.
Let's see how this works.
Example.  Let's apply the same logic to variable codes.  If we want to extract the observations with codes
vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
we would use
Can you write Python code for it?

vlist = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
weo['WEO Subject Code'].isin(vlist)


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Find variable and country codes. Which ones do we want?  Let's start by seeing that's available.  Here we create special dataframes that include all the variables and their definitions and all the countries.
Note the use of the drop_duplicates method, which does what it sounds like: remove duplicate rows (!)
Can you write Python code for it?

variable_list = weo[['Country', 'Subject Descriptor', 'Units']].drop_duplicates()
print('Number of variables: ', variable_list.shape[0])
variable_list.head()

country_list = weo['Country'].drop_duplicates()
print('Number of countries: ', country_list.shape[0])
country_list


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Missing values
Can you write Python code for it?

weo = pd.read_csv(url,
                  sep = '\t',
                  index_col='ISO',
                  usecols=details_list + years_list,
                  skipfooter=1, engine='python',
                  na_values='n/a')
weo.head()

weo.dtypes[:10]     # still not ok


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Notice the , for thousands. As we saw before, there is an easy fix
Can you write Python code for it?

weo = pd.read_csv(url,
                  sep = '\t',
                  index_col='ISO',
                  usecols=details_list + years_list,
                  skipfooter=1, engine='python',
                  na_values='n/a',
                  thousands =',')
weo.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Pandas string methods. We can do the same thing to all the observations of a variable with so-called string methods.  We append .str to a variable in a dataframe and then apply the string method of our choice.  If this is part of converting a number-like entry that has mistakenly been given dtype object, we then convert its dtype with the astype method.
Example.  Let's use a string method to fix the item_price variable in the Chipotle dataframe.  This has three parts:

Use the method str to identify this as a string method.
Apply the string method of our choice (here replace) to fix the string.
Use the astype method to convert the fixed-up string to a float.

We start by making a copy of the chp dataframe that we can experiment with.
Can you write Python code for it?

chipotle.head()

# create a copy of the df to play with

chipotle_num = chipotle.copy()
print('Original dtype:', chipotle_num['item_price'].dtype)

# delete dollar signs (dtype does not change!)

chipotle_num['item_price'].str.replace('$', '').head()

# delete dollar signs, convert to float, AND assign back to chipotle_num in one line

chipotle_num['item_price'] = chipotle_num['item_price'].str.replace('$', '').astype(float)
print('New dtype:', chipotle_num['item_price'].dtype)

# assign back to chp for future use
chipotle = chipotle_num

print('Variable dtypes:\n', chipotle.dtypes, sep='')
chipotle.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Review
Let's remind ourselves what we've learned.
Exercise. We explore the Census's Business Dynamics Statistics, a huge collection of data about firms. We've extracted a small piece of one of their databases that includes these variables for 2013:

Size: size category of firms based on number of employees
Firms: number of firms in each size category
Emp: number of employees in each size category

Run the code cell below to load the data.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Second: The second one is harder.  The question about special topics of interest says "mark all that apply."  In the spreadsheet, we have a list of every choice the person checked.  Our want is to count the number of each type of response.  For example, we might want a bar chart that gives us the number of each response. The question is how we get there.
Can you write Python code for it?

# check multi-response question to see what we're dealing with
entry_poll['Topics'].head(20)


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Series indexing
Indexing a Series is a little different because we only have one column, so all indexing operations interact with rows.
The rules here are a little subtle, so we'll show examples and add comments that explain what each example does
In the list below s is a Series, n is an integer, nlist = [0, 3] is a list of integers, and i is a string, and is is a list of strings

s[n]: if the index has dtype int, this extracts the row with index n. Otherwise extracts the nth row (starting at zero)
s[nlist]: if the index has dtype int, this extracts rows with indices in nlist returning NaN if they don't appear. Otherwise extracts the rows at positions in nlist, filling with NaN for invalid positions
s[i]: if the index has dtype object, this extracts the row with index i, otherwise it is an error
s[is]:
Can you write Python code for it?

s1 = pd.Series([5, 6, 7, 8], index=["a", "b", "c", "d"])
s1

s2 = pd.Series([50, 60, 70, 80], index=[0, 4, 2, 999])
s2

# index has dtype object, so using an int returns the value in that row (starting at 0)
s1[1]

# index has dtype int, so using an integer tries to find the that int in the
# index and return the corresponding value and throws an error if it can't find it
s2[1]

s2[0]  # no error, 0 is in the index

# index has dtype object, so a list of ints extracts those rows
s1[[0, 3]]

# index has dtype int, so a list of ints tries to match each int to the index
# it returns NaN where it can't find the index. Notice it **did not** return
# `80` for 3
s2[[0, 3, 999]]

# index has type object, so a string finds row with matching index
s1["c"]

# index has dtype int, so using a string causes an error
s2["c"]

# similar behavior for lists of strings
s1[["a", "b", "penguin"]]

# index has dtype int, so list of strings returns NaN's everywhere
s2[["a", "b"]]

# lists of True/False work the same for any dtype of index
bools = [True, False, False, True]
s1[bools]

s2[bools]


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
The first three lines are empty. Skip those
Can you write Python code for it?

oecd = pd.read_excel(url1 + url2, sheetname='Physicians', skiprows=3)
oecd.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
The last two columns contain junk
Can you write Python code for it?

oecd.shape   # drop 57th and 58th columns

# There is no skipcols argument, so let's google "read_excel skip columns" -> usecols
oecd = pd.read_excel(url1 + url2,
                     sheetname='Physicians',
                     skiprows=3,
                     index_col=0,
                     usecols=range(57))
oecd.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
This error indicates that somewhere in weo['1980'] there is a string value --. We want to convert that into NaN. Later we will see how we can do that directly. For now use read_csv() again
Can you write Python code for it?

weo = pd.read_csv(url,
                  sep = '\t',
                  index_col='ISO',
                  usecols=details_list + years_list,
                  skipfooter=1, engine='python',
                  na_values=['n/a', '--'],
                  thousands =',')
weo.head(1)

# With that out of our way, we can do the conversion for one variable
weo['1980'].astype(float)

# or for all numeric variables
years = [str(year) for year in range(1980, 2017)]
weo[years] = weo[years].astype(float)

weo.dtypes


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Useful columns:
   - 1, 2, 3, 4, 6 (indices)
   - years, say from 1980 to 2016
Need a list that specifies these
Can you write Python code for it?

names = list(weo.columns)
names[:8]

# for var details
details_list = names[1:5] + [names[6]]
# for years
years_list = names[9:-6]

details_list

weo = pd.read_csv(url,
                  sep = '\t',
                  index_col='ISO',
                  usecols=details_list + years_list)
weo.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Want to convert the year variables into float
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
We still have a couple issues.

The index includes a space and a number:  Australia 1, Chile 3, etc.  We care about this because when we plot the data across countries, the country labels are going to be country names, so we want them in a better form than this.
The ..'s in the sheet lead us to label any column that includes them as dtype object.  Here we want to label them as missing values.
If we want to plot each country against time, then we'll need to switch the rows and columns somehow, so that the x axis in the plot (the year) is the index and not the column label.

Example:  World Economic Outlook
The IMF's World Economic Outlook database contains a broad range of macroeconomic data for a large number of countries.  It's updated twice a year and is a go-to source for things like current account balances (roughly, the trade balance) and government debt and deficits.  It also has a few quirks, as we'll see.
Example. Run the following code as is, and with the thousands and na_values parameters commented out.  How do the dtypes differ?
Can you write Python code for it?

url = 'http://www.imf.org/external/pubs/ft/weo/2016/02/weodata/WEOOct2016all.xls'

# Try
weo = pd.read_excel(url)         # NOT an excel file!

# try to open the file with a plain text editor (it is a TSV)
weo = pd.read_csv(url, sep = '\t')
weo.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
What about the bottom of the table?
Can you write Python code for it?

oecd.tail()    # we are downloading the footnotes too

?pd.read_excel  # -> skip_footer

# How many rows to skip??
oecd.tail(25)

oecd = pd.read_excel(url1 + url2,
                     sheetname='Physicians',
                     skiprows=3,
                     index_col=0,
                     usecols=range(57),
                     skip_footer=20)
oecd.tail()

oecd.dtypes[:5]


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
What to do.  We use the replace method on the whole dataframe.  To mark something as missing, we replace it as None, which Pandas interprets as missing and labels NaN.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
Working with missing values
Can you write Python code for it?

# grab a variable to play with
var = docsna[2013].head(10)
var

# why not '2013'? check the type
docsna.columns

# which ones are missing ("null")?
var.isnull()

# which ones are not missing ("not null")?
var.notnull()

# drop the missing
var.dropna()


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
and a pie chart
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv1-clean.ipynb".
The first task is:
str.title() returns a copy of the string in which first characters of all the words are capitalized.
Can you write Python code for it?

newnames

# (3) assign newnames to the variables
entry_poll.columns = newnames
entry_poll.head(1)


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:

Review
We return to the OECD's healthcare data, specifically a subset of their table on the number of doctors per one thousand population.  This loads and cleans the data:
Can you write Python code for it?

url1 = 'http://www.oecd.org/health/health-systems/'
url2 = 'OECD-Health-Statistics-2017-Frequently-Requested-Data.xls'
docs = pd.read_excel(url1+url2,
                     skiprows=3,
                     usecols=[0, 51, 52, 53, 54, 55, 57],
                     sheetname='Physicians',
                     na_values=['..'],
                     skip_footer=21)
# rename country variable
names = list(docs)
docs = docs.rename(columns={names[0]: 'Country'})
# strip footnote numbers from country names
docs['Country'] = docs['Country'].str.rsplit(n=1).str.get(0)
docs = docs.head()
docs


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Comment. Note that all the parameters here are columns.  That's not a choice, it's the way the the pivot method is written.
We do a plot for fun:
Can you write Python code for it?

# plot 2013 as bar chart
fig, ax = plt.subplots()
bdsp[2013].plot(ax=ax, kind='barh')
ax.set_ylabel('')
ax.set_xlabel('Number of Employees (millions)')


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Comment. We see here that the multi-index for the rows has been turned into a multi-index for the columns.  Works the same way.
The only problem here is that the column labels are more complicated than we might want.  Here, for example, is what we get with the plot method.  As usual, .plot() plots all the columns of the dataframe, but here that means we're mixing variables.  And the legend contains all the levels of the column labels.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Exercise.  For the dataframe ddi do the following in separate code cells:

Use the reset_index method to move the Units level of the index to a column of the dataframe.
Use the drop parameter of reset_index to delete Units from the dataframe.

Switching rows and columns
If we take the dataframe ddi, we see that the everything's been put into the index but the data itself.  Perhaps we can get what we want if we just flip the rows and columns.  Roughly speaking, we refer to this as pivoting.
First look at switching rows and columns
The simplest way to flip rows and columns is to use the T or transpose property.  When we do that, we end up with a lot of stuff in the column labels, as the multi-index for the rows gets rotated into the columns.  Other than that, we're good. We can even do a plot.  The only problem is all the stuff we've pushed into the column labels -- it's kind of a mess.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Exercise. Set Variable as the index.
Comment. Note that the new index brought its name along:  Country in the two examples, Variable in the exercise.  That's incredibly useful because we can refer to index levels by name.  If we happen to have an index without a name, we can set it with
python
df.index.name = 'Whatever name we like'
Multi-indexes
We can put more than one variable in an index, which gives us a multi-index.  This is sometimes called a hierarchical index because the levels of the index (as they're called) are ordered.
Multi-indexes are more common than you might think.  One reason is that data itself is often multi-dimensional.  A typical spreadsheet has two dimensions:  the variable and the observation.  The WEO data is naturally three dimensional:  the variable, the year, and the country.  (Think about that for a minute, it's deeper than it sounds.)
The problem we're having is fitting this nicely into two dimensions.  A multi-index allows us to manage that.  A two-dimensional index would work here -- the country and the variable code -- but right now we have some redundancy.
Example. We push all the descriptive, non-numerical columns into the index, leaving the dataframe itself with only numbers, which seems like a step in thee right direction.
Can you write Python code for it?

ddi = dd.set_index(['Variable', 'Country', 'ISO', 'Description', 'Units'])
ddi


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Exercise. Use a combination of xs and standard slicing with [...] to extract the variable Debt for Greece.
SOL

Exercise. Use the dataframe ddt -- and the xs method -- to plot Debt and Surplus across time for Argentina.
SOL

Stacking and unstacking
The set_index and reset_index methods work on the row labels -- the index.  They move columns to the index and the reverse.  The stack and unstack methods move index levels to and from column levels:

stack moves the "inner most" (closest to the data when printed) column label into a row label. This creates a long dataframe.
unstack does the reverse, it moves the inner most level of the index up to become the inner most column label. This creates a wide dataframe.

We use both to shape (or reshape) our data.  We use set_index to push things into the index.  And then use reset_index to push some of them back to the columns.  That gives us pretty fine-grainded control over the shape of our data. Intuitively

stacking (vertically): wide table $\rightarrow$ long table
unstacking: long table $\rightarrow$ wide table
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Let's get a smaller subset of this data to work with so we can see things a bit more clearly
Can you write Python code for it?

# drop some of the index levels (think s for small)
dds = ddi.reset_index(level=[1, 3, 4], drop=True)
dds

# give a name to the column labels
dds.columns.name = 'Year'
dds


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Pivoting the data
Let's think specifically about what we want.  We want to graph Emp against fsize for (say) 2013.  This calls for:

The index should be the size categories fsize.
The column labels should be the entries of year2, namely 2012, 2013 and `2014.
The data should come from the variable Emp.

These inputs translate directly into the following pivot method:
Can you write Python code for it?

bdsp = bds.pivot(index='fsize', columns='year2', values='Emp')

# divide by a million so bars aren't too long
bdsp = bdsp/10**6
bdsp


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Single level index
Can you write Python code for it?

# example from docstring
dic = {'a': [1, 3], 'b': [2, 4]}
s = pd.DataFrame(data=dic, index=['one', 'two'])
print(s)

s.stack()


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Swapping levels
Since variables refer to the first level of the column index, it's not clear how we would group data by country.  Suppose, for example, we wanted to plot Debt and Surplus for a specific country.  What would we do?
One way to do that is to make the country the top level with the swaplevel method.  Note the axis parameter.  With axis=1 we swap column levels, with axis=0 (the default) we swap row levels.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
That's a lot to process, so we break it into pieces.

ddi.index.names contains a list of level names.  (Remind yourself that lists are ordered, so this tracks levels.)
ddi.index.levels contains the values in each level.

Here's what they like like here:
Can you write Python code for it?

# Chase and Spencer like double quotes
print("The level names are:\n", ddi.index.names, "\n", sep="")
print("The levels (aka level values) are:\n", ddi.index.levels, sep="")


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Use this data to:

Set the index as Country.
Construct a horizontal bar chart of the number of doctors in each country in "2013 (or nearest year)".
Apply the drop method to docs to create a dataframe new that's missing the last column.
Challenging. Use stack and unstack to "pivot" the data so that columns are labeled by country names and rows are labeled by year.  This is challenging because we have left out the intermediate steps.
Plot the number of doctors over time in each country as a line in the same plot.

Comment. In the last plot, the x axis labels are non-intuitive.  Ignore that.
Resources
Far and away the best material on this subject is Brandon Rhodes' 2015 Pycon presentation.  2 hours and 25 minutes and worth every second.

Video:  https://youtu.be/5JnMutdy6Fw
Materials:  https://github.com/brandon-rhodes/pycon-pandas-tutorial
Outline:  https://github.com/brandon-rhodes/pycon-pandas-tutorial/blob/master/script.txt
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
Wants
We might imagine doing several different things with this data:

Plot a specific variable (debt or surplus) for a given date.
Time series plots for a specific country.
Time series plots for a specific variable.

Depending on which we want, we might organize the data differently.  We'll focus on the last two.
Here's a brute force approach to the problem:  simply transpose the data.  This is where that leads:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv2-shape.ipynb".
The first task is:
What's going on?  The theme is that we can reference the top level, which in ddi is the Variable.  If we try to access a lower level, it bombs.
Exercise. With the dataframe ddt:

What type of object is ddt["Debt"]?

Construct a line plot of Debt over time with one line for each country.

SOL
<!--
ddt['Debt'].dtypes
-->

SOL
<!--
ddt['Debt'].plot()
-->

Example. Let's do this together.  How would we fix up the legend?  What approaches cross your mind?  (No code, just the general approach.)
Can you write Python code for it?

fig, ax = plt.subplots()
ddt['Debt'].plot(ax=ax)
ax.legend(['ARG', 'DEU', 'GRE'], loc='best')
#ax.axhline(100, color='k', linestyle='--', alpha=.5)


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
(2) Selecting data using iloc
iloc is primarily used for integer position based indexing. That is, it selects rows and columns by number, in the order that they appear in the data frame. Numbers are from $0$ to df.shape-1 of both axes.
The syntax is
python
data.iloc[<row selection>, <column selection>]
Can you write Python code for it?

# Rows:
df.iloc[0]             # first row

df.iloc[-1]            # last row

# Columns:
df.iloc[:, 0]          # first column = first variable (first_name)

df.iloc[:, -1]         # last column (web)


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
(3) Selecting data using ix
ix is hybrid of loc and iloc. In general,
1. it is label-location based and acts just like loc
2. However, it also supports integer-location based selection just like iloc where passed an integer
Second option only works where the index of the DataFrame is NOT an integer.
The syntax is
python
data.ix[<row selection>, <column selection>]
Explicit usage of loc and iloc is preferred
Can you write Python code for it?

# ix indexing works just the same as loc when passed labels
dff.ix['Butt', 'city'] == dff.loc['Butt', 'city']

# ix indexing works the same as iloc when passed integers.
dff.ix[33, 7] == dff.iloc[33, 7]


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
A tricky one: we can pass a function that returns True/False values to .apply() and evaluate it at each row
Can you write Python code for it?

def short_company_name(x):
    """
    returns True if x contains less than 2 words
    """
    return len(x.split(' ')) < 2

dff.loc[dff['company_name'].apply(short_company_name)]


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
Boolean indexing using loc
The most common method to work with data
Pass an array of True/False values to the .loc to select the rows/columns with True values.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
Hierarchical indexing with loc and iloc
Multi-level indexing allows us to work with higher dimensional data while storing info in lower dimensional data structures like 2D DataFrame or 1D Series. More on this here.
Read the WEO dataset that we have already used a couple of times
Can you write Python code for it?

url_weo = 'http://www.imf.org/external/pubs/ft/weo/2016/02/weodata/WEOOct2016all.xls'

# (1) define the column indices
col_indices = [1, 2, 3] + list(range(9, 46))

# (2) download the dataset
weo = pd.read_csv(url_weo,
                  sep = '\t',
                  usecols=col_indices,
                  skipfooter=1, engine='python',
                  na_values=['n/a', '--'],
                  thousands =',')

# (3) change column labels to something more intuitive
weo = weo.rename(columns={'WEO Subject Code': 'Variable',
                          'Subject Descriptor': 'Description'})

# (4) create debt and deficits dataframe
variables = ['GGXWDG_NGDP', 'GGXCNL_NGDP']
data = weo[weo['Variable'].isin(variables)]

data['Variable'] = data['Variable'].replace(to_replace=['GGXWDG_NGDP', 'GGXCNL_NGDP'], value=['Debt', 'Surplus'])

data1 = data.set_index(['ISO', 'Variable'])
data1.head()

data1.ix[('AFG', slice(None)), '1980':'1983']


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
How would you get the same dataframe without loc?
Can you write Python code for it?

dff[dff['city'] == 'New Orleans'][['company_name', 'zip']]         # matter of taste


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
In fact, we don't need the loc indexer for this kind of task
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
Now we can directly select rows by their index (last_name) values (just like we do with columns)
Can you write Python code for it?

dff.loc['Butt']

# multiple rows
dff.loc[['Butt', 'Venere']]

# select a subset of the data (subDataFrame)
dff.loc[['Butt', 'Foller'], ['city', 'email']]

# ranges of index labels
dff.loc[['Butt', 'Foller'], 'address':'phone2']


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
Recall the string methods applicable to DataFrames
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv3-indexing.ipynb".
The first task is:
and the isin method?
Can you write Python code for it?

dff.loc[dff['city'].isin(['New Orleans', 'New York'])]

# intersection fo the two?

gmails = dff['email'].str.endswith("gmail.com")
NYNO = (dff['city'] == 'New Orleans') | (dff['city'] =='New York')
dff[gmails & NYNO]


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:

Merging ratings and movie titles
The movie ratings in the dataframe ratings give us individual opinions about movies, but they don't include the name of the movie.   Why not?  Rather than include the name every time a movie is rated, the MovieLens data associates each rating with a movie code, than stores the names of movies associatd with each movie code in the dataframe movies.  We run across this a lot:  some information is in one data table, other information is in another.
Our want is therefore to add the movie name to the ratings dataframe.  We say we merge the two dataferames.  There are lots of ways to merge.  Here we do one as an illustration.
Let's start by reminding ourselves what we have.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:

Population "estimates" and "projections"
We look (again) at the UN's  population data, specifically the age distribution of the population.  The data comes in two sheets:  estimates that cover the period 1950-2015 and projections that cover 2016-2100.  Our mission is to combine them.
Load data
We start, as usual, by loading the data.  This takes a minute or so.
Can you write Python code for it?

url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2015_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLS'
url = url1 + url2 + url3

cols = [2, 5] + list(range(6,28))
est = pd.read_excel(url, sheetname=0, skiprows=16, parse_cols=cols, na_values=['…'])
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])

print('Dimensions and dtypes of estimates: ', est.shape, '\n', est.dtypes.head(), sep='')
print('\nDimensions and dtypes of projections: ', prj.shape, '\n', prj.dtypes.head(), sep='')

est.to_csv('un_pop_est.csv')
prj.to_csv('un_pop_proj.csv')


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:
Comment. Note that they have different numbers of columns.  Let's see where that comes from.
Can you write Python code for it?

list(est)[15:]

list(prj)[15:]

est.head()

prj.head()


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:
Exercise.  Use set_index, stack, and unstack to shape the dataframe popi into popt.
Can you write Python code for it?

popi

popi.stack().unstack(level='Year')

list(range(1950, 2016, 5))

countries = ['United States of America', 'Japan']
past      = list(range(1950, 2016, 5))
future    = list(range(2015, 2101, 5))

e_US_J = cleanpop(est, countries, past)
p_US_J = cleanpop(prj, countries, future)

# fix top-coding in estimates
e_US_J['80-84'] = e_US_J['80-84'].fillna(0.0) + e_US_J['80+'].fillna(0.0)
e_US_J = e_US_J.drop(['80+'], axis=1)

e_US_J.head()

p_US_J[p_US_J['Country']=='United States of America'].head()

pop_US_J = pd.concat([e_US_J, p_US_J], axis=0)#.fillna(0.0)
pop_US_J.shape

pop_US_J

pop_i = pop_US_J.set_index(['Country', 'Year'])

pop_i.index

pop_i.columns.name = 'Age'

pop_st = pop_i.stack()

pop_st.head()

fig, ax = plt.subplots(2, 1, figsize=(12, 8))
pop_st.reorder_levels([2, 0, 1])['5-9']['United States of America'].plot(ax=ax[0], kind='line')
pop_st.reorder_levels([2, 0, 1])['5-9']['Japan'].plot(ax=ax[1], kind='line')
plt.show()

pop_st.head()

pop_st.loc[('Japan', 2100, slice(None))].plot(kind='bar')
plt.show()

pop_st.ix[('Japan', slice(None), '0-4')]


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:
Exercise. What happens if we try to merge the original dataframes, including the one with the extra 80+ column?  Run the code below and comment on what you get.
Can you write Python code for it?

popalt = pd.concat([ealt, palt], axis=0)
popalt


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:
Merge estimates and projections
If we have two blocks of data, and just want to put them on top of each other, we use the Pandas' concatenate function.  Ditto two blocks next to each other.
But first we need to fix the difference in the columns of the two dataframes.
Can you write Python code for it?

pop = pd.concat([e, p], axis=0).fillna(0.0)
pop


I am working on the file "Code/notebooks/bootcamp_pandas_adv4-merge-extended.ipynb".
The first task is:
Merging
Here's roughly what's involved in what we're doing.  We take the movieId variable from ratings and look it up in movies.  When we find it, we look up the title and add it as a column in ratings.  The variable movieId is common, so we can use it to link the two dataframes.
Can you write Python code for it?

combo = pd.merge(ratings, movies,   # left and right df's
                 how='left',        # add to left
                 on='movieId'       # link with this variable/column
                )

print('Dimensions of ratings:', ratings.shape)
print('Dimensions of movies:', movies.shape)
print('Dimensions of new df:', combo.shape)

combo.head(20)

combo_1 = ratings.merge(movies, how='left', on='movieId')

combo_1.head()

combo_2 = ratings.merge(movies, how='inner', on='movieId')
combo_2.shape

combo_3 = movies.merge(ratings, how='right', on='movieId')
combo_3.shape

# save as csv file for future use
combo.to_csv('mlcombined.csv')

count_2 = movies['movieId'].isin(ratings['movieId'])

count_2.sum()

print('Current directory:\n', os.getcwd(), sep='')
print('List of files:', os.listdir(), sep='\n')


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Background
Plotly is a javascript based plotting library. Plotly leverages industry grade
javascript technologies to provide great flexibility and good performance.
Being a javascript library, plotly graphics are inherently interactive meant to be
viewed in a webbrowser. The good news is that we can embed our interactive
plots in any website: Jupyter notebooks, blog posts, etc. The great news is
that we don't have to write any javascript ourselves!
The plotly project was started about four years ago. Over that time, plotly has
transitioned between three phases:

Online only mode: plotly started as a web service where you uploaded your
data and constructed plots on their website.
Online plotting mode: the next phase was to allow you to build plots from
your favorite programming language (Python!), but the plots were actually
created on their servers and you were given a link to view the plot on their
website.
Offline mode: You can now construct plotly graphics 100% offline. While all
three modes still exist, we will use the purely offline mode in our notes.

Matplotlylib
As a warmup, let's utilize our expertise of Matplotlib to quickly generate some basic plotly graphics.
The main steps in this process are:

Construct a matplotlib figure as we normally do.
Pass the Figure object (usually named fig in our examples) to the function iplot_mpl.

That's it!
Disclaimer: the functions that convert matplotlib figures to plotly figures are not perfect. We'll see some issues below, but will show how to build the plots using plotly's API so they look as we expect.
We'll start by looking at some examples from the seaborn documentation. The actual figures are not important here. We are mostly concerned with how well matplotlylib can take a matplotlib figure and construct a plotly figure.
Can you write Python code for it?

ax = sns.swarmplot(x="day", y="total_bill", data=tips)
fig_mpl = ax.get_figure()


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Choropleth
The other type of map plotly can create is called a choropleth map. Here we have each region filled in with a solid color.
Let's use our country data to see what this looks like
Can you write Python code for it?

trace = dict(type="choropleth",
             locations=papers["ISO"],   # use ISO names
             z=papers["publications"], # defines the color
             colorscale="Viridis",    # change pallette
             text=papers.index,         # change text on hover
             )

# reuse the same layout
iplot(go.Figure(data=[trace], layout=layout), link_text="")


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Data Sources
Before we get too far, we'll need some data. Let's get some now.
First we will download national data from the World Bank for all countries in Europe. In order to put this data on a map we will need to have either the 3 letter ISO code for the country or latitude and longitude coordinates. We'll grab both here. This will be a little sophistocated, so bear with us.
We show these steps here so that you can re-use this code if you want to do something similar in the future.
Can you write Python code for it?

# read a list of latitude and longitude coordinates for
# country capitals
lat_lon = pd.read_html("http://www.csgnetwork.com/llinfotable.html", header=0,
                       attrs={"align": "center", "cellpadding": 5, "bgcolor": "#FFFFFF"})[0]

# clean up so lat and long are numeric in degrees east and degrees north
def clean_latlon(series, to_negate):
    # get XX.YY data
    split1 = series.str.split("°")
    split2 = split1.str.get(1).str.split("'")
    data = split1.str.get(0) + "." + split1.str.get(0)

    # now add a negative side if last character == to_negate
    signs = split2.str.get(1) == to_negate
    signs = signs.replace({True: "-", False: ""})

    data = signs.str[:] + data.str[:]
    return data.astype(float)

lat_lon["Latitude"] = clean_latlon(lat_lon["Latitude"], "S")
lat_lon["Longitude"] = clean_latlon(lat_lon["Longitude"], "W")
lat_lon = lat_lon.drop("Capital", axis=1)
lat_lon = lat_lon.set_index("Country")
lat_lon.head()

# dataframe of country names and iso codes
url = "http://unstats.un.org/unsd/methods/m49/m49alpha.htm"
iso = pd.read_html(url, attrs={"border": "0", "cellpadding": "2"}, header=0)[0]
iso = iso.rename(columns={"ISO ALPHA-3 code": "ISO",
                          "Country or area name": "Country"})
iso = iso.drop("Numerical  code", axis=1)
iso = iso.set_index("Country")
iso.head()

def euro_wb_data(indicators, year=2013):# get data from worldbank
    europe = ["Albania",  "Andorra",  "Armenia",  "Austria",  "Azerbaijan",  "Belarus",
              "Belgium",  "Bosnia and Herzegovina",  "Bulgaria",  "Croatia",  "Cyprus",
              "Czech Republic",  "Denmark",  "Estonia",  "Finland",  "France",  "Georgia",
              "Germany",  "Greece",  "Hungary",  "Iceland",  "Ireland",  "Italy",
              "Kazakhstan",  "Kosovo",  "Latvia",  "Liechtenstein",  "Lithuania",
              "Luxembourg",  "Macedonia",  "Malta",  "Moldova",  "Monaco",  "Montenegro",
              "Netherlands",  "Norway",  "Poland",  "Portugal",  "Romania",  "Russia",
          ...


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Exercise: Look at the figure attribute reference and figure out how to remove the grid lines from the figure above. First remove the veritcal ones, then horizontal, then both. (Hint: Look for an attribute on the xaxis and yaxis of the layout) (Hint 2 you can get the layout by doing fig.layout)
Maps
We will finish this notebook by looking at some new things we can now do because we are using plotly.
There are two map-based traces in plotly:

scattergeo: this allows you do draw lines or dots on a map
choropleth: this allows you to fill regions with different colors

There is also the geo layout attribute. We'll look
scattergeo dots
We'll look at the scattergeo trace type first. Suppose we want to draw dots on the map. There are two possible sets of trace attributes we can work with:

Set lat and lon each to a list that specifiy the latitide and longitude for each point repsectively
Set the locationmode to be one of "ISO-3", "USA-states", or "country names" and then set location to be a valid member of that mode.

We can then set any other attributes
Let's see an example of each version:
Can you write Python code for it?

import numpy as np

# first create marker/layout objects we can re-use in both plots
marker = {"color": papers["publications"],
          "size": papers["publications"]/5000,
          "colorscale": "Reds",
          "colorbar": {"title": "# of papers"}}

layout = dict(geo={"scope": "europe", "resolution": 50},
              width=750, height=550)

# using location mode
trace = dict(type="scattergeo",      # trace type
             mode="markers",         # draw points
             locations=papers["ISO"],  # use ISO code
             marker=marker           # marker settings (size, color, ...)
            )

iplot(go.Figure(data=[trace], layout=layout), link_text="")

# using lat/lon mode
trace = dict(type="scattergeo",     # trace type
             mode="markers",        # draw dots
             lat=papers["Latitude"],  # latitude coordinate
             lon=papers["Longitude"], # longitude coordinate
             marker=marker          # marker settings (color, size...)
            )

iplot(go.Figure(data=[trace], layout=layout), link_text="")


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Exercise: now generate a similar plot using the df.iplot method. (Hint you can reuse the layout object from above)
Example: Dumbell plot. The striking fact in the data is that there is a gap between earnings of men and women. To highlight that, here's another way we might visualize the same data:
Can you write Python code for it?

men2 = dict(type="scatter",
            name="Men",
            mode="markers",                       # draw dots
            x=earnings["Men"],                    # x data
            y=earnings.index,                     # y data
            marker={"color": "Blue", "size": 12}  # dot color/size
           )
women2 = dict(type="scatter", name="Women", mode="markers",
              x=earnings["Women"], y=earnings.index,
              marker={"color": "Pink", "size": 12})

def draw_line(row):
    sc = row.name
    line = dict(type="scatter",                # trace type
                x=[row["Women"], row["Men"]],  # x data
                y=[sc, sc],                    # y data flat
                mode="lines",                  # draw line
                name=sc,                       # name trace
                showlegend=False,              # no legend entry
                line={"color": "gray"}         # line color
               )
    return line
lines = list(earnings.apply(draw_line, axis=1))

# use + for two lists
data = [men2, women2] + lines

# build and display the figure
fig = go.Figure(data=data, layout=layout)
iplot(fig)


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
For this example we see that the converter did a decent job, though it didn't quite get the xlabels correct
Let's try another example
Can you write Python code for it?

out = sns.pointplot(x="class", y="survived", hue="sex", data=titanic,
                    palette={"male": "g", "female": "m"},
                    markers=["^", "o"], linestyles=["-", "--"]);

iplot_mpl(out.get_figure())


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Now let's convert our Matplotlib figure fig_mpl into a plotly figure named fig_py. To do this we will use the function iplot_mpl as follows:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
Other data sources
Can you write Python code for it?

# Earnings by school and gender. Source plotly docs. Real source, unknown
url = "https://raw.githubusercontent.com/plotly/datasets/master/school_earnings.csv"
earnings = pd.read_csv(url)
print(earnings.head())

earnings = earnings.set_index("School")
earnings = earnings.sort_values("Men")
print("\n\nAfter set_index and sort_values:\n")
print(earnings.head())

# tips at restaurants in NYC. Source unknown, but classic dataset
tips = sns.load_dataset("tips")
tips.head()

# info on titanic passengers. Source unknown.
titanic = sns.load_dataset("titanic")
titanic.head()


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
USA States
This example was taken directly from the plotly python documentation. See here
Can you write Python code for it?

df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_us_ag_exports.csv')

for col in df.columns:
    df[col] = df[col].astype(str)

scl = [[0.0, 'rgb(242,240,247)'],[0.2, 'rgb(218,218,235)'],[0.4, 'rgb(188,189,220)'],\
            [0.6, 'rgb(158,154,200)'],[0.8, 'rgb(117,107,177)'],[1.0, 'rgb(84,39,143)']]

df['text'] = df['state'] + '
' +\
    'Beef '+df['beef']+' Dairy '+df['dairy']+'
'+\
    'Fruits '+df['total fruits']+' Veggies ' + df['total veggies']+'
'+\
    'Wheat '+df['wheat']+' Corn '+df['corn']

data = [ dict(
        type='choropleth',
        colorscale = scl,
        autocolorscale = False,
        locations = df['code'],
        z = df['total exports'].astype(float),
        locationmode = 'USA-states',
        text = df['text'],
        marker = dict(
            line = dict (
                color = 'rgb(255,255,255)',
                width = 2
            ) ),
        colorbar = dict(
            title = "Millions USD")
        ) ]

layout = dict(
        title = '2011 US Agriculture Exports by State
(Hover for breakdown)',
        geo = dict(
            scope='usa',
            projection=dict( type='albers usa' ),
            showlakes = True,
            lakecolor = 'rgb(255, 255, 255)'),
             )
iplot(go.Figure(data=data, layout=layout), link_text="")


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
What did and didn't work here?
Below we'll recreate this same figure using plotly's api and overcome these issues
Plotly API
Let's now consider how to use plotly's own API to construct plots instead of
building the graphics through matplotlib.
Plotly has over 20 core chart types and many more can be created by combining one or more chart types in the same figure. We don't have time to cover all of them here, but please check out the documentation.
Plotly has a purely declarative API. This means that we describe all the
features we want in our figure at once, without worrying about which functions
to call in what order.
The plotly can achieve this is by fully describing the plot in a data format called JSON.
For our purposes we can think of JSON as dictionaries, where values can be of any type,
including other dictionaries.
Plotly figures are composed of two things:

A list of traces that describe the data that will be plotted and how it should appear
A layout that describes features of the overall plot. Things like ticks, labels, titles, fonts, margins, etc.

The traces describe the data that should be plotted as well as how it should be displayed. Here's an example of a trace defining a scatter plot:
python
trace = dict(type="scatter",  # trace type
             x=[1, 2, 3],     # x data
             y=[1, 4, 9],     # y data
             name="Squares"   # legend label
             )
In this example, x, y, name and marker are called the attributes of the trace. All traces have a type attribute that describes the type of chart to generate for a particular piece of data. Here we chose scatter, which is what plotly calls scatter plots or line plots.
An example of a layout is
python
l = dict(title="Penguins food",            # plot title
         yaxis=dict(title="Quantity (%)",  # yaxis label
                    range=(0, 1)           # set limits for y axis
         )
Notice that the value associated with yaxis had type dict. This allowed us to control features of the yaxis.
For an...
Can you write Python code for it?

# the long way. Construct all the dicts by hand
men = dict(type="bar",                                      # trace type
           orientation="h",                                 # make bars horizontal
           name="Men",                                      # legend entry
           x=earnings["Men"],                               # x data
           y=earnings.index,                                # y data
           marker={"color": "Blue"}                         # blue bars
          )
women = dict(type="bar",                                    # trace type
             orientation="h",                               # horizontal bars
             name="Women",                                  # legend entry
             x=earnings["Women"],                           # x data
             y=earnings.index,                              # y data
             marker={"color": "Pink"}                       # pink bars
            )
layout = dict(width=650, height=750,                        # plot width/height
              yaxis={"title": "School"},                    # yaxis label
              title="Gender earnings disparity",            # title
              xaxis={"title": "Annual Salary (thousands)"}  # xaxis label}
             )
iplot(go.Figure(data=[men, women], layout=layout))


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
What worked well in this conversion? What didn't work?
Let's do one more example using the college graduate data
Can you write Python code for it?

fig_mpl, ax = plt.subplots(figsize=(6, 10))
earnings.plot.barh(ax=ax, y="Men", color="Blue")
earnings.plot.barh(ax=ax, y="Women", color="Pink");

iplot_mpl(fig_mpl)


I am working on the file "Code/notebooks/bootcamp_plotly.ipynb".
The first task is:
scattergeo lines
Now supppose we want to draw lines from the Italy to France, Germany, and Spain.
We can do this using scattergeo traces, where the lat and lon attributes on each trace are of length 2. The point will be drawn from (lat[0], lon[0]) to (lat[1], lon[1]):
Can you write Python code for it?

papers["Latitude"][papers.ISO == "ITA"].iloc[0]

def get_lat_lon_for(df, iso):
    lat = df["Latitude"][df["ISO"] == iso]
    lon = df["Longitude"][df["ISO"] == iso]
    return float(lat.iloc[0]), float(lon.iloc[0])

i_lat, i_lon = get_lat_lon_for(papers, "ITA")
traces = []

for country in ["FRA", "ESP", "DEU"]:
    lat, lon = get_lat_lon_for(papers, country)
    trace = dict(type="scattergeo",                # trace type
                 mode="lines",                     # draw lines
                 lat=[i_lat, lat],                 # latitude coordinates
                 lon=[i_lon, lon],                 # longitude coordinates
                 line={"width": 4.0},              # thick lines
                 name="RUS to {}".format(country)  # legend entry
                )
    traces.append(trace)

iplot(go.Figure(data=traces, layout=layout))


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Background
Plotly is a javascript based plotting library. Plotly leverages industry grade
javascript technologies to provide great flexibility and good performance.
Being a javascript library, plotly graphics are inherently interactive meant to be
viewed in a webbrowser. The good news is that we can embed our interactive
plots in any website: Jupyter notebooks, blog posts, etc. The great news is
that we don't have to write any javascript ourselves!
The plotly project was started about five years ago. Over that time, plotly has
transitioned between three phases:

Online only mode: plotly started as a web service where you uploaded your
data and constructed plots on their website.
Online plotting mode: the next phase was to allow you to build plots from
your favorite programming language (Python!), but the plots were actually
created on their servers and you were given a link to view the plot on their
website.
Offline mode: You can now construct plotly graphics 100% offline. While all
three modes still exist, we will use the purely offline mode in our notes.

Matplotlylib
As a warmup, let's utilize our expertise of Matplotlib to quickly generate some basic plotly graphics.
The main steps in this process are:

Construct a matplotlib figure as we normally do.
Pass the Figure object (usually named fig in our examples) to the function iplot_mpl.

That's it!
Disclaimer: the functions that convert matplotlib figures to plotly figures are not perfect. We'll see some issues below, but will show how to build the plots using plotly's API so they look as we expect.
We'll start by looking at some examples from the seaborn documentation. The actual figures are not important here. We are mostly concerned with how well matplotlylib can take a matplotlib figure and construct a plotly figure.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Choropleth
The other type of map plotly can create is called a choropleth map. Here we have each region filled in with a solid color.
Let's use our country data to see what this looks like
Can you write Python code for it?

trace = dict(type="choropleth",
             locations=papers["ISO"],   # use ISO names
             z=papers["publications"],  # defines the color
             colorscale="Viridis",      # change pallette
             text=papers.index,         # change text on hover
             )

# reuse the same layout
iplot(go.Figure(data=[trace], layout=layout), link_text="")


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Exercise: Look at the figure attribute reference and figure out how to remove the grid lines from the figure above. First remove the veritcal ones, then horizontal, then both. (Hint: Look for an attribute on the xaxis and yaxis of the layout) (Hint 2 you can get the layout by doing fig.layout)
Can you write Python code for it?

fig.layout

layout = fig.layout
layout['xaxis']['showgrid'] = False
layout['yaxis']['showgrid'] = False

fig = go.Figure(data=data, layout=layout)
iplot(fig)


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Financial charts
Another one from the docs https://plot.ly/python/candlestick-charts/#custom-candlestick-colors
Can you write Python code for it?

from plotly.tools import FigureFactory as FF
from plotly.graph_objs import Line, Marker
from datetime import datetime

df = web.DataReader("aapl", 'yahoo', datetime(2008, 1, 1), datetime(2009, 4, 1))
fig = FF.create_candlestick(df.Open, df.High, df.Low, df.Close, dates=df.index)

# Make increasing ohlc sticks and customize their color and name
fig_increasing = FF.create_candlestick(df.Open, df.High, df.Low, df.Close, dates=df.index,
    direction='increasing', name='AAPL',
    marker=Marker(color='rgb(150, 200, 250)'),
    line=Line(color='rgb(150, 200, 250)'))

# Make decreasing ohlc sticks and customize their color and name
fig_decreasing = FF.create_candlestick(df.Open, df.High, df.Low, df.Close, dates=df.index,
    direction='decreasing',
    marker=Marker(color='rgb(128, 128, 128)'),
    line=Line(color='rgb(128, 128, 128)'))

# Initialize the figure
fig = fig_increasing

# Add decreasing data with .extend()
fig['data'].extend(fig_decreasing['data'])

iplot(fig)


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
ISO codes
Access the ISO codes from this website
'https://unstats.un.org/unsd/methodology/m49/'
Can you write Python code for it?

#import ssl
#ssl._create_default_https_context = ssl._create_unverified_context

import requests
from bs4 import BeautifulSoup

# dataframe of country names and iso codes
url = 'https://unstats.un.org/unsd/methodology/m49/'
iso_raw = requests.get(url)
iso_soup = BeautifulSoup(iso_raw.content, 'html.parser')

iso_soup.find_all('table')[0]


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Maps
We will finish this notebook by looking at some new things we can now do because we are using plotly.
There are two map-based traces in plotly:

scattergeo: this allows you to draw lines or dots on a map
choropleth: this allows you to fill regions with different colors

There is also the geo layout attribute. We'll look
Can you write Python code for it?

# first create layout/marker objects we can re-use in both plots
layout = dict(geo={"scope": "europe", "resolution": 50},
              width=750, height=550)

marker = {"color": papers["publications"],
          "size": papers["publications"]/5000,
          "colorscale": "Reds",
          "colorbar": {"title": "# of papers"}}


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Other data sources
Can you write Python code for it?

# Earnings by school and gender. Source plotly docs. Real source, unknown
url = "https://raw.githubusercontent.com/plotly/datasets/master/school_earnings.csv"
earnings = pd.read_csv(url)
print(earnings.head())

earnings = earnings.set_index("School")
earnings = earnings.sort_values("Women")
print("\n\nAfter set_index and sort_values:\n")
print(earnings.head())

# tips at restaurants in NYC. Source unknown, but classic dataset
tips = sns.load_dataset("tips")
tips.head()

# info on titanic passengers. Source unknown.
titanic = sns.load_dataset("titanic")
titanic.head()


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
Publications (scientific and technical journal articles)
Can you write Python code for it?

# select 44 european countries
europe = ["Albania",  "Andorra",  "Armenia",  "Austria",  "Azerbaijan",  "Belarus",
              "Belgium",  "Bosnia and Herzegovina",  "Bulgaria",  "Croatia",  "Cyprus",
              "Czech Republic",  "Denmark",  "Estonia",  "Finland",  "France",  "Georgia",
              "Germany",  "Greece",  "Hungary",  "Iceland",  "Ireland",  "Italy",
              "Kazakhstan",  "Kosovo",  "Latvia",  "Liechtenstein",  "Lithuania",
              "Luxembourg",  "Macedonia",  "Malta",  "Moldova",  "Monaco",  "Montenegro",
              "Netherlands",  "Norway",  "Poland",  "Portugal",  "Romania",  "Russia",
              "San Marino",  "Serbia",  "Slovakia",  "Slovenia",  "Spain",  "Sweden",
              "Switzerland",  "Turkey",  "Ukraine",  "United Kingdom",  "Vatican City"]

iso[iso['Country'].isin(europe)]

def euro_wb_data(indicators, year=2013):                   # get data from worldbank

    iso_europe = iso[iso['Country'].isin(europe)]

    # IP.JRN.ARTC.SC is "scientific and technical journal articles"
    # NOTE: visit the world bank website to pick a different subject if you'd like
    # link: http://data.worldbank.org/indicator
    df = wb.download(country=iso_europe["ISO"], indicator=indicators,
                         start=year, end=year)
    df = df.reset_index(level="year")
    df.index.name = "Country"

    # some countries didn't have data. Drop them now
    df = df.dropna()

    return df

papers = euro_wb_data(["IP.JRN.ARTC.SC"])
papers.rename(columns={"IP.JRN.ARTC.SC": "publications"}, inplace=True)
papers = papers.reset_index()
papers.head()


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
What did and didn't work here?
Below we'll recreate this same figure using plotly's api and overcome these issues
Plotly API
Let's now consider how to use plotly's own API to construct plots instead of
building the graphics through matplotlib.
Plotly has over 20 core chart types and many more can be created by combining one or more chart types in the same figure. We don't have time to cover all of them here, but please check out the documentation.
Plotly has a purely declarative API. This means that we describe all the
features we want in our figure at once, without worrying about which functions
to call in what order.
The plotly can achieve this is by fully describing the plot in a data format called JSON.
For our purposes we can think of JSON as dictionaries, where values can be of any type,
including other dictionaries.
Plotly figures are composed of two things:

A list of traces that describe the data that will be plotted and how it should appear
A layout that describes features of the overall plot. Things like ticks, labels, titles, fonts, margins, etc.

The traces describe the data that should be plotted as well as how it should be displayed. Here's an example of a trace defining a scatter plot:
python
trace = dict(type="scatter",  # trace type
             x=[1, 2, 3],     # x data
             y=[1, 4, 9],     # y data
             name="Squares"   # legend label
             )
In this example, x, y, name and marker are called the attributes of the trace. All traces have a type attribute that describes the type of chart to generate for a particular piece of data. Here we chose scatter, which is what plotly calls scatter plots or line plots.
An example of a layout is
python
l = dict(title="Penguins food",            # plot title
         yaxis=dict(title="Quantity (%)",  # yaxis label
                    range=(0, 1)           # set limits for y axis
         )
Notice that the value associated with yaxis had type dict. This allowed us to control features of the yaxis.
For an...
Can you write Python code for it?

# the long way. Construct all the dicts by hand
men = dict(type="bar",                                      # trace type
           orientation="h",                                 # make bars horizontal
           name="Men",                                      # legend entry
           x=earnings["Men"],                               # x data
           y=earnings.index,                                # y data
           marker={"color": "Blue"}                         # blue bars
          )
women = dict(type="bar",                                    # trace type
             orientation="h",                               # horizontal bars
             name="Women",                                  # legend entry
             x=earnings["Women"],                           # x data
             y=earnings.index,                              # y data
             marker={"color": "Pink"}                       # pink bars
            )
layout = dict(width=650, height=750,                        # plot width/height
              yaxis={"title": "School"},                    # yaxis label
              title="Gender earnings disparity",            # title
              xaxis={"title": "Annual Salary (thousands)"}  # xaxis label
             )
iplot(go.Figure(data=[men, women], layout=layout))


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
What worked well in this conversion? What didn't work?
Let's do one more example using the college graduate data
Can you write Python code for it?

fig_mpl, ax = plt.subplots(figsize=(6, 10))
earnings.plot.barh(ax=ax, y="Men", color="Blue")
earnings.plot.barh(ax=ax, y="Women", color="Pink")

iplot_mpl(fig_mpl)


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
scattergeo dots
We'll look at the scattergeo trace type first. Suppose we want to draw dots on the map. There are two possible sets of trace attributes we can work with:

Set lat and lon each to a list that specifiy the latitide and longitude for each point respectively
Set the locations to be one of "ISO-3", "USA-states", or "country names" and then set location to be a valid member of that mode.

We can then set any other attributes
Let's see an example of each version:
Can you write Python code for it?

# using location mode
trace = dict(type="scattergeo",        # trace type
             mode="markers",           # draw points
             locations=papers["ISO"],  # use ISO code
             marker=marker             # marker settings (size, color, ...)
            )

iplot(go.Figure(data=[trace], layout=layout), link_text="")

# using lat/lon mode
trace = dict(type="scattergeo",     # trace type
             mode="markers",        # draw dots
             lat=papers["Latitude"],  # latitude coordinate
             lon=papers["Longitude"], # longitude coordinate
             marker=marker          # marker settings (color, size...)
            )

iplot(go.Figure(data=[trace], layout=layout), link_text="")


I am working on the file "Code/notebooks/bootcamp_plotly_update.ipynb".
The first task is:
scattergeo lines
Now supppose we want to draw lines from the Italy to France, Germany, and Spain.
We can do this using scattergeo traces, where the lat and lon attributes on each trace are of length 2. The point will be drawn from (lat[0], lon[0]) to (lat[1], lon[1]):
Can you write Python code for it?

papers["Latitude"][papers.ISO == "ITA"]

papers["Latitude"][papers.ISO == "ITA"].iloc[0]

def get_lat_lon_for(df, iso):
    lat = df["Latitude"][df["ISO"] == iso]
    lon = df["Longitude"][df["ISO"] == iso]
    return float(lat.iloc[0]), float(lon.iloc[0])

italy_lat, italy_lon = get_lat_lon_for(papers, "ITA")
traces = []

for country in ["FRA", "ESP", "DEU"]:
    lat, lon = get_lat_lon_for(papers, country)
    trace = dict(type="scattergeo",                # trace type
                 mode="lines",                     # draw lines
                 lat=[italy_lat, lat],             # latitude coordinates
                 lon=[italy_lon, lon],             # longitude coordinates
                 line={"width": 4.0},              # thick lines
                 name="ITA to {}".format(country)  # legend entry
                )
    traces.append(trace)

iplot(go.Figure(data=traces, layout=layout))


I am working on the file "Code/notebooks/bootcamp_practice_a.ipynb".
The first task is:
Answer these questions below. Code is sufficient, but it's often helpful to add comments to remind yourself what you did, and why.

Get help for the set.title method by typing ax.set_title? in a new cell and running it.  Note that you can open the documentation this produces in a separate tab with the icon in the upper right (hover text = "Open the pager in an external window").
Add a title with ax.set_title('Your title').
Change the fontsize of the title to 14.
What happens if we add the argument/parameter lw=2 to the ax.plot() statements?
Add a label to the x axis with ax.set_xlabel().
Add a label to the y axis.
Why did we use a log scale (logy=True)?  What happens if we don't?
Use the color argument/parameter to choose a more effective set of colors.
In what sense do you see "winner takes all"?  A "long tail"?

Question 4.  Japan's aging population
Populations are getting older throughout the world, but Japan is a striking example.  One of our favorite quotes:

Last year, for the first time, sales of adult diapers in Japan exceeded those for babies.

Let's see what the numbers look like using projections fron the United Nations' Population Division.  They have several projections; we use what they call the "medium variant."
We have a similar issue with the data:  population by age for a given country and date goes across rows, not down columns.  So we choose the ones we want and transpose them.  Again, more than we've done so far.
Can you write Python code for it?

# data input (takes about 20 seconds on a wireless network)
url1 = 'http://esa.un.org/unpd/wpp/DVD/Files/'
url2 = '1_Indicators%20(Standard)/EXCEL_FILES/1_Population/'
url3 = 'WPP2017_POP_F07_1_POPULATION_BY_AGE_BOTH_SEXES.XLSX'
url = url1 + url2 + url3

cols = [2, 4, 5] + list(range(6,28))
prj = pd.read_excel(url, sheetname=1, skiprows=16, parse_cols=cols, na_values=['…'])
print('Dimensions: ', prj.shape)
print('Column labels: ', prj.columns)

# rename some variables
pop = prj
pop = pop.rename(columns={'Reference date (as of 1 July)': 'Year',
                          'Region, subregion, country or area *': 'Country',
                          'Country code': 'Code'})
# select Japan and years
countries = ['Japan']
years     = [2015, 2035, 2055, 2075, 2095]
pop = pop[pop['Country'].isin(countries) & pop['Year'].isin(years)]
pop = pop.drop(['Country', 'Code'], axis=1)
pop = pop.set_index('Year').T
pop = pop/1000    # convert population from thousands to millions
pop.head()

pop.tail()


I am working on the file "Code/notebooks/bootcamp_practice_a.ipynb".
The first task is:
Comment. Now we have the number of people in any five-year age group running down columns.  The column labels are the years.
With the dataframe df:

Plot the current age distribution with pop[[2015]].plot().  Note that 2015 here does not have quotes around it:  it's an unusual case of integer column labels.
Plot the current age distribution as a bar chart.  Which do you think looks better?
Create figure and axis objects
Use the axis object to plot the age distribution for all the years in the dataframe.
Add titles and axis labels.
Plot the age distribution for each date in a separate subplot.  What argument parameter does this?  Bonus points: Change the size of the figure to accomodate the subplots.

Question 5.  Dynamics of the yield curve
One of our favorite topics is the yield curve:  a plot of the yield to maturity on a bond against the bond's maturity.  The foundation here is yields on zero coupon bonds, which are simpler objects than yields on coupon bonds.
We often refer to bond yields rising or falling, but in fact the yield curve often does different things at different maturities.  We will see that here.  For several years, short yields have been stuck at zero, yet yields for bond with maturities of two years and above have varied quite a bit.
We use the Fed's well-known Gurkaynak, Sack, and Wright data, which provides daily data on US Treasury yields from 1961 to the present.  The Fed posts the data, but it's in an unfriendly format.  So we saved it as a csv file, which we read in below.  The variables are yields:  SVENYnn is the yield for maturity nn years.
Can you write Python code for it?

# data input (takes about 20 seconds on a wireless network)
url = 'http://pages.stern.nyu.edu/~dbackus/Data/feds200628.csv'
gsw = pd.read_csv(url, skiprows=9, index_col=0, usecols=list(range(11)), parse_dates=True)
print('Dimensions: ', gsw.shape)
print('Column labels: ', gsw.columns)
print('Row labels: ', gsw.index)

# grab recent data
df = gsw[gsw.index >= dt.datetime(2010,1,1)]
# convert to annual, last day of year
df = df.resample('A', how='last').sort_index()
df.head()

df.columns = list(range(1,11))
ylds = df.T
ylds.head(3)


I am working on the file "Code/notebooks/bootcamp_practice_a.ipynb".
The first task is:
If you get something like "Pandas version:  0.17.1" you're fine.  If you get an error, bring your computer by and ask for help.  If you're unusually brave, go to StackOverflow and read the instructions.  Then come ask for help.  (This has to do with how your computer processes unicode.  When you hear that word -- unicode -- you should run away at high speed.)
Question 1. Setup
Import packages, arrange for graphs to display in the notebook.
Can you write Python code for it?

import pandas as pd
import matplotlib.pyplot as plt
import datetime as dt
%matplotlib inline


I am working on the file "Code/notebooks/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Comment. Now we have the number of people in any five-year age group running down columns.  The column labels are the years.
With the dataframe df:

Plot the current age distribution with pop[[2015]].plot().  Note that 2015 here does not have quotes around it:  it's an unusual case of integer column labels.
Plot the current age distribution as a bar chart.  Which do you think looks better?
Create figure and axis objects
Use the axis object to plot the age distribution for all the years in the dataframe.
Add titles and axis labels.
Plot the age distribution for each date in a separate subplot.  What argument parameter does this?  Bonus points: Change the size of the figure to accomodate the subplots.
Can you write Python code for it?

pop[[2015]].plot()

pop[[2015]].plot(kind='bar')

# my fav
pop[[2015]].plot(kind='barh')

fig, ax = plt.subplots(figsize=(10,6))
pop.plot(ax=ax)
ax.set_title('Population by age')
ax.set_xlabel('Population (millions)')
ax.set_ylabel('Age Range')

pop.plot(kind='bar', subplots=True, figsize=(6,8), sharey=True)


I am working on the file "Code/notebooks/bootcamp_practice_a_answerkey.ipynb".
The first task is:
If you get something like "Pandas version:  0.17.1" you're fine.  If you get an error, bring your computer by and ask for help.  If you're unusually brave, go to StackOverflow and read the instructions.  Then come ask for help.  (This has to do with how your computer processes unicode.  When you hear that word -- unicode -- you should run away at high speed.)
Question 1. Setup
Import packages, arrange for graphs to display in the notebook.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Question. Can you see consolidation here?
Can you write Python code for it?

# a basic plot
fig, ax = plt.subplots()

pdf[1947].plot(ax=ax, logy=True)
pdf[1967].plot(ax=ax, logy=True)
pdf[1987].plot(ax=ax, logy=True)
pdf[2004].plot(ax=ax, logy=True)
ax.legend()


I am working on the file "Code/notebooks/bootcamp_practice_a_answerkey.ipynb".
The first task is:
Remind yourself:

What does the pandas package do?  [data management]
What does the matplotlib package do?  [graphics]
What does %matplotlib inline do?  [displays plots in the notebook]

Question 2.  Jupyter basics

We refer to the cell that's highlighted as the current cell.
Clicking once on any cell makes it the current cell.  Clicking again allows you to edit it.
The + in the toolbar at the top creates a new cell below the current cell.
Change a cell from Code to Markdown (in other words, text) with the dropdown menu in the toolbar.
To run a cell, hit shift-enter or click on the run-cell icon in the tooolbar (sideways triangle and vertical line).
For more information, click on Help at the top.  User Interface Tour is a good place to start.

Practice with the following:

Make this cell the current cell.
Add an empty cell below it.
Add text to the new cell:  your name and the date, for example.
Optional:  Add a link to your LinkedIn or Facebook page.  Hint: Look at the text in the top cell to find an example of a link.
Run the cell.

Question 3. Winner take all and the long tail in the US beer industry
The internet has produced some interesting market behavior, music being a great example.  Among them:

Winner take all.  The large producers (Beyonce, for example) take larger shares of the market than they had in the past.
The long tail.  At the same time, small producers in aggregate increase their share.

Curiously enough, we see the same thing in the US beer industry:

Scale economies and a reduction in transportation costs (the interstate highway system was built in the 1950s and 60s) led to consolidation, with the large firms getting larger, and the small ones either sellingout or going bankrupt.  (How many beer brands can you think of that no longer exist?)
Starting in the 1980s, we saw a significant increase in the market share of small firms ("craft brewers") overall, even though each of them remains small.

We illustrate this with data from Victor and Carol Tremblay...
Can you write Python code for it?

url = 'http://pages.stern.nyu.edu/~dbackus/Data/beer_production_1947-2004.xlsx'
beer = pd.read_excel(url, skiprows=12, index_col=0)

print('Dimensions:', beer.shape)
beer[list(range(1,11))].head(3)

vars = list(range(1,101))   # extract top 100 firms
pdf = beer[vars].T          # transpose (flip rows and columns)
pdf[[1947, 1967, 1987, 2004]].head()


I am working on the file "Code/notebooks/bootcamp_practice_a_answerkey.ipynb".
The first task is:
With the dataframe ylds:

Create figure and axis objects
Use the axis object to plot the yield curve for all the years in the dataframe.
Add a title and axis labels.
Explain what you see:  What happened to the yield curve over the past six years?
Challenging.  Compute the mean yield for each maturity.  Plot them on the same graph in black.
Can you write Python code for it?

fig, ax = plt.subplots()
ylds.plot(ax=ax)
ax.set_title('US Treasury Yields')
ax.set_ylabel('Yield')
ax.set_xlabel('Maturity in Years')

ybar = ylds.mean(axis=1)
ybar.plot(ax=ax, color='black', linewidth=3, linestyle='dashed')


I am working on the file "Code/notebooks/bootcamp_scraping-sortof.ipynb".
The first task is:

Sometimes we get lucky
We sometimes find that we can access data straight from a web page with Pandas' read_html.  It works just like read_csv or read_excel.
The first example is baseball-reference.com.  The same people run similar sites for football and basketball.  Many of their pages are collections of tables.  See, for example, this one for Pittsburgh's Andrew McCucthen.
Can you write Python code for it?

# baseball reference
url = 'http://www.baseball-reference.com/players/m/mccutan01.shtml'
am  = pd.read_html(url)

print('Ouput has type', type(am), 'and length', len(am))
print('First element has type', type(am[0])')


I am working on the file "Code/notebooks/bootcamp_scraping-sortof.ipynb".
The first task is:
Here's another one:  Google's stock price from Yahoo finance.
Can you write Python code for it?

url = 'http://finance.yahoo.com/q/hp?s=GOOG+Historical+Prices'
ggl = pd.read_html(url)

type(ggl)

len(ggl)

ggl[8]

url = 'http://databootcamp.nyuecon.com/'
url = 'google.com'
db  = pd.read_html(url)


I am working on the file "Code/notebooks/bootcamp_scraping-sortof.ipynb".
The first task is:
Question. What do we have here?  A list of length 10?  Whose elements are dataframes?  Evidently this reads in all the tables from the page into dataframes and collects them in a list.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_scraping-sortof.ipynb".
The first task is:
Scanning urls
Itamar adds:
Walk through the following steps before running the code:
1) Go to : http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices
2) Enter the dates you want and hit the get prices button.
3) Once the results are shown, look on the url address.
4) The new url will include several parameters, each one is seperated by the & character.
5) Try to explore the meanning of each parameter (s, a,b,c,d,e,f and g)
6) After some trial and error you can realize that each parameter represents the data you entered as input:
    the day, month and year, the stock sybmol, and the frequency  you chose (daily, weekly etc)
7) Scroll down to the bottom of the page. there is a link which allows downloading the data as a csv file. click on it
8) Open the CSV in excel and see the structure of the file.
9) Go back to the web page, instead of clicking on the csv link, right click on it and copy the link address
10) Paste the address in a notebook - This is the url link we can use to access the data from our coding environment
Accessing web pages
Requests again...
Can you write Python code for it?

url = 'http://databootcamp.nyuecon.com/'
db = requests.get(url)

db.headers

db.url

db.status_code


I am working on the file "Code/notebooks/bootcamp_test.ipynb".
The first task is:
Now the test:

If you get the message below "Program halted, old version of Python, etc," you need to go back and install Anaconda again, this time following the instructions EXACTLY!  Yes, we know that's discouraging, but better to know now than run into problems later.

If you get the message "Congratulations etc," you're all set.  Pat yourself on the back. You can close out the program and go back to whatever you were doing.
Can you write Python code for it?

import sys

print('What version of Python are we running? \n', sys.version, '\n', sep='')

if float(sys.version_info[0]) < 3.0 :
    raise Exception('Program halted, old version of Python. \n',
                    'Sorry, you need to install Anaconda again.')
else:
    print('Congratulations, Python is up to date!')


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Dates in Pandas
Now we will look at how to use date and dateime functionality in pandas.
To begin, lets take a closer look at the type of index we have on our ffr and vc dataframes:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Dates in python
The date and time functionality in python comes from the built in datetime module. Notice above that we ran
python
import datetime as dt
We've been using the dt.date.today() function throughout this course when we print the date at the top of our notebooks, but we haven't given it very much thought. Let's take a closer look now.
To start, let's see what the type of dt.date.today() is
Can you write Python code for it?

today = dt.date.today()
print("the type of today is ", type(today))


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Given that we have an object of type datetime.date we can do things like ask for the day, month, and year
Can you write Python code for it?

print("the day of the month is: ", today.day)
print("we are curretly in month number", today.month)
print("The year is", today.year)


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Here we have a DatetimeIndex, which menas pandas recogizes this DataFrame as containing time series data.
What can we do now? A lot, here's a brief list:

subset the data using strings to get data for a particular time frame
resample the data to a diffrent frequency: this means we could convert daily to monthly, quarterly, etc.
quickly access things like year, month, and day for the observation
rolling computations: this will allow us to compute statistics on a rolling subset of the data. We'll show a simple example here, but check out the docs for more info
snap the observations to a particular frequency -- this one is a bit advanced and we won't cover it here

For a much more comprehensive list with other examples see the docs
For now, let's look at how to do these things with the data we obtained from quandl
NOTE You can only do these things when you have a DatetimeIndex. This means that even if one of the columns in your DataFrame has date or datetime information, you will need to set it as the index to access this functionality.
subsetting
Suppose we wanted to extract all the data for the federal funds rate for the year 2008.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Merging with dates
Let's see what happens when we merge the ffr and vc datasets
Can you write Python code for it?

# do a left merge on the index (date info)
df = pd.merge(ffr, vc, left_index=True, right_index=True, how="left")
df_info(df)

vc.head()


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Note that this is different from just resampling because we will have an observation for every date in the original dataframe (except the number of dates at the front needed to construct the initial window).
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Note that we can also combine numbers with the specification of the resampling frequency. As an example, we can resample to a bi-weekly frequency using
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Notice how the data associated with "M" and first is the same as the data for "MS" and first. The same holds for last.
Access year, month, day...
Given a DatetimeIndex you can access the day, month, or year (also second, millisecond, etc.) by simply accessing the .XX property; where XX is the data you want
Can you write Python code for it?

ffr.index.year

ffr.index.day

ffr.index.month


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Notice that the argument to strftime is a python string that can contain normal text (e.g. Today is) and a special formatters (the stuff starting with %). We haven't talked much about how to do string formatting, but in Python and many other languages using % inside strings has special meaning.
Exercise (6 min) Using the documentation for the string formatting behavior, figure out how to write the following strings using the method strftime method on the spencer_bday_time object

"Spencer was born on 1989-04-25"
"Spencer was born on a Tuesday"
"Spencer was born on Tuesday, April 25th"
(bonus) "Spencer was born on Tuesday, April 25th at 04:33 PM"
Can you write Python code for it?

spencer_bday_time.strftime("Spencer was born on %A, %B %dth at %I:%M %p")


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Notice that we ended up with a lot of missing data. This happened for two reasons:

The ffr data goes back to 1954, but the vc data starts in 1985
The ffr data is at a daily frequency, but vc is at quarterly.

To resolve the first issue we can subset the ffr data and only keep from 1985 on
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Notice that when we call resample we don't get back a DataFrame at that frequency. This is because there is some ambiguity regarding just how the frequency should be converted: should we take the average during the period, the first observation, last observation, sum the observations?
In order to get a DataFrame we have to call a method on our DatetimeIndexResampler object. For this example, let's do the first observation in each period:
Can you write Python code for it?

ffrM = ffrM_resample.first()
df_info(ffrM)


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Now, let's read in the data sets we found were interesting. Feel free to use the codes you looked up, or the ones I'm using here.
Can you write Python code for it?

my_data = {
    "FRED/DFF": "risk_free_rate",
    "NVCA/VENTURE_3_09C": "vc_investments"
}

dfs = []

for k in my_data.keys():
    dfs.append(quandl.get(k))

df_info(dfs[0])

df_info(dfs[1])


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Quandl
quandl is a company that collects and maintains financial and economic data from standard sources (e.g. FRED, IMF, BEA, etc.) and non-standard sources (Fx data, company level data, trader receipts). The data is viewable on their webpage (see here or there for examples), but made available to programming languages via their API. We will access their API using their python library.
Suppose, for example, that we wanted to get data on taxes in the US. Here's how we might find some:

Open up the quandl search page
Type in "US tax revenue"
Click on one of the results that seems interesting to us
Checkout things like the frequency (Annual for this data set), the quandl code (top right, here it is OECD/REV_NES_TOTALTAX_TAXUSD_USA) and description.

Exercise (5 min): Go to Quandl's website and explore some of the data quandl has available. Come up with 2 datasets and make a dictionary that maps the quandl code into a reasonable name. For example, for the us tax revenue dataset above I could have done
python
my_data = {"OECD/REV_NES_TOTALTAX_TAXUSD_USA": "US_tax_rev"}
We can download the data using the quandl.get function and passing it one of the Quandl codes we collected in the previous exercise
Can you write Python code for it?

us_tax = quandl.get("OECD/REV_NES_TOTALTAX_TAXUSD_USA")
df_info(us_tax)


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
Rolling computations
We can use pandas to do rolling computations.
For example, suppose we want to plot the maximum and minimum of the risk free rate within the past week at each date (think about that slowly -- for every date, we want to look back 7 days and compute the max).
Here's how we can do that
Can you write Python code for it?

fig, ax = plt.subplots()
ffr.rolling(window=7).max().plot(ax=ax)
ffr.rolling(window=7).min().plot(ax=ax)
ax.legend(["max", "min"])


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
The second dataframe we dowloaded (using code NVCA/VENTURE_3_09C) contains quarterly data on total investment by venture capital firms in the US, broken down by the stage of the project.
The column names here are ok, so we don't need to change anything.
Exercise (5 min) do a simlar analysis/report for whatever datasets you chose to work with. Make sure to do the following:

Make note of the frequency of the data (e.g. daily, monthly, quarterly, yearly, etc.)
Check the column names

If you chose to use the same data as me, do something interesting with the data. Perhaps construct plots of differenet variables, or compute summary statistics -- use your imagination here.
So that we have the data easily acessible for later on, let's store these two variables in individual dataframes:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
To resolve the second issue we will do two-steps:

resample the ffr data to a monthly frequency
resample the vc data to a monthly frequency by padding. This is called upsampling because we are going from a lower frequency (quarterly) to a higher one (monthly)
Can you write Python code for it?

ffr_recentM = ffr_recent.resample("M").first()
vc_M = vc.resample("M").pad()

vc_M.head()


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
We can also pass start_date and end_date parameters to control the dates for the downloaded data:
Can you write Python code for it?

us_tax_recent = quandl.get("OECD/REV_NES_TOTALTAX_TAXUSD_USA", start_date="2000-01-01")
df_info(us_tax_recent)


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
We can use this same functionality to extract ranges of dates. To get the data starting in june 2007 and going until march 2011 we would do
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
datetime
Being able to work with dates and the difference between dates is very useful, but sometimes we need to also think about times. To do that, we will look to the dt.datetime module.
We can get the current date and time using dt.datetime.now():
Can you write Python code for it?

now = dt.datetime.now()
print("type of now:", type(now))
now


I am working on the file "Code/notebooks/bootcamp_timeseries.ipynb".
The first task is:
strftime
Once we have date and time information, a very common thing to do is to print out a formatted version of that date. For example, suppose we wanted to print out a string in the format YYYY-MM-DD. To do this we use the strftime method. Here's an example
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
"Spencer was born on Tuesday, Apr 25th"
Can you write Python code for it?

spencer_bday_time.strftime("Spencer was born on %A, %b %dth")


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
(bonus) "Spencer was born on Tuesday, April 25th at 04:33 PM"
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
An annual frequency -- use the end of the year
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Exercise Using one of your datasets from quandl, plot one or more variables for the last 3 years (2013 through 2016)
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Exercise: Use tab completion to see what else we can access on our dt.datetime object now
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Exercise: Using the documentation for the most common frequencies, figure out how to resample one of your datasets to

A quarterly frequency -- make sure to get the start of the quarter
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Exercise: write a python function named days_until that accepts one argument (a datetime.date) and returns the number of days between today and that date. Apply your function to

May 5, 2017 (day the UG project is due)
Your birthday
Can you write Python code for it?

def days_until(date):
    today = dt.date.today()
    numb_days = date - today

    return numb_days.days

project_due = dt.date(2017, 5, 5)

days_until(project_due)


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Here we have a DatetimeIndex, which means pandas recogizes this DataFrame as containing time series data.
What can we do now? A lot, here's a brief list:

subset the data using strings to get data for a particular time frame
resample the data to a diffrent frequency: this means we could convert daily to monthly, quarterly, etc.
quickly access things like year, month, and day for the observation
rolling computations: this will allow us to compute statistics on a rolling subset of the data. We'll show a simple example here, but check out the docs for more info
snap the observations to a particular frequency -- this one is a bit advanced and we won't cover it here

For a much more comprehensive list with other examples see the docs
For now, let's look at how to do these things with the data we obtained from quandl
NOTE You can only do these things when you have a DatetimeIndex. This means that even if one of the columns in your DataFrame has date or datetime information, you will need to set it as the index to access this functionality.
subsetting
Suppose we wanted to extract all the data for the federal funds rate for the year 2008.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Merging with dates
Let's see what happens when we merge the ffr and vc datasets
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Note that this is different from just resampling because we will have an observation for every date in the original dataframe (except the number of dates at the front needed to construct the initial window).
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Note that we can also combine numbers with the specification of the resampling frequency. As an example, we can resample to a bi-weekly frequency using
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Notice how the data associated with "M" and first is the same as the data for "MS" and first. The same holds for last.
Access year, month, day...
Given a DatetimeIndex you can access the day, month, or year (also second, millisecond, etc.) by simply accessing the .XX property; where XX is the data you want
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Notice that the argument to strftime is a python string that can contain normal text (e.g. Today is) and a special formatters (the stuff starting with %). We haven't talked much about how to do string formatting, but in Python and many other languages using % inside strings has special meaning.
Exercise Using the documentation for the string formatting behavior, figure out how to write the following strings using the method strftime method on the spencer_bday_time object

"Spencer was born on 1989-04-25"
Can you write Python code for it?

spencer_bday_time.strftime("Spencer was born on %Y-%m-%d")


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Notice that using pad here just copied data forwards to fill in missing months (e.g. the data for March 1985 was applied to April and May)
Now let's try that merge again
Can you write Python code for it?

df = pd.merge(ffr_recentM, vc_M, left_index=True, right_index=True, how="left")
print(df.head(6))
print("\n\n", df.tail(8))


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Notice that we ended up with a lot of missing data. This happened for two reasons:

The ffr data goes back to 1954, but the vc data starts in 1985
The ffr data is at a daily frequency, but vc is at quarterly.

To resolve the first issue we can subset the ffr data and only keep from 1985 on
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Now, let's read in the data sets we found were interesting. Feel free to use the codes you looked up, or the ones I'm using here.
Can you write Python code for it?

# For multiple data sources, it's useful to define a list/dict
my_data = {"FRED/DFF": "risk_free_rate",
           "NVCA/VENTURE_3_09C": "vc_investments"}

dfs = []

for k in my_data.keys():
    dfs.append(quandl.get(k))

df_info(dfs[0])


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Quandl
quandl is a company that collects and maintains financial and economic data from standard sources (e.g. FRED, IMF, BEA, etc.) and non-standard sources (Fx data, company level data, trader receipts). The data is viewable on their webpage (see here or there for examples), but made available to programming languages via their API. We will access their API using their python library.
Suppose, for example, that we wanted to get data on taxes in the US. Here's how we might find some:

Open up the quandl search page
Type in "US tax revenue"
Click on one of the results that seems interesting to us
Checkout things like the frequency (Annual for this data set), the quandl code (top right, here it is OECD/REV_NES_TOTALTAX_TAXUSD_USA) and description.

Exercise (5 min): Go to Quandl's website and explore some of the data quandl has available. Come up with 2 datasets and make a dictionary that maps the quandl code into a reasonable name. For example, for the us tax revenue dataset above I could have done
python
my_data = {"OECD/REV_NES_TOTALTAX_TAXUSD_USA": "US_tax_rev"}
We can download the data using the quandl.get function and passing it one of the Quandl codes we collected in the previous exercise
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
So, "FRED/DFF" is the federal funds rate, or the interest rate at which banks can trade federal assets with each other overnight. This is often used as a proxy for the risk free rate in economic analysis.
From the printout above it looks like we have more than 22k observations starting in 1954 at a daily frequency.
Notice, however that the column name is DFF. Let's use our dict to clean up that name:
Can you write Python code for it?

dfs[1].rename(columns={"DFF": my_data["FRED/DFF"]}, inplace=True)
df_info(dfs[1])