I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
NER and Civil War-Era Novels
Wilkens uses a technique called "NER", or "Named Entity Recognition" to let the computer identify all of the geographic place names. Wilkens writes:

Text strings representing named locations in the corpus were identified using
the named entity recognizer of the Stanford CoreNLP package with supplied training
data. To reduce errors and to narrow the results for human review, only those
named-location strings that occurred at least five times in the corpus and were used
by at least two different authors were accepted. The remaining unique strings were
reviewed by hand against their context in each source volume. [883]

While we don't have the time for a human review right now, spacy does allow us to annotate place names (among other things!) in the same fashion as Stanford CoreNLP (a native Java library):
Can you write Python code for it?

ner_tab = Table()
ner_tab.append_column(label="NER Label", values=[ent.label_ for ent in parsed_text.ents])
ner_tab.append_column(label="NER Text", values=[ent.text for ent in parsed_text.ents])
ner_tab.show()


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
Now we can plot a nice choropleth.
Can you write Python code for it?

%matplotlib inline

from scripts.choropleth import us_choropleth
us_choropleth(geo_tab)


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
SpaCy: Industrial-Strength NLP
The tradtional NLP library has always been NLTK. While NLTK is still very useful for linguistics analysis and exporation, spacy has become a nice option for easy and fast implementation of the NLP pipeline. What's the NLP pipeline? It's a number of common steps computational linguists perform to help them (and the computer) better understand textual data. Digital Humanists are often fond of the pipeline because it gives us more things to count! Let's what spacy can give us that we can count.
Can you write Python code for it?


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
That was quick! So what happened? We've talked a lot about tokenizing, either in words or sentences.
What about sentences?
Can you write Python code for it?

sents_tab = Table()
sents_tab.append_column(label="Sentence", values=[sentence.text for sentence in parsed_text.sents])
sents_tab.show()


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
To parse an entire text we just call the model on a string.
Can you write Python code for it?

parsed_text = nlp(my_string)
parsed_text


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
We'll leave the chapter headers for now, it shouldn't affect much. Now we need to parse this with that nlp function:
Can you write Python code for it?


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
What about parts of speech?
Can you write Python code for it?

toks_tab.append_column(label="POS", values=[word.pos_ for word in parsed_text])
toks_tab.show()


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
What else? Let's just make a function tablefy that will make a table of all this information for us:
Can you write Python code for it?

def tablefy(parsed_text):
    toks_tab = Table()
    toks_tab.append_column(label="Word", values=[word.text for word in parsed_text])
    toks_tab.append_column(label="POS", values=[word.pos_ for word in parsed_text])
    toks_tab.append_column(label="Lemma", values=[word.lemma_ for word in parsed_text])
    toks_tab.append_column(label="Stop Word", values=[word.is_stop for word in parsed_text])
    toks_tab.append_column(label="Punctuation", values=[word.is_punct for word in parsed_text])
    toks_tab.append_column(label="Space", values=[word.is_space for word in parsed_text])
    toks_tab.append_column(label="Number", values=[word.like_num for word in parsed_text])
    toks_tab.append_column(label="OOV", values=[word.is_oov for word in parsed_text])
    toks_tab.append_column(label="Dependency", values=[word.dep_ for word in parsed_text])
    return toks_tab

tablefy(parsed_text).show()


I am working on the file "05-Intro-to-SpaCy/01-Intro-to-SpaCy.ipynb".
The first task is:
Words?
Can you write Python code for it?

toks_tab = Table()
toks_tab.append_column(label="Word", values=[word.text for word in parsed_text])
toks_tab.show()


I am working on the file "05-MultiDimMultiFormatData.ipynb".
The first task is:
<img src='./files/2dbase1.png', width="300">
Can you write Python code for it?


I am working on the file "05-MultiDimMultiFormatData.ipynb".
The first task is:
<img src='./files/2dbase2.png', width="300">
Can you write Python code for it?


I am working on the file "05-MultiDimMultiFormatData.ipynb".
The first task is:
<img src='./files/3dbase2.png', width="300">
Can you write Python code for it?


I am working on the file "05-MultiDimMultiFormatData.ipynb".
The first task is:
<img src='./files/3dbase5.png', width="300">
Can you write Python code for it?


I am working on the file "05-MultiDimMultiFormatData.ipynb".
The first task is:
Mais que 3 dimensões

Exemplo real

Can you write Python code for it?

from netCDF4 import Dataset

nc = Dataset('./data/mdt_cnes_cls2009_global_v1.1.nc')

nc

u = nc.variables['Grid_0002']
u

v = nc.variables['Grid_0003']
v

u, v = u[:], v[:]

lon = nc.variables['NbLongitudes'][:]
lat = nc.variables['NbLatitudes'][:]

import numpy as np

lon, lat = np.meshgrid(lon, lat)

lon.shape, lat.shape, u.shape, v.shape

%matplotlib inline
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
sub = 5
ax.quiver(lon[::sub, ::sub], lat[::sub, ::sub], u.T[::sub, ::sub], v.T[::sub, ::sub])

from oceans import wrap_lon180
lon = wrap_lon180(lon)

import matplotlib.pyplot as plt

import cartopy.crs as ccrs
from cartopy.io import shapereader
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER

def make_map(projection=ccrs.PlateCarree()):
    fig, ax = plt.subplots(figsize=(9, 13),
                           subplot_kw=dict(projection=projection))
    gl = ax.gridlines(draw_labels=True)
    gl.xlabels_top = gl.ylabels_right = False
    gl.xformatter = LONGITUDE_FORMATTER
    gl.yformatter = LATITUDE_FORMATTER
    return fig, ax

mask_x = np.logical_and(lon > -40, lon < -36)
mask_y = np.logical_and(lat > -15, lat < -12)
mask = np.logical_and(mask_x, mask_y)

import cartopy.feature as cfeature
land_10m = cfeature.NaturalEarthFeature('physical', 'land', '10m',
                                        edgecolor='face',
                                        facecolor=cfeature.COLORS['land'])

fig, ax = make_map()
ax.quiver(lon[mask], lat[mask], u.T[mask], v.T[mask])
ax.add_feature(land_10m)
ax.coastlines('10m')
ax.set_extent([-40, -36, -15, -12])


I am working on the file "05-Naredba-grananja-if.ipynb".
The first task is:
Ključna reč elif, skraćeno od else-if, omogućava da naredbu grananja koristimo na sledeći način:

* Ukoliko je uslov 1 zadovoljen, izvrši blok 1
* Ukoliko nije, a zadovoljen je uslov 2, izvrši blok 2
* Ukoliko nije, a zadovoljen je uslov 3, izvrši blok 3
* ...
* Ukoliko nije, a zadovoljen je uslov N, izvrši blok N
* Ukoliko nije, izvrši blok
Sledi primer u kome ćemo učitati jedan samoglasnik s tastature, a zatim ispisati ime koje počinje tim samoglasnikom.
Can you write Python code for it?

# The names
a_name = "Adriana"
e_name = "Ernest"
i_name = "Ian"
o_name = "Olivia"
u_name = "Ursula"

# Get a string from the keyboard
instr = input("Please enter a vowel: ")

out_template = "The name starting with \"%s\" is %s."

# Print the name or throw an exception if the given string is not a vowel
if ("a" == instr) or ("A" == instr):
    print(out_template % (instr, a_name))
elif ("e" == instr) or ("E" == instr):
    print(out_template % (instr, e_name))
elif ("i" == instr) or ("I" == instr):
    print(out_template % (instr, i_name))
elif ("o" == instr) or ("O" == instr):
    print(out_template % (instr, o_name))
elif ("u" == instr) or ("U" == instr):
    print(out_template % (instr, u_name))
else:
    print("The entered text is not a vowel.")


I am working on the file "05-Naredba-grananja-if.ipynb".
The first task is:
Ukoliko želimo da ispitamo da li je zadovoljeno dva ili više uslova, koristimo Pajtonov logički operator AND (matematička operacija konjukcije, odnosno matematičko "i").
Can you write Python code for it?

number = int(input("Enter a number: "))

if (number > 0) and (number < 10):
    print("The number is between 0 and 10.")
else:
    print("The number is not between 0 and 10.")


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Atlanta Police Department
The Atlanta Police Department provides Part 1 crime data at http://www.atlantapd.org/i-want-to/crime-data-downloads
A recent copy of the data file is stored in the cluster. Please, do not copy this data file into your home directory!
Can you write Python code for it?

### Load libraries
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Convert into date-time type
Can you write Python code for it?

df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)

#df[['offense_id', 'occur_date', 'occur_time', 'occur_ts', 'rpt_date']][1:10]

df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)

df['occur_month'] = df['occur_ts'].map(lambda x: x.month)
df['occur_woy'] = df.occur_ts.dt.weekofyear

df.describe()

df.shape

df.columns

df.iloc[9]

#resdf.index

resdf.loc[['AUTO_THEFT', 6]]

resdf = df.groupby(['UC2 Literal', 'occur_date']).offense_id.count()
resdf

resdf['BURGLARY-RESIDENCE'].as_matrix()

resdf['BURGLARY-RESIDENCE'].iloc(0)

resdf['BURGLARY-RESIDENCE']

%matplotlib inline
fig = plt.figure(figsize=(10,6)) # 10inx10in
#plt.plot(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'])
plt.scatter(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'].values, marker='x')
# plt.scatter(resdf['BURGLARY-NONRES'].index, resdf['BURGLARY-NONRES'], marker='o')

# plt.ylim(0, 500)
# plt.title('BURGLARY-RESIDENCE')
# plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
# fig.savefig('BurglaryResidence_over_month.svg')
# x = 1

def getTheMonth(x):
    return x.month

df['occur_month'] = df['occur_ts'].map(getTheMonth)

df = pd.read_excel('/home/data/APD/COBRA083016_2015.xlsx', sheetname='Query')
df['occur_ts'] = pd.to_datetime(df.occur_date+' '+df.occur_time)
df['occur_month'] = df['occur_ts'].map(lambda x: x.month)
df['occur_woy'] = df.occur_ts.dt.weekofyear

%matplotlib inline
resdf = df.groupby(['UC2 Literal', 'occur_month']).offense_id.count()
fig = plt.figure(figsize=(10,6))
plt.scatter(resdf['BURGLARY-RESIDENCE'].index, resdf['BURGLARY-RESIDENCE'], marker='x')
plt.ylim(0, 500)
plt.title('BURGLARY-RESIDENCE')
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.savefig('quiz3-burglary-residence.png')

''

plt.savefig('quiz3-burglary-residence.png')


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Exploring Dates
Can you write Python code for it?

df[['offense_id', 'occur_date', 'occur_time', 'rpt_date']][1:10]


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Fitting the regression line
Suppose there are $n$ data points {{math|{(''xi'', ''yi''), ''i'' {{=}} 1, ..., ''n''}.}} The function that describes x and y is:
$$y_i = \alpha + \beta x_i + \varepsilon_i.$$
The goal is to find the equation of the straight line
$$y = \alpha + \beta x,$$
which would provide a "best" fit for the data points. Here the "best" will be understood as in the [[Ordinary least squares|least-squares]] approach: a line that minimizes the sum of squared residuals of the linear regression model. In other words, {{mvar|α}} (the {{mvar|y}}-intercept) and {{mvar|β}} (the slope) solve the following minimization problem:
$$\text{Find }\min_{\alpha,\,\beta} Q(\alpha, \beta), \qquad \text{for } Q(\alpha, \beta) = \sum_{i=1}^n\varepsilon_i^{\,2} = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2\ $$
By using either [[calculus]], the geometry of [[inner product space]]s, or simply expanding to get a quadratic expression in {{mvar|α}} and {{mvar|β}}, it can be shown that the values of {{mvar|α}} and {{mvar|β}} that minimize the objective function {{mvar|Q}}<ref>Kenney, J. F. and Keeping, E. S. (1962) "Linear Regression and Correlation." Ch. 15 in ''Mathematics of Statistics'', Pt. 1, 3rd ed. Princeton, NJ: Van Nostrand, pp. 252–285</ref> are
: <math>\begin{align}
  \hat\beta &= \frac{ \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^n (x_i - \bar{x})^2 } \[6pt]
            &= \frac{ \sum_{i=1}^{n} (x_i y_i - x_i \bar{y} - \bar{x} y_i + \bar{x} \bar{y})} { \sum_{i=1}^n (x_i^2 - 2 x_i \bar{x} + \bar{x}^2) } \[6pt]
            &= \frac{ \sum_{i=1}^{n} (x_i y_i) - \bar{y} \sum_{i=1}^{n}  x_i  - \bar{x} \sum_{i=1}^{n} y_i + n  \bar{x} \bar{y}} { \sum_{i=1}^n (x_i^2) - 2 \bar{x}  \sum_{i=1}^n  x_i  + n \bar{x}^2 } \[6pt]
            &= \frac{ \frac{1}{n} \sum_{i=1}^{n} x_i y_i - \bar{x} \bar{y} }{ \frac{1}{n}\sum_{i=1}^n {x_i^2} -   \overline{x}^2 } \[6pt]
            &= \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } = \frac{ \operatorname{Cov}[x, y] }{...
Can you write Python code for it?

### in case we want to save a DataFrame
#writer = pd.ExcelWriter('myresults.xlsx')
#df.to_excel(writer,'Results')
#writer.save()

#resdf


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Let's look at residential burglary.
Can you write Python code for it?

c = 3
crime_types[c]

crime_year_month = dff[dff.occur_year.between(2009, 2015)].groupby(by=['UC2 Literal', 'occur_year', 'occur_month']).offense_id.count()

c = 3 ## 'BURGLARY-RESIDENCE'
resburglaries = crime_year_month.loc[crime_types[c]]
fig = plt.figure(figsize=(20,10))
for y in years:
    plt.plot(resburglaries.loc[y].index, resburglaries.loc[y], label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s"%crime_types[c], fontsize=20)
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
None


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Part 1 - Observations from the data
Can you write Python code for it?

pd.unique(df) # how to get duplicate records?

pd.unique(df['UC2 Literal'])

len(pd.unique(df.MI_PRINX))


I am working on the file "05-Operating-with-Multiple-Tables/APD-Crime-Data_edit.ipynb".
The first task is:
Part 2 - Seasonal Model
Can you write Python code for it?

## load complete dataset
dff = pd.read_excel('/home/data/APD/COBRA083016.xlsx', sheetname='Query')

dff.shape

for evt in ['occur', 'poss']:
    dff['%s_ts'%evt] = pd.to_datetime(dff['%s_date'%evt]+' '+dff['%s_time'%evt])
dff['rpt_ts'] = pd.to_datetime(dff.rpt_date)

', '.join(dff.columns)

dff['occur_year'] = dff.occur_ts.dt.year
dff['occur_month'] = dff.occur_ts.dt.month
dff['occur_dayweek'] = dff.occur_ts.dt.dayofweek


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
"For the list above, limit the results to just salespeople"
Can you write Python code for it?

Ans2 = Ans[Ans.JobTitle=='Sales Representative']
Ans2

# Overachiever: What about *all* employees associated with sales?
Ans2 = Ans[Ans["JobTitle"].apply(lambda x: 'Sales' in x)]
Ans2


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
"Give me a list of all sales territories, also show what customers fall under them"
Can you write Python code for it?

Ans = pd.merge(Territory, Customers, how="inner", left_on="TerritoryID", right_on="SalesTerritoryID")

Ans


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
"Give me a list of our customers, and also tell me which sales territory they fall in."
This looks like another question for "merge"! We have a list of customers with their addresses, and we have a list of territories, but they are in separate tables.
Let's recover a list of customer names and IDs, together with corresponding sales territory names.
This time, we have to be careful, because "TerritoryID" in the Territory table matches "SalesTerritoryID" in the table Customers. So, we'll have to specify different columns names to merge on for the two tables.
Can you write Python code for it?

Ans3 = pd.merge(Customers[["CustomerID","FirstName","LastName","SalesTerritoryID"]],
                Territory[["TerritoryID","Name"]],
                how='left',
                left_on='SalesTerritoryID', right_on='TerritoryID', )
Ans3


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
"Give me a list of the customers we have in North Carolina, and tell me how many there are."
Can you write Python code for it?

# In-class exercise! :)

Customers[Customers.StateName=="North Carolina"].CustomerID.count()


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
"Show a list of customers, and the total amount of money they have spent with AdventureWorks. I want the highest spenders to appear first!"
Can you write Python code for it?

# In-class exercise! :)

pd.merge(Customers[["FirstName","LastName","CustomerID"]],Orders[["CustomerID","TotalItemPrice"]], how="inner", on="CustomerID").groupby(["CustomerID","FirstName","LastName"]).sum()


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
Advanced Tables
Why are databases so complex?

Data stored in a database may be split into multiple tables, each containing multiple columns. A column stores a single attribute of the data; a table stores a collection of related attributes.
The database also keeps track of the relationships between different tables.

Databases are designed to minimize redundancy and maintain data integrity, particularly when data is added, changed, or deleted.

Consistency when updating: no duplicate places for information https://en.wikipedia.org/wiki/Database_normalization
Performance https://en.wikipedia.org/wiki/Star_schema

Side note: you may also have to think about isolation level when working with a database where someone may be updating data as you're trying to read it. The isolation level determines the database read behavior in this situation. See https://en.wikipedia.org/wiki/Isolation_(database_systems).

Working with multiple tables

Two tables can be joined at a time. 'Join' is a binary operator. See https://en.wikipedia.org/wiki/Join_(SQL).
Tables must have key values that can be matched. Usually one table has a primary key and the other table has a foreign key.

Pandas

Pandas allows "merge", "join", and "concatenate" operations. See http://pandas.pydata.org/pandas-docs/version/0.18.1/merging.html#merge-join-and-concatenate for additional reading.
Pandas also allows reshaping and pivoting data tables, see http://pandas.pydata.org/pandas-docs/version/0.18.1/reshaping.html.

In this class, we will cover table joining, merging and concatenation. We will also go over using some of the time-series handling capabilities in Pandas.
Can you write Python code for it?


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
Another side note:
Can you write Python code for it?

help(pd.DataFrame.combine_first)

help(pd.DataFrame.update)

Customers


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
Concatenating tables in Pandas
To introduce join operations, we will be working with the AdventureWorks dataset, a standard dataset from Microsoft SLQ Server for learing to work with databases. It contains data for the fictitious bicycle manufacturer (Adventure Works Cycles).
Let's starts by importing some tables from AdventureWorks in /home/data/AdventureWorks. These tables contain data on AdventureWorks employees, sales territories, customers, and orders placed by the customers.
Can you write Python code for it?

Employees = pd.read_excel('/home/data/AdventureWorks/Employees.xls')
Territory = pd.read_excel('/home/data/AdventureWorks/SalesTerritory.xls')
Customers = pd.read_excel('/home/data/AdventureWorks/Customers.xls')
Orders = pd.read_excel('/home/data/AdventureWorks/ItemsOrdered.xls')


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
Joining and merging tables in Pandas
Join and merge are powerful tools for working with multiple tables. We will use them to answer some questions about the
AdventureWorks dataset that you might encounter in real-life situations.
Join does fast table joining on a shared index.
Merge does the same thing, but gives you the option to specify columns to join on.
The idea of joining on a column will become clearer with some examples.
Example 1. "I want a list of all employees, and if any are salespeople then show me the details about their sales territory"
From AdventureWorks, we have a table "Employees" that gives a lot of information about AdventureWorks employees, like 'EmployeeID', 'ManagerID', 'TerritoryID', 'Title', 'FirstName','MiddleName', 'LastName', 'Suffix', 'JobTitle', 'NationalIDNumber', 'BirthDate', 'MaritalStatus', 'Gender', 'HireDate', 'SalariedFlag', 'VacationHours', 'SickLeaveHours', 'PhoneNumber', 'PhoneNumberType', 'EmailAddress', 'AddressLine1', 'AddressLine2', 'City', 'StateProvinceName', 'PostalCode', 'CountryName'. \
Since we're just being asked for a list of employees, we'll give the EmployeeID and their first, middle, and last names, and their role in the company (since additional information is requested for salespeople only). Then, for the salespeople, we must attach information about their sales territories, which is contained in the Territories table.
Notice that the Employees table has a column 'TerritoryID', which corresponds to the primary key in the 'Territory' table (in 'Territory', each territory has a unique 'TerritoryID'). We'll do a join on TerritoryID.
Can you write Python code for it?

help(pd.merge)

Ans = pd.merge(Employees.loc[:,["EmployeeID","FirstName","MiddleName","LastName","JobTitle","TerritoryID"]],
               Territory,
               how='left', on='TerritoryID')
Ans.head()

# Overachiever answer:
Ans['EmployeeName'] = Ans[["FirstName","MiddleName","LastName"]].apply(lambda x: x.LastName+", "+x.FirstName+" "+str(x.MiddleName), axis=1)
Ans = Ans[['EmployeeName', 'EmployeeID', 'JobTitle', 'TerritoryID', 'Name', 'CountryCode', 'Region', 'SalesYTD', 'SalesLastYear']]
Ans


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
Let's construct a slightly artificial example. Suppose that AdventureWorks was formed by merging two companies, AdventuresUSA which operated in the US and AdventuresWorld, which operated in other countries. Now we want information on their combined sales territories.
The Pandas "concat" function is good for stacking tables on top of each other. We will use it to combine the AdventuresUSA and AdventuresWorld territories data tables.
Can you write Python code for it?

help(pd.concat)

# constructing the territory tables... as noted, this is an artificial example
TerritoryUSA = Territory[Territory.CountryCode=='US']; TerritoryUSA['RepID'] = np.random.randint(1,1000,5)
TerritoryWorld = Territory[Territory.CountryCode!='US']

TerritoryUSA

TerritoryWorld

# we'll concatenate the databases, but keep separate keys so that we can keep track of which entries came from AdventuresUSA and
# which from AdventuresWorld.
# We'll use "join='inner'" to only keep colunms that are common to both tables;
# that is, we will drop the no-longer needed RepID in AdventuresUSA.
Territory2 = pd.concat([TerritoryUSA, TerritoryWorld], keys=['usa', 'world'], join='inner')

Territory2


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig-Copy1.ipynb".
The first task is:
Let's take a look at the data we'll be working with:
Can you write Python code for it?

Employees.head()

Territory.head()

Customers.head()

Orders.head()


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
"For each of the items ordered, show the total price (sometimes they ordered more than 1 item)"
Can you write Python code for it?

# We'll use the Orders table for this! In-class exercise :)

Orders["TotalItemPrice"] = Orders.Quantity * Orders.Price
Orders.head()

Orders.groupby(["CustomerID","OrderDate"]).TotalItemPrice.sum()


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
"Give me a list of our customers, and also tell me which sales territory they fall in."
This looks like another question for "merge"! We have a list of customers with their addresses, and we have a list of territories, but they are in separate tables.
Let's recover a list of customer names and IDs, together with corresponding sales territory names.
This time, we have to be careful, because "TerritoryID" in the Territory table matches "SalesTerritoryID" in the table Customers. So, we'll have to specify different columns names to merge on for the two tables.
Can you write Python code for it?


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
"Give me a list of the customers we have in North Carolina, and tell me how many there are."
Can you write Python code for it?

# In-class exercise! :)

ListNC = Customers[Customers.StateName=='North Carolina']
ListNC

ListNC.CustomerID.count()

Customers = Customers.set_index("StateName")
Customers.loc['North Carolina']


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
"Show a list of customers, and the total amount of money they have spent with AdventureWorks. I want the highest spenders to appear first!"
Can you write Python code for it?

# In-class exercise! :)

Ans_1 = pd.merge(Customers[["CustomerID","FirstName","LastName"]], Orders[["CustomerID","TotalItemPrice"]], how="inner", on="CustomerID" )

Ans_1

pd.DataFrame(Ans_1.groupby("CustomerID").TotalItemPrice.sum())

Ans_1a = Ans_1[["FirstName","LastName","CustomerID"]].set_index("CustomerID")
Ans_1b = Ans_1a[Ans_1a.duplicated()==False]

Ans_2 = pd.DataFrame(Ans_1.groupby("CustomerID").TotalItemPrice.sum()).join(Ans_1b)

Ans_2.sort_values("TotalItemPrice", ascending=False)

Ans_2.sort_values(ascending=False)

Territory.apply(max, axis=0)


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
Another side note:
Can you write Python code for it?


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
Concatenating tables in Pandas
To introduce join operations, we will be working with the AdventureWorks dataset, a standard dataset from Microsoft SLQ Server for learing to work with databases. It contains data for the fictitious bicycle manufacturer (Adventure Works Cycles).
Let's starts by importing some tables from AdventureWorks in /home/data/AdventureWorks. These tables contain data on AdventureWorks employees, sales territories, customers, and orders placed by the customers.
Can you write Python code for it?

    print(x)

Employees = pd.read_excel('/home/data/AdventureWorks/Employees.xls')
Territory = pd.read_excel('/home/data/AdventureWorks/SalesTerritory.xls')
Customers = pd.read_excel('/home/data/AdventureWorks/Customers.xls')
Orders = pd.read_excel('/home/data/AdventureWorks/ItemsOrdered.xls')


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
Joining and merging tables in Pandas
Join and merge are powerful tools for working with multiple tables. We will use them to answer some questions about the
AdventureWorks dataset that you might encounter in real-life situations.
Join does fast table joining on a shared index.
Merge does the same thing, but gives you the option to specify columns to join on.
The idea of joining on a column will become clearer with some examples.
Example 1. "I want a list of all employees, and if any are salespeople then show me the details about their sales territory"
From AdventureWorks, we have a table "Employees" that gives a lot of information about AdventureWorks employees, like 'EmployeeID', 'ManagerID', 'TerritoryID', 'Title', 'FirstName','MiddleName', 'LastName', 'Suffix', 'JobTitle', 'NationalIDNumber', 'BirthDate', 'MaritalStatus', 'Gender', 'HireDate', 'SalariedFlag', 'VacationHours', 'SickLeaveHours', 'PhoneNumber', 'PhoneNumberType', 'EmailAddress', 'AddressLine1', 'AddressLine2', 'City', 'StateProvinceName', 'PostalCode', 'CountryName'. \
Since we're just being asked for a list of employees, we'll give the EmployeeID and their first, middle, and last names, and their role in the company (since additional information is requested for salespeople only). Then, for the salespeople, we must attach information about their sales territories, which is contained in the Territories table.
Notice that the Employees table has a column 'TerritoryID', which corresponds to the primary key in the 'Territory' table (in 'Territory', each territory has a unique 'TerritoryID'). We'll do a join on TerritoryID.
Can you write Python code for it?

help(pd.merge)

Ans = pd.merge(Employees.loc[:,["EmployeeID","FirstName","MiddleName","LastName","JobTitle","TerritoryID"]],
               Territory,
               how='left', on='TerritoryID')
Ans.head()

X = Ans[["FirstName","MiddleName","LastName"]]

Ans['EmployeeName'] = X.apply(lambda x: x.LastName+", "+x.FirstName+" "+str(x.MiddleName), axis=1)

Ans

# Overachiever answer:
Ans['EmployeeName'] = Ans[["FirstName","MiddleName","LastName"]].apply(lambda x: x.LastName+", "+x.FirstName+" "+str(x.MiddleName), axis=1)
Ans = Ans[['EmployeeName', 'EmployeeID', 'JobTitle', 'TerritoryID', 'Name', 'CountryCode', 'Region', 'SalesYTD', 'SalesLastYear']]
Ans


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
Let's construct a slightly artificial example. Suppose that AdventureWorks was formed by merging two companies, AdventuresUSA which operated in the US and AdventuresWorld, which operated in other countries. Now we want information on their combined sales territories.
The Pandas "concat" function is good for stacking tables on top of each other. We will use it to combine the AdventuresUSA and AdventuresWorld territories data tables.
Can you write Python code for it?


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
Let's take a look at the data we'll be working with:
Can you write Python code for it?


I am working on the file "05-Operating-with-Multiple-Tables/AdvancedTables_orig.ipynb".
The first task is:
Pandas "append" behaves just like "concat" with axis=0 and join='outer' (i.e., keep all column names). Missing values are set to NaN.
Can you write Python code for it?

help(pd.DataFrame.append)

Territory3 = TerritoryUSA.append(TerritoryWorld)

Territory3


I am working on the file "05-Operating-with-Multiple-Tables/HW05/CheckHomework05.ipynb".
The first task is:
Now, import your solutions from hw5_answers.py. The following code looks a bit redundant. However, we do this to allow reloading the hw5_answers.py in case you made some changes. Normally, Python assumes that modules don't change and therefore does not try to import them again.
Can you write Python code for it?

import hw5_answers
reload(hw5_answers)
from hw5_answers import *


I am working on the file "05-Operating-with-Multiple-Tables/HW05/CheckHomework05.ipynb".
The first task is:
Problem 1
Write a function called get_manager that takes as its one argument the Pandas DataFrame "Employees" and returns a DataFrame containing list of all employees (EmployeeID, first name, middle name, last name), and their manager's first and last name. The columns in the output DataFrame should be: EmployeeID, FirstName, MiddleName, LastName, ManagerFirstName, ManagerLastName.
Can you write Python code for it?

df1 = get_manager(Employees)
print "Shape of resulting table: ", df1.shape
print "Columns: ", ', '.join(df1.columns)
df1.head()


I am working on the file "05-Operating-with-Multiple-Tables/HW05/CheckHomework05.ipynb".
The first task is:
Shape of resulting table:  (32, 4)
Columns:  FirstName, LastName, Item, TotalSpent

|FirstName | LastName | Item | TotalSpent
----------|----------|------|-----------
0 | Anthony | Sanchez   | Umbrella  | 4.5
1 | Conrad  | Giles | Ski Poles | 51.0
2 | Conrad  | Giles | Tent  | 88.0
3 | Donald  | Davids    | Lawnchair | 128.0
4 | Elroy   | Keller    | Inflatable Mattress   | 38.0
Problem 3
Write a function called get_order_location that takes three arguments: "Orders", "Customers", and "Territory", and returns a DataFrame containing the following columns:  "CustomerID", "Name", and "TotalItems", that gives, for each order, the  CustomerID, the name of the territory where the order was placed, and the total number of items ordered (yes, 2 ski poles counts as 2 items).
Can you write Python code for it?

df3 = get_order_location(Orders, Customers, Territory)
print "Shape of resulting table: ", df3.shape
print "Columns: ", ', '.join(df3.columns)
df3.head()


I am working on the file "05-Operating-with-Multiple-Tables/HW05/CheckHomework05.ipynb".
The first task is:
The Employees, Territory, Customers, and Orders tables are the same as those we used in class.
Can you write Python code for it?


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling.ipynb".
The first task is:

2. Document by Topic Distribution
One thing we may want to do with the output is find the most representative texts for each topic. A simple way to do this (but not memory efficient), is to merge the topic distribution back into the Pandas dataframe.
First get the topic distribution array.
Can you write Python code for it?

topic_dist = lda.transform(tf)
topic_dist


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling.ipynb".
The first task is:

3. Words Aligned with each Topic
Following DiMaggio et al., we can calculate the total number of words aligned with each topic, and compare by author gender.
Can you write Python code for it?

#first create word count column

df_w_topics['word_count'] = df_w_topics['text'].apply(lambda x: len(str(x).split()))
df_w_topics['word_count']

#multiple topic weight by word count

df_w_topics['0_wc'] = df_w_topics[0] * df_w_topics['word_count']
df_w_topics['0_wc']

#create a for loop to do this for every topic

topic_columns = range(0, n_topics)
col_list = []
for num in topic_columns:
    col = "%d_wc" % num
    col_list.append(col)
    #Solution
    df_w_topics[col] = df_w_topics[num] * df_w_topics['word_count']

df_w_topics

#EX: What is the total number of words aligned with each topic, by author gender?
#EX: What is the proportion of total words aligned with each topic, by author gender?


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling.ipynb".
The first task is:
Merge back in with the original dataframe.
Can you write Python code for it?

topic_dist_df = pandas.DataFrame(topic_dist)
df_w_topics = topic_dist_df.join(df_lit)
df_w_topics


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling.ipynb".
The first task is:
Question: Why might we want to do one calculation over the other? Take average topic weight per documents versus the average number of words aligned with each topic?
This brings us to...

4. Topic Prevalence
Can you write Python code for it?

###EX:
#       Find the most prevalent topic in the corpus.
#       Find the least prevalent topic in the corpus.
#       Hint: How do we define prevalence? What are different ways of measuring this,
#              and the benefits/drawbacks of each?


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling_ExerciseSolutions.ipynb".
The first task is:

2. Document by Topic Distribution
One thing we may want to do with the output is find the most representative texts for each topic. A simple way to do this (but not memory efficient), is to merge the topic distribution back into the Pandas dataframe.
First get the topic distribution array.
Can you write Python code for it?


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling_ExerciseSolutions.ipynb".
The first task is:
Now we can sort the dataframe for the topic of interest, and view the top documents for the topics.
Below we sort the documents first by Topic 0 (looking at the top words for this topic I think it's about family, health, and domestic activities), and next by Topic 1 (again looking at the top words I think this topic is about children playing outside in nature). These topics may be a family/nature split?
Look at the titles for the two different topics. Look at the gender of the author. Hypotheses?
Can you write Python code for it?

print(df_w_topics[['title', 'author gender', 0]].sort_values(by=[0], ascending=False))

print(df_w_topics[['title', 'author gender', 1]].sort_values(by=[1], ascending=False))

#EX: What is the average topic weight by author gender, for each topic?
### Grapth these results
#Hint: You can use the python 'range' function and a for-loop

grouped_mean=df_w_topics.groupby('author gender').mean()
grouped_mean[[0,1,2,3]].plot(kind='bar')
plt.show()


I am working on the file "05-TextExploration/00-IntroductionToTopicModeling_ExerciseSolutions.ipynb".
The first task is:
Question: Why might we want to do one calculation over the other? Take average topic weight per documents versus the average number of words aligned with each topic?
This brings us to...

4. Topic Prevalence
Can you write Python code for it?

###EX:
#       Find the most prevalent topic in the corpus.
#       Find the least prevalent topic in the corpus.
#       Hint: How do we define prevalence? What are different ways of measuring this,
#              and the benefits/drawbacks of each?

for e in col_list:
    print(e)
    print(df_w_topics[e].sum()/df_w_topics['word_count'].sum())

for e in topic_columns:
    print(e)
    print(df_w_topics[e].mean())


I am working on the file "05-TextExploration/01-VectorSpaceModel.ipynb".
The first task is:

3. Clustering texts based on distance
Clustering texts into discrete groups of similar texts is often a useful exploratory step. For example, a researcher may be wondering if certain textual features partition a collection of texts by author or by genre. Pairwise distances alone do not produce any kind of classification. To put a set of distance measurements to work in classification requires additional assumptions, such as a definition of a group or cluster.
The ideas underlying the transition from distances to clusters are, for the most part, common sense. Any clustering of texts should result in texts that are closer to each other (in the distance matrix) residing in the same cluster. There are many ways of satisfying this requirement; there no unique clustering based on distances that is the “best”. One strategy for clustering in circulation is called Ward’s method. Rather than producing a single clustering, Ward’s method produces a hierarchy of clusterings, as we will see in a moment. All that Ward’s method requires is a set of pairwise distance measurements–such as those we calculated a moment ago. Ward’s method produces a hierarchical clustering of texts via the following procedure:

Start with each text in its own cluster
Until only a single cluster remains,
Find the closest clusters and merge them. The distance between two clusters is the change in the sum of squared distances when they are merged.

Return a tree containing a record of cluster-merges.

The function scipy.cluster.hierarchy.ward performs this algorithm and returns a tree of cluster-merges. The hierarchy of clusters can be visualized using scipy.cluster.hierarchy.dendrogram.
Can you write Python code for it?

from scipy.cluster.hierarchy import ward, dendrogram
linkage_matrix = ward(euc_dist)

dendrogram(linkage_matrix, orientation="right", labels=names)

plt.tight_layout()  # fixes margins

plt.show()


I am working on the file "05-TextExploration/01-VectorSpaceModel.ipynb".
The first task is:

1. Comparing texts
Arranging our texts in a document-term matrix make available a range of exploratory procedures. For example, calculating a measure of similarity between texts becomes simple. Since each row of the document-term matrix is a sequence of a novel’s word frequencies, it is possible to put mathematical notions of similarity (or distance) between sequences of numbers in service of calculating the similarity (or distance) between any two novels. One frequently used measure of distance between vectors (a measure easily converted into a measure of similarity) is Euclidean distance. The Euclidean distance between two vectors in the plane should be familiar from geometry, as it is the length of the hypotenuse that joins the two vectors. For instance, consider the Euclidean distance between the vectors  \begin{align}
\overrightarrow{x}=(1,3) \space \space and\space\space\overrightarrow{y}=(4,2)  \end{align}
the Euclidian distance can be calculated as follows:
\begin{align}
\sqrt{(1-4)^2 + (3-2)^2} = \sqrt{10}
\end{align}

Note
Measures of distance can be converted into measures of similarity. If your measures of distance are all between zero and one, then a measure of similarity could be one minus the distance. (The inverse of the distance would also serve as a measure of similarity.)

Distance between two vectors:

Note
More generally, given two vectors \begin{align} \overrightarrow{x} \space \space and\space\space\overrightarrow{y}\end{align}
in p-dimensional space, the Euclidean distance between the two vectors is given by
\begin{align}
||\overrightarrow{x} −\overrightarrow{y}||=\sqrt{\sum_{i=1}^P (x_i−y_i)^2}
\end{align}

This concept of distance is not restricted to two dimensions. For example, it is not difficult to imagine the figure above translated into three dimensions. We can also persuade ourselves that the measure of distance extends to an arbitrary number of dimensions; for any two matched components in a pair of vectors (such as x2 and y2),...
Can you write Python code for it?

from sklearn.metrics.pairwise import euclidean_distances
euc_dist = euclidean_distances(dtm)
print(filenames[1])
print(filenames[2])
print("\nDistance between Austen and Machiavelli:")
# the distance between Austen and Machiavelli
print(euc_dist[1, 2])

# which is greater than the distance between *Austen* and *Alcott* (index 0)
print("\nDistance between Austen and Machiavelli is greater than the distance between Austen and Alcott:")
euc_dist[1, 2] > euc_dist[0, 1]


I am working on the file "05-TextExploration/01-VectorSpaceModel.ipynb".
The first task is:

Exercise:

Find the Euclidian distance and cosine distance for the 5 sentences below. Do the distance measures make sense?
Visualize the potential clusters using a dendrogram. Do the clusters make sense?
How might we make the clusters better?
Can you write Python code for it?

text0 = 'I like to eat broccoli and bananas.'
text1 = 'I ate a banana and spinach smoothie for breakfast.'
text2 = 'Chinchillas and kittens are cute.'
text3 = 'My sister adopted a kitten yesterday.'
text4 = 'Look at this cute hamster munching on a piece of broccoli.'

text_list = [text0, text1, text2, text3, text4]
#create vector for text "names"
names = ['eat', 'smoothie', 'chinchillas', 'adopted', 'munching']


I am working on the file "05-TextExploration/01-VectorSpaceModel.ipynb".
The first task is:

4. K-Means Clustering
From the dendrogram above, we might expect these four novels to have two clusters: Austen and Alcott, and Machiavelli and Marx.
Let's see if this is the case using k-means clustering, which clusters on Euclidean distance.
Can you write Python code for it?

from sklearn.cluster import KMeans

km = KMeans(n_clusters=2, random_state=0)

clusters = km.fit(dtm)
clusters.labels_

list(zip(filenames, clusters.labels_))

print("Top terms per cluster:")
order_centroids = clusters.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(2):
    print("Cluster %d:" % i,)
    for ind in order_centroids[i, :20]:
        print(' %s' % terms[ind],)
    print()


I am working on the file "05-TextExploration/01-VectorSpaceModel.ipynb".
The first task is:

2. Visualizing distances
It is often desirable to visualize the pairwise distances between our texts. A general approach to visualizing distances is to assign a point in a plane to each text, making sure that the distance between points is proportional to the pairwise distances we calculated. This kind of visualization is common enough that it has a name, “multidimensional scaling” (MDS) and family of functions in scikit-learn.
Can you write Python code for it?

import os  # for os.path.basename
import matplotlib.pyplot as plt
from sklearn.manifold import MDS

# two components as we're plotting points in a two-dimensional plane
# "precomputed" because we provide a distance matrix
# we will also specify `random_state` so the plot is reproducible.

mds = MDS(n_components=2, dissimilarity="precomputed", random_state=1)

pos = mds.fit_transform(euc_dist)  # shape (n_components, n_samples)
xs, ys = pos[:, 0], pos[:, 1]

# short versions of filenames:
# convert 'data/austen-brontë/Austen_Emma.txt' to 'Austen_Emma'
names = [os.path.basename(fn).replace('.txt', '') for fn in filenames]

for x, y, name in zip(xs, ys, names):
    plt.scatter(x, y)
    plt.text(x, y, name)

plt.show()


I am working on the file "05-TextExploration/01-VectorSpaceModel.ipynb".
The first task is:
And if we want to use a measure of distance that takes into consideration the length of the novels (an excellent idea), we can calculate the cosine similarity by importing sklearn.metrics.pairwise.cosine_similarity and use it in place of euclidean_distances.
Cosine similarity measure the angle between two vectors:

Question: How does length factor into these two equations?
Keep in mind that cosine similarity is a measure of similarity (rather than distance) that ranges between 0 and 1 (as it is the cosine of the angle between the two vectors). In order to get a measure of distance (or dissimilarity), we need to “flip” the measure so that a larger angle receives a larger value. The distance measure derived from cosine similarity is therefore one minus the cosine similarity between two vectors.
Can you write Python code for it?

from sklearn.metrics.pairwise import cosine_similarity

cos_dist = 1 - cosine_similarity(dtm)

np.round(cos_dist, 2)

##EX:
## 1. Print the cosine distance between Austen and Machiavelli
## 2. Is this distance greater or less than the distance between Austen and Alcott?


I am working on the file "05-TextExploration/01-VectorSpaceModel_ExerciseSolutions.ipynb".
The first task is:

3. Clustering texts based on distance
Clustering texts into discrete groups of similar texts is often a useful exploratory step. For example, a researcher may be wondering if certain textual features partition a collection of texts by author or by genre. Pairwise distances alone do not produce any kind of classification. To put a set of distance measurements to work in classification requires additional assumptions, such as a definition of a group or cluster.
The ideas underlying the transition from distances to clusters are, for the most part, common sense. Any clustering of texts should result in texts that are closer to each other (in the distance matrix) residing in the same cluster. There are many ways of satisfying this requirement; there no unique clustering based on distances that is the “best”. One strategy for clustering in circulation is called Ward’s method. Rather than producing a single clustering, Ward’s method produces a hierarchy of clusterings, as we will see in a moment. All that Ward’s method requires is a set of pairwise distance measurements–such as those we calculated a moment ago. Ward’s method produces a hierarchical clustering of texts via the following procedure:

Start with each text in its own cluster
Until only a single cluster remains,
Find the closest clusters and merge them. The distance between two clusters is the change in the sum of squared distances when they are merged.

Return a tree containing a record of cluster-merges.

The function scipy.cluster.hierarchy.ward performs this algorithm and returns a tree of cluster-merges. The hierarchy of clusters can be visualized using scipy.cluster.hierarchy.dendrogram.
Can you write Python code for it?


I am working on the file "05-TextExploration/01-VectorSpaceModel_ExerciseSolutions.ipynb".
The first task is:

1. Comparing texts
Arranging our texts in a document-term matrix make available a range of exploratory procedures. For example, calculating a measure of similarity between texts becomes simple. Since each row of the document-term matrix is a sequence of a novel’s word frequencies, it is possible to put mathematical notions of similarity (or distance) between sequences of numbers in service of calculating the similarity (or distance) between any two novels. One frequently used measure of distance between vectors (a measure easily converted into a measure of similarity) is Euclidean distance. The Euclidean distance between two vectors in the plane should be familiar from geometry, as it is the length of the hypotenuse that joins the two vectors. For instance, consider the Euclidean distance between the vectors  \begin{align}
\overrightarrow{x}=(1,3) \space \space and\space\space\overrightarrow{y}=(4,2)  \end{align}
the Euclidian distance can be calculated as follows:
\begin{align}
\sqrt{(1-4)^2 + (3-2)^2} = \sqrt{10}
\end{align}

Note
Measures of distance can be converted into measures of similarity. If your measures of distance are all between zero and one, then a measure of similarity could be one minus the distance. (The inverse of the distance would also serve as a measure of similarity.)

Distance between two vectors:

Note
More generally, given two vectors \begin{align} \overrightarrow{x} \space \space and\space\space\overrightarrow{y}\end{align}
in p-dimensional space, the Euclidean distance between the two vectors is given by
\begin{align}
||\overrightarrow{x} −\overrightarrow{y}||=\sqrt{\sum_{i=1}^P (x_i−y_i)^2}
\end{align}

This concept of distance is not restricted to two dimensions. For example, it is not difficult to imagine the figure above translated into three dimensions. We can also persuade ourselves that the measure of distance extends to an arbitrary number of dimensions; for any two matched components in a pair of vectors (such as x2 and y2),...
Can you write Python code for it?


I am working on the file "05-TextExploration/01-VectorSpaceModel_ExerciseSolutions.ipynb".
The first task is:
And if we want to use a measure of distance that takes into consideration the length of the novels (an excellent idea), we can calculate the cosine similarity by importing sklearn.metrics.pairwise.cosine_similarity and use it in place of euclidean_distances.
Cosine similarity measure the angle between two vectors:

Question: How does length factor into these two equations?
Keep in mind that cosine similarity is a measure of similarity (rather than distance) that ranges between 0 and 1 (as it is the cosine of the angle between the two vectors). In order to get a measure of distance (or dissimilarity), we need to “flip” the measure so that a larger angle receives a larger value. The distance measure derived from cosine similarity is therefore one minus the cosine similarity between two vectors.
Can you write Python code for it?

from sklearn.metrics.pairwise import cosine_similarity

cos_dist = 1 - cosine_similarity(dtm)

np.round(cos_dist, 2)

##EX:
## 1. Print the cosine distance between Austen and Machiavelli
## 2. Is this distance greater or less than the distance between Austen and Alcott?

print(cos_dist[1, 2])

# which is greater than the distance between *Austen* and
# *Alcott* (index 0)
cos_dist[1, 2] > cos_dist[1, 0]


I am working on the file "05-basic-numerical-analysis.ipynb".
The first task is:
5.1.6. ndarray supports coordinatewise operations. Observe, in particular, that * does not result in matrix multiplication.
Can you write Python code for it?


I am working on the file "05-basic-numerical-analysis.ipynb".
The first task is:
5.2. Floating-Point Arithmetic
5.2.1. The real and complex number systems that we use in mathematics unfortunately cannot be stored in a computer. On a computer, we use a finite approximation known as the floating-point number system.
The floating-point number system in base $\beta$ with precision $p$, minimum exponent $e_\min$, and maximum exponent $e_\max$ is the discrete subset $\mathscr{F} = \mathscr{F}(\beta,p,e_\min,e_\max)$ consisting of 0, $\infty$, $-\infty$, NaN, and numbers of the form
$$\pm(d_0 + d_1 / \beta + \cdots + d_{p-1} / \beta^{p-1}) \beta^e,$$
where
- $d_0,\ldots,d_{p-1},e$ are integers,
- $1 \leq d_0 < \beta$,
- $0 \leq d_i < \beta$ for all $1 \leq i \leq p-1$, and
- $e_\min \leq e \leq e_\max$.
Given a floating-point number of the above form, we call the number $d_0 + d_1/\beta + \cdots + d_{p-1}/\beta^{p-1}$ the significand, or the mantissa. The number $e$ is called the exponent. The sign $\pm$ is called, naturally, the sign.
The condition $d_0 \geq 1$ makes the floating-point representation of a number unique. For this reason, a floating-point number with $d_0 \geq 1$ is called normalized.
5.2.2. How do we compute the floating-point representation $\operatorname{fl}(x)$ of a real number $x$?
It makes sense to define $\operatorname{fl}(x)$ as the floating-point number nearest to $x$. To approximate $\operatorname{fl}(x)$, we observe that $x$ admits a unique infinite-sum representation
$$x = \pm (d_0 + d_1/\beta + \cdots  + d_{p-1}/\beta_{p-1} + d_p /\beta_p + \cdots) \beta^e,$$
provided that $d_0 \geq 1$ and $d_i \neq \beta - 1$ for at least one $i \geq 1$. The infinite-sum representation of $x$ shows that $\operatorname{fl}(x)$ must lie between
$$\pm (d_0 + d_1/\beta + \cdots  + d_{p-1}/\beta_{p-1}) \beta^e - \beta_{p-1}\beta^e$$
and
$$\pm (d_0 + d_1/\beta + \cdots  + d_{p-1}/\beta_{p-1}) \beta^e + \beta_{p-1}\beta^e$$
It follows that
$$\vert\operatorname{fl}(x) - x\vert \leq 2\beta^{-(p-1)}\beta^e.$$
To obtain an error bound independent of the choice of $x$, we normalize the...
Can you write Python code for it?


I am working on the file "05-basic-numerical-analysis.ipynb".
The first task is:
Operations with mismatching dimensions sometimes result in legitimate operation, via broadcasting. For example, scalar multiplication works fine:
Can you write Python code for it?


I am working on the file "05-basic-numerical-analysis.ipynb".
The first task is:
Row-wise operations and column-wise operations are also possible:
Can you write Python code for it?

x = np.array([5, -1, 3])
y = np.arange(9).reshape((3, 3))

print(y)
print()

print(x.reshape((3, 1)) + y)
print()

print(x.reshape((1, 3)) + y)


I am working on the file "05-basic-numerical-analysis.ipynb".
The first task is:
Similarly, a matrix can be entered by first typing out a one-dimensional array and then putting the array through reshape:
Can you write Python code for it?

print(b)
print('shape of b: {}'.format(b.shape))
print()

print(b.reshape((2,2)))
print('shape of b.reshape((2,2)): {}'.format(b.transpose().reshape((2,2))))
print()

print(b.reshape((4,1)))
print('shape of b.reshape((4,1)): {}'.format(b.transpose().reshape((4,1))))


I am working on the file "05-bonos.ipynb".
The first task is:
Ejemplo.-- Realice un análisis de sensibilidad si el valor del bono es 666.7 y se condieran tasas del 3, 4 y 5%?
Can you write Python code for it?

cf.bond(face_value=1000, coupon_value=20, num_coupons=28, value=666.7, ytm=[3, 4, 5])


I am working on the file "05-bonos.ipynb".
The first task is:
Ejemplo.-- ¿Cuál es el valor del bono anterior si se considerán valores entre el 3% y el 5% para la tasa de interés?
Can you write Python code for it?

cf.bond(face_value=1000, coupon_value=20, num_coupons=28, ytm=[3, 4, 5])


I am working on the file "05-bonos.ipynb".
The first task is:
bond(face_value=None, coupon_rate=None, coupon_value=None,
   num_coupons=None, value=None, ytm=None)
Con:

face_value -- es el valor pagado por el bono a la fecha de expiración.
coupon_rate -- es la tasa de interés pagada por el bono como un porcentaje del face_value. Se usa para calcular el coupon_value cuando dicho parámetro no es suministrado.
coupon_value -- es el interés pagado por el bono. Si este valor no es suministrado es calculado como coupon_rate * face_value.
num_coupons -- número de pagos de interés antes de la fecha de expiración.
value -- es el valor presente del bono.
ytm-- (yield-to-maturity) es la tasa de interés del bono (períodica).

Esta función retorna value cuando ytm es especificado; y ytm cuando value es especificado.
Ejemplo.-- Se tiene un bono que pagará $ 1000 en 14 años y cupones semestrales a una tasa del 2%. ¿Cuál es el valor actual del bono si la tasa atractiva mínima es del 4%?
Can you write Python code for it?

cf.bond(face_value=1000, coupon_value=20, num_coupons=28, ytm=4)


I am working on the file "05-classes-oop.ipynb".
The first task is:
Another special function that is very useful is __repr__. This gives a representation of the class. In essence, if you ask Python to print a variable, it will print the string returned by the __repr__ function. We can use this to create a simple string representation of the polynomial:
Can you write Python code for it?

class Polynomial(object):
    """Representing a polynomial."""
    explanation = "I am a polynomial"

    def __init__(self, roots, leading_term):
        self.roots = roots
        self.leading_term = leading_term
        self.order = len(roots)

    def __repr__(self):
        string = str(self.leading_term)
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

    def explain_to(self, caller):
        print("Hello, {}. {}.".format(caller,self.explanation))
        print("My roots are {}.".format(self.roots))

p = Polynomial(p_roots, p_leading_term)
print(p)
q = Polynomial((1,1,0,-2), -1)
print(q)


I am working on the file "05-classes-oop.ipynb".
The first task is:
At the moment the class is not doing anything interesting. To do something interesting we need to store (and manipulate) relevant variables. The first thing to do is to add those variables when the instance is actually created. We do this by adding a special function (method) which changes how the variables of type Polynomial are created:
Can you write Python code for it?

class Polynomial(object):
    """Representing a polynomial."""
    explanation = "I am a polynomial"

    def __init__(self, roots, leading_term):
        self.roots = roots
        self.leading_term = leading_term
        self.order = len(roots)

    def explain_to(self, caller):
        print("Hello, {}. {}.".format(caller,self.explanation))
        print("My roots are {}.".format(self.roots))


I am working on the file "05-classes-oop.ipynb".
The first task is:
In fact, we can be a bit smarter than this. Note that the __init__ function of the Monomial class is identical to that of the Polynomial class, just with the leading_term set explicitly to 1. Rather than duplicating the code and modifying a single value, we can call the __init__ function of the Polynomial class directly. This is because the Monomial class is built on the Polynomial class, so knows about it. We regenerate the class, but only change the __init__ function:
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""
    explanation = "I am a monomial"

    def __init__(self, roots):
        Polynomial.__init__(self, roots, 1)

    def __repr__(self):
        string = ""
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

v = Monomial((2, -3))
v.explain_to("Fred")
print(v)


I am working on the file "05-classes-oop.ipynb".
The first task is:
The final special function we'll look at (although there are many more, many of which may be useful) is __mul__. This allows Python to multiply two variables together. With this we can take the product of two polynomials:
Can you write Python code for it?

class Polynomial(object):
    """Representing a polynomial."""
    explanation = "I am a polynomial"

    def __init__(self, roots, leading_term):
        self.roots = roots
        self.leading_term = leading_term
        self.order = len(roots)

    def __repr__(self):
        string = str(self.leading_term)
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

    def __mul__(self, other):
        roots = self.roots + other.roots
        leading_term = self.leading_term * other.leading_term
        return Polynomial(roots, leading_term)

    def explain_to(self, caller):
        print("Hello, {}. {}.".format(caller,self.explanation))
        print("My roots are {}.".format(self.roots))

p = Polynomial(p_roots, p_leading_term)
q = Polynomial((1,1,0,-2), -1)
r = p*q
print(r)


I am working on the file "05-classes-oop.ipynb".
The first task is:
The first line, p = Polynomial(), creates an instance of the class. That is, it creates a specific Polynomial. It is assigned to the variable named p. We can access class variables using the "dot" notation, so the string can be printed via p.explanation. The method that prints the class variable also uses the "dot" notation, hence p.explain(). The self variable in the definition of the function is the instance itself, p. This is passed through automatically thanks to the dot notation.
Note that we can change class variables in specific instances in the usual way (p.explanation = ... above). This only changes the variable for that instance. To check that, let us define two polynomials:
Can you write Python code for it?

p = Polynomial()
p.explanation = "Changed the string again"
q = Polynomial()
p.explanation = "Changed the string a third time"
p.explain()
q.explain()


I am working on the file "05-classes-oop.ipynb".
The first task is:
The order of the polynomial is given by the number of coefficients (minus one), which is given by len(p_normal)-1.
However, there are many other ways it could be written, which are useful in different contexts. For example, we are often interested in the roots of the polynomial, so would want to express it in the form
\begin{equation}
  p(x) = 2 (x - 1)(x - 2)(x + 3).
\end{equation}
This allows us to read off the roots directly. We could imagine representing this in Python using a container containing the roots, such as:
Can you write Python code for it?


I am working on the file "05-classes-oop.ipynb".
The first task is:
This __init__ function is called when a variable is created. There are a number of special class functions, each of which has two underscores before and after the name. So now we can create a variable that represents a specific polynomial by storing its roots and the leading term:
Can you write Python code for it?

p = Polynomial(p_roots, p_leading_term)
p.explain_to("Alice")
q = Polynomial((1,1,0,-2), -1)
q.explain_to("Bob")


I am working on the file "05-classes-oop.ipynb".
The first task is:
This has had no effect on the original Polynomial class and variables, which can be used as before:
Can you write Python code for it?

s = Polynomial((2, 3), 4)
s.explain_to("David")
print(s)


I am working on the file "05-classes-oop.ipynb".
The first task is:
Variables of the  Monomial class are also variables of the Polynomial class, so can use all the methods and functions from the Polynomial class automatically:
Can you write Python code for it?

m = Monomial((-1, 4, 9))
m.explain_to("Caroline")
print(m)


I am working on the file "05-classes-oop.ipynb".
The first task is:
We can of course make the methods take additional variables. We modify the class (note that we have to completely re-define it each time):
Can you write Python code for it?

class Polynomial(object):
    explanation = "I am a polynomial"

    def explain_to(self, caller):
        print("Hello, {}. {}.".format(caller,self.explanation))


I am working on the file "05-classes-oop.ipynb".
The first task is:
We have defined a class, which is a single object that will represent a polynomial. We use the keyword class in the same way that we use the keyword def when defining a function. The definition line ends with a colon, and all the code defining the object is indented by four spaces.
The name of the object - the general class, or type, of the thing that we're defining - is Polynomial. The convention is that class names start with capital letters, but this convention is frequently ignored.
The type of object that we are building on appears in brackets after the name of the object. The most basic thing, which is used most often, is the object type as here.
Class variables are defined in the usual way, but are only visible inside the class. Variables that are set outside of functions, such as explanation above, will be common to all class variables.
Functions are defined inside classes in the usual way (using the def keyword, indented by four additional spaces). They work in a special way: they are not called directly, but only when you have a member of the class. This is what the self keyword does: it takes the specific instance of the class and uses its data. Class functions are often called methods.
Let's see how this works on a specific example:
Can you write Python code for it?

p = Polynomial()
print(p.explanation)
p.explain()
p.explanation = "I change the string"
p.explain()


I am working on the file "05-classes-oop.ipynb".
The first task is:
We note that these functions, methods and variables may not be exactly right, as they are given for the general Polynomial class, not by the specific Monomial class. If we redefine these functions and variables inside the Monomial class, they will override those defined in the Polynomial class. We do not have to override all the functions and variables, just the parts we want to change:
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""
    explanation = "I am a monomial"

    def __init__(self, roots):
        self.roots = roots
        self.leading_term = 1
        self.order = len(roots)

    def __repr__(self):
        string = ""
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

m = Monomial((-1, 4, 9))
m.explain_to("Caroline")
print(m)


I am working on the file "05-classes-oop.ipynb".
The first task is:
We now have a simple class that can represent polynomials and multiply them together, whilst printing out a simple string form representing itself. This can obviously be extended to be much more useful.
Inheritance
As we can see above, building a complete class from scratch can be lengthy and tedious. If there is another class that does much of what we want, we can build on top of that. This is the idea behind inheritance.
In the case of the Polynomial we declared that it started from the object class in the first line defining the class: class Polynomial(object). But we can build on any class, by replacing object with something else. Here we will build on the Polynomial class that we've started with.
A monomial is a polynomial whose leading term is simply 1. A monomial is a polynomial, and could be represented as such. However, we could build a class that knows that the leading term is always 1: there may be cases where we can take advantage of this additional simplicity.
We build a new monomial class as follows:
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""

    def __init__(self, roots):
        self.roots = roots
        self.leading_term = 1
        self.order = len(roots)


I am working on the file "05-classes-oop.ipynb".
The first task is:
We see that the order of the polynomial is given by the number of roots (and hence by len(p_roots)). This form represents the same polynomial but requires two pieces of information (the roots and the leading coefficient).
The different forms are useful for different things. For example, if we want to add two polynomials the standard form makes it straightforward, but the factored form does not. Conversely, multiplying polynomials in the factored form is easy, whilst in the standard form it is not.
But the key point is that the object - the polynomial - is the same: the representation may appear different, but it's the object itself that we really care about. So we want to represent the object in code, and work with that object.
Classes
Python, and other languages that include object oriented concepts (which is most modern languages) allow you to define and manipulate your own objects. Here we will define a polynomial object step by step.
Can you write Python code for it?

class Polynomial(object):
    explanation = "I am a polynomial"

    def explain(self):
        print(self.explanation)


I am working on the file "05-classes-oop.ipynb".
The first task is:
We then use this, remembering that the self variable is passed through automatically:
Can you write Python code for it?


I am working on the file "05-classes-oop.ipynb".
The first task is:
combined with a single variable containing the leading term,
Can you write Python code for it?


I am working on the file "05-continuous-training.ipynb".
The first task is:
1. Create a Pub/Sub topic
Can you write Python code for it?

!gcloud pubsub topics create {PUBSUB_TOPIC}


I am working on the file "05-continuous-training.ipynb".
The first task is:
2. Deploy the Cloud Function
Can you write Python code for it?

ENV_VARS=f"""\
PROJECT={PROJECT},\
REGION={REGION},\
GCS_PIPELINE_FILE_LOCATION={GCS_PIPELINE_FILE_LOCATION}
"""

!echo {ENV_VARS}

!rm -r src/pipeline_triggering/.ipynb_checkpoints

!gcloud functions deploy {CLOUD_FUNCTION_NAME} \
    --region={REGION} \
    --trigger-topic={PUBSUB_TOPIC} \
    --runtime=python37 \
    --source=src/pipeline_triggering\
    --entry-point=trigger_pipeline\
    --stage-bucket={BUCKET}\
    --update-env-vars={ENV_VARS}

cloud_fn_url = f"https://console.cloud.google.com/functions/details/{REGION}/{CLOUD_FUNCTION_NAME}"
html = f'See the Cloud Function details here.'
IPython.display.display(IPython.display.HTML(html))


I am working on the file "05-continuous-training.ipynb".
The first task is:
3. Trigger the pipeline
Can you write Python code for it?

from google.cloud import pubsub

publish_client = pubsub.PublisherClient()
topic = f'projects/{PROJECT}/topics/{PUBSUB_TOPIC}'
data = {
    'num_epochs': 7,
    'learning_rate': 0.0015,
    'batch_size': 512,
    'hidden_units': '256,126'
}
message = json.dumps(data)

_ = publish_client.publish(topic, message.encode())


I am working on the file "05-continuous-training.ipynb".
The first task is:
Set configurations
Can you write Python code for it?

VERSION = 'v01'
DATASET_DISPLAY_NAME = 'chicago-taxi-tips'
MODEL_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier-{VERSION}'
PIPELINE_NAME = f'{MODEL_DISPLAY_NAME}-train-pipeline'

PIPELINES_STORE = f'gs://{BUCKET}/{DATASET_DISPLAY_NAME}/compiled_pipelines/'
GCS_PIPELINE_FILE_LOCATION = os.path.join(PIPELINES_STORE, f'{PIPELINE_NAME}.json')
PUBSUB_TOPIC = f'trigger-{PIPELINE_NAME}'
CLOUD_FUNCTION_NAME = f'trigger-{PIPELINE_NAME}-fn'

!gsutil ls {GCS_PIPELINE_FILE_LOCATION}


I am working on the file "05-continuous-training.ipynb".
The first task is:
Setup Google Cloud project
Can you write Python code for it?

PROJECT = '[your-project-id]' # Change to your project id.
REGION = 'us-central1' # Change to your region.
BUCKET =  '[your-bucket-name]' # Change to your bucket name.

if PROJECT == "" or PROJECT is None or PROJECT == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT = shell_output[0]

if BUCKET == "" or BUCKET is None or BUCKET == "[your-bucket-name]":
    # Get your bucket name to GCP projet id
    BUCKET = PROJECT

print("Project ID:", PROJECT)
print("Region:", REGION)
print("Bucket name:", BUCKET)


I am working on the file "05-continuous-training.ipynb".
The first task is:
Wait for a few seconds for the pipeline run to be submitted, then you can see the run in the Cloud Console
Can you write Python code for it?

from kfp.v2.google.client import AIPlatformClient

pipeline_client = AIPlatformClient(
    project_id=PROJECT, region=REGION)

job_display_name = pipeline_client.list_jobs()['pipelineJobs'][0]['displayName']
job_url = f"https://console.cloud.google.com/vertex-ai/locations/{REGION}/pipelines/runs/{job_display_name}"
html = f'See the Pipeline job here.'
IPython.display.display(IPython.display.HTML(html))


I am working on the file "05-donnees-spatiales.ipynb".
The first task is:
Géocodage
Le géocodage consiste à obtenir les points de référence géographique d'objets du monde réel. Un cas intéressant est celui des adresses physiques.
Il est possible de faire du géocodage à la main dans les outils cartographique publiques tels que Google ou OpenStreetMap. Il est aussi possible d'utiliser des bibliothèques Python comme geopandas pour faire du géocodage systématique. Le service nominatim d'OpenStreetMap permet le géocodage.
Autre exemple de geopandas: https://geopandas.org/geocoding.html
Can you write Python code for it?

geopandas.tools.geocode('2900 boulevard Edouard Montpetit, Montreal', provider='nominatim', user_agent="mon-application")


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Finding it difficult to pick the Winning Model - Why not Average the Results

After all we collectively make the right decisions, on average.
Can you write Python code for it?

from sklearn.ensemble import VotingClassifier

classifiers = [("Logistic Regression", model_1),
               ("Naive Bayes", model_2),
               ("Random Forest", model_3),
               ("Gradient Boosted", model_4),
               ("Neural Nets", model_5)]

classifiers

final_model = VotingClassifier(classifiers, n_jobs=-1)

# Unfortuately, have to run Fit Again on the ensembled model before using it
# Wish there was an option to not have to fit again
final_model.fit(X_train_vect, y_train)

# Drum Rolls - Accuracy on the final Model
print("Training Accuracy: {:.3f}".format(final_model.score(X_train_vect, y_train)))
print("Validation Accuracy: {:.3f}".format(final_model.score(X_valid_vect, y_valid)))


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Let's Update Kaggle Submission
Steps:

Load Test Dataset
Vectorize the Features (Review)
Predict the sentiment
Create the CSV file and update the submission
Can you write Python code for it?

# Read in the Test Dataset
# Note that it's missing the Sentiment Column.  That's what we need to Predict
#
test_df = pd.read_csv("data/test.tsv", sep="\t")
test_df.head()

# Vectorize the Review Text

X_test = test_df.review
X_test_vect = vect.transform(X_test)

y_test_pred = final_model.predict(X_test_vect)

df = pd.DataFrame({
    "document_id": test_df.document_id,
    "sentiment": y_test_pred
})

df.to_csv("data/ensemble_submission1.csv", index=False)


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Load Data
Can you write Python code for it?

train_df = pd.read_csv("data/train.tsv", sep="\t")

train_df.sample(10)


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Model 3 - Random Forest
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

model_3 = RandomForestClassifier(min_samples_leaf=3, n_estimators=25, n_jobs=-1)
model_3.fit(X_train_vect, y_train)

# Training Accuracy
print("Training Accuracy: {:.3f}".format(model_3.score(X_train_vect, y_train)))
print("Validation Accuracy: {:.3f}".format(model_3.score(X_valid_vect, y_valid)))


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Model 5 - Neural Networks (CPU Only)
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier

model_5 = MLPClassifier(hidden_layer_sizes=(32,), max_iter=100)
model_5.fit(X_train_vect, y_train)

# Training Accuracy
print("Training Accuracy: {:.3f}".format(model_5.score(X_train_vect, y_train)))
print("Validation Accuracy: {:.3f}".format(model_5.score(X_valid_vect, y_valid)))


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Neural Nets - Textbook Case of Overfitting.  Maybe the model is too powerful :-D
Model Tuning

All these models have tons of Parameters that could be tweaked to reduce overfitting
Can you write Python code for it?


I am working on the file "05-ensemble-models.ipynb".
The first task is:
Training process

Split the Overall Training examples into Training and Validation
Build the Models on Training Data
Score on Validation data
Choose the best model and submit to Kaggle

Caution:  If you do this enough times, you will be overfitting to the Validation data.  To avoid that it might be advisable to split into three ways like Train-Validation-Test and generate the final score on Test Data.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_valid, y_train, y_valid = train_test_split(train_df["review"], train_df["sentiment"], test_size=0.2)

print("Training Data: {}, Validation: {}".format(len(X_train), len(X_valid)))


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:

 Image components

Create a 2x2 grid of plots using `plt.subplots()`. For either the Hubble image or another RGB image of your choice from `skimage.data`, plot the true colour image and each RGB component on one of these subplots.

Can you write Python code for it?

# 1

my_image = data.coffee()

# Create 2x2 grid of subplots
fig, axes = plt.subplots(2, 2)

# Plot image and image components with appropriate colour maps.
axes[0, 0].imshow(my_image)
axes[0, 1].imshow(my_image[..., 0], cmap='Reds')
axes[1, 0].imshow(my_image[..., 1], cmap='Greens')
axes[1, 1].imshow(my_image[..., 2], cmap='Blues')


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:

 Moving around an image

Plot a small portion at one end of your chosen image. Then animate this plot so that it pans across to the other side of the image.

Can you write Python code for it?

# 1
fig, ax = plt.subplots()
y0 = 0
y_ext = 120

display = plt.imshow(galaxy_image[y0:y0+y_ext, 200:400])

def pan(i):
    y1 = y0 + i
    display.set_data(galaxy_image[y1:y1+y_ext, 200:400])
    return display

panimation = ani.FuncAnimation(fig, pan, range(galaxy_image.shape[0]-y_ext), interval=10)
plt.show()


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
As you can see, this allows us to increase the contrast of the image at the cost of discounting extreme values, or we can include a broader range of values but see less detail. Similar effects can also be achieved with the norm keyword, which allows you to set how imshow() scales values in order to map them to colours (linear or logarithmic scaling, for example).
Axes
You will notice in the above plots that the axes are labelled with the pixel coordinates of the image. You will also notice that the origin of the axes is in the top left corner rather than the bottom left. This is a convention in image-drawing, but can be changed if necessary by setting the origin keyword to 'lower' when calling imshow():
Can you write Python code for it?


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
Loading an image from a file
The image used in the examples above uses an image which is already supplied as an array by scikit-image. But what if we have been given an image file and we want to read it into Python?
There are many ways to do this, depending on the type of file. Typically in astronomy, images are stored in FITS format, which will be covered in detail later on. For now, we will return to the example of the Hubble image from earlier, which is stored in this repo in fig/galaxy.jpg. To load image data from a JPEG, we need the plt.imread() function. This takes a filename which points at an image file and loads it into Python as a NumPy array.
Can you write Python code for it?

# Load the Hubble image from fig/galaxy.jpg
galaxy_image = plt.imread('fig/galaxy.jpg')
# PLot the image with imshow()
plt.imshow(galaxy_image)


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
Once read in as an array, the image can be processed in the same ways as any other array. For instance, we can easily find the highest, lowest and mean values of the image, the type of the variables stored in the array, and the resolution of the image:
Can you write Python code for it?

# Output the image minimum, mean and maximum.
print('Image min:', moon.min(), '; Image mean:', moon.mean(), '; Image max: ', moon.max())

# Output the array dtype.
print('Data type:', moon.dtype)

# Output image size.
print('Image size:', moon.shape)


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
Rather than just being a 2D array with a shape equivalent to the image resolution, the array has an extra dimension of length 3. This is because the image has been split into red, blue and green components, each of which are stored in a slice of the array. When given an n x m x 3 array like this, imshow() interprets it as an RGB image and combines the layers into a single image.
However, if we wish to see the individual components they can be accessed and displayed by taking a slice of the array corresponding to the layer we wish to use.
Can you write Python code for it?

plt.imshow(galaxy_image[..., 0], cmap='Reds') # Plot the red layer of the image
plt.show()
plt.imshow(galaxy_image[..., 1], cmap='Greens') # Plot the green layer of the image
plt.show()
plt.imshow(galaxy_image[..., 2], cmap='Blues') # Plot the blue layer of the image
plt.show()


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
Slicing images
We saw above that an RGB image array can be sliced to access one colour component. But the array can also be sliced in one or both of the image dimensions to crop the image. For instance, the smaller galaxy at the bottom of the image above occupies the space between about 200 and 400 pixels in the x direction, and stretches from about 240 pixels to the edge of the image in the y direction. This information allows us to slice the array appropriately:
Can you write Python code for it?

# Crop the image in x and y directions but keep all three colour components.
cropped_galaxy = galaxy_image[240:, 200:400, :]
plt.imshow(cropped_galaxy)


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
The full list of available colour maps (for matplotlib 1.5) can be found here.

 Colour maps

As the images above demonstrate, the choice of colour map can make a significant difference to how your image appears, and is therefore extremely important. This is partly due to discrepancies between how quickly the colour map changes and how quickly the data changes, and partly due to the fact that [different people see colour differently](https://en.wikipedia.org/wiki/The_dress_%28viral_phenomenon%29).

In particular, matplotlib's default `'jet'` colour map is notoriously bad for displaying data. This is because it is not designed taking into account how the human eye percieves colour. This leads to some parts of the colour map appearing to change very slowly, while other parts of the colour map shift from one hue to another in a very short space. The practical effect of this is to both smooth some parts of the image, obscuring the data, and to create artificial features in the image where the data is smooth.

There is no single 'best' colour map - different colour maps display different kinds of image most clearly - but the `jet` map is almost never an appropriate choice for displaying any data. In general, colour maps which vary luminosity uniformly (such as the `'gray'` colour map above or the `'cubehelix'` colour map) tend to be better. Plots of various colour maps' luminosities can be found [here](http://matplotlib.org/users/colormaps.html).

For a good background on this topic and a description of a decent all-round colour map scheme, see [this paper](http://www.kennethmoreland.com/color-maps/ColorMapsExpanded.pdf).

 Load and plot an image

 Try loading and plotting some other image arrays from...
Can you write Python code for it?

# 1

# Load an image from skimage.data
my_image = data.coins()
# Print image shape and size
print(my_image.shape, my_image.size)
# Print data type and min&max  of array
print(my_image.dtype, my_image.min(), my_image.max())

# 2

# Display my image
plt.imshow(my_image, cmap='cubehelix')
plt.colorbar()


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
This can be a useful way to change how the image appears. For instance, if the exact values of the data are extremely important, little or no interpolation may be appropriate so the original values are easier to discern, whereas a high level of interpolation can be used if the smoothness of the image is more important than the actual numbers.
Note that that 'none' in the imshow() call above is NOT the same as None. None tells imshow() you are not passing it a variable for the interpolation keyword, so it uses the default, whereas 'none' explicitly tells it not to interpolate.
Animation
We have already seen animation of data points on basic plots in a previous lesson. Animating an image is no different in principle. To demonstrate this, we'll set up an animation that shows the Hubble image and then cycles through each of the RGB components. This task requires all the same parts as an animation of a line or scatter plot:

First, we'll need matplotlib.animation, a figure and an axes. Then we'll plot the initial image we want to display and return the plot object to a variable we can use for the animation.
Now we need to define the function that will adjust the image. This function, like the ones we used for line plots, needs to take as input an integer which counts the number of 'frames', adjust the displayed data and return the adjusted object.
Then we can define the animation object and plot it to see the finished product.
Can you write Python code for it?

import matplotlib.animation as ani
# We'll need this for displaying animations
%matplotlib nbagg

fig, ax = plt.subplots()
display = plt.imshow(galaxy_image)

titles = ['Red component', 'Green component', 'Blue component', 'Combined image']
cmaps = ['Reds_r', 'Greens_r', 'Blues_r']

def animate(i):
    try:
        display.set_data(galaxy_image[..., i])
        display.set_cmap(cmaps[i])
    except IndexError:
        display.set_data(galaxy_image)
    ax.set_title(titles[i])
    return display

myanim = ani.FuncAnimation(fig, animate, range(4), interval=1000)
plt.show()


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
Value limits
The default behaviour of imshow() in terms of colour mapping is that the colours cover the full range of the data so that the lower end (blue, in the plots above) represents the smallest value in the array, and the higher end (red) represents the greatest value.
This is fine if rest of the values are fairly evenly spaced between these extremes. However, if we have a very low minimum or very high maximum compared to the rest of the image, this default scaling is unhelpful. To deal with this problem, imshow() allows you to set the minimum and maximum values used for the scaling with the vmin and vmax keywords.
Can you write Python code for it?

plt.imshow(moon, cmap='gray', vmin=75, vmax=150)
plt.colorbar()


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
When plotting an image in this way, you will often need to know what actual values correspond to the colours. To find this out, we can draw a colour bar alongside the image which indicates the mapping of values to colours:
Can you write Python code for it?


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
You may notice that instead of using a colour map, this image has been plotted in full colour so it looks the same as the original image above. We can see why if we inspect the shape of the image array:
Can you write Python code for it?


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
You may notice that the default mapping of values to colours doesn't show the features of this image very well. Fortunately, matplotlib provides a large variety of colour maps which are suitable for various different purposes (more on this later). plt.imshow() has a cmap keyword argument which can be passed a string defining the desired colour map.
Can you write Python code for it?

# Display the image with a better colour map.
plt.imshow(moon, cmap='gray')
plt.colorbar()


I am working on the file "05-images-and-plotting-instructor.ipynb".
The first task is:
plt.subplots()
As we've already seen, multiple axes can be added to a single figure using plt.add_subplot(). There is also a function that allows you to define several axes and their arrangement at the same time as the figure, plt.subplots().
This function returns a tuple of two objects - the figure and an array of axes objects with the specified shape. Referencing the axes array allows things to be plotted on the individual subplots.
Can you write Python code for it?

# Make a grid of 1 x 3 plots and show the Hubble image on the right.
fig, ax = plt.subplots(1, 3)
ax[2].imshow(galaxy_image)
plt.show()


I am working on the file "05-intro-to-mpi.ipynb".
The first task is:
Blocking risks
- Send data larger than available network buffer (Blocking send)
- Lost data (or missing sender) leading to receiver hanging indefinitely (Blocking receive)
Can you write Python code for it?

%%writefile codes/openmpi/deadlock_send_recv.c
#include <mpi.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char* argv[])
{
  int my_rank;       /* rank of process     */
  int size;             /* number of processes */
  int source;        /* rank of sender      */
  int dest;          /* rank of receiver    */

  int tag=0;         /* tag for messages    */
  char message[100]; /* storage for message */
  MPI_Status status; /* return status for receive */

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  fprintf(stderr,"I am here!  ID = %d\n", my_rank);
  sprintf(message, "Greetings from process %d!", my_rank);

  if (my_rank == 0) {
    dest = 1;
    MPI_Recv(message, 100, MPI_CHAR, dest, tag, MPI_COMM_WORLD, &status);
    MPI_Send(message, strlen(message)+1, MPI_CHAR, dest, tag, MPI_COMM_WORLD);
    printf("Process 0 printing:  %s\n", message);
  }
  else {
    /* my rank == 1 */
    dest = 0;
    MPI_Recv(message, 100, MPI_CHAR, dest, tag, MPI_COMM_WORLD, &status);
    MPI_Send(message, strlen(message)+1, MPI_CHAR, dest, tag, MPI_COMM_WORLD);
    printf("Process 1 printing:  %s\n", message);
  }

  MPI_Finalize();
} /* end main */

!mpicc codes/openmpi/deadlock_send_recv.c -o ~/deadlock_send_recv
!mpirun -np 2 ~/deadlock_send_recv

# The [*] is indicative of a running notebook shell, and if it does not turn into a number,
# it means the cell is hanged (deadlocked by MPI).
# To escape a hanged cell, click the Square (Stop) button in the tool bar


I am working on the file "05-intro-to-mpi.ipynb".
The first task is:
How do we do point-to-point communication at scale?
- Rely on rank and size
Can you write Python code for it?

%%writefile codes/openmpi/multi_send_recv.c
#include <mpi.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char** argv)
{
  int my_rank;
  int size;
  int tag=0;
  int buf,i;
  int des1,des2;
  MPI_Status status;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  /*  set up data */
  buf = my_rank;

  //printf("Process %2d has original value %2d \n",my_rank,buf);

  /* set up source and destination */
  des1 = (my_rank + 1) % size;
  des2 = (my_rank + size - 1) % size;
  //printf("Process %2d has des1 %2d and des2 %2d\n",my_rank,des1,des2);

  /* shift the data n/2 steps */
  for (i = 0; i < size/2; i++){
    MPI_Send(&buf,1,MPI_INT,des1,tag,MPI_COMM_WORLD);
    MPI_Recv(&buf,1,MPI_INT,MPI_ANY_SOURCE,tag,MPI_COMM_WORLD,&status);
    MPI_Barrier(MPI_COMM_WORLD);
  }

  MPI_Send(&buf,1,MPI_INT,des2,tag,MPI_COMM_WORLD);
  MPI_Recv(&buf,1,MPI_INT,MPI_ANY_SOURCE,tag,MPI_COMM_WORLD,&status);

  MPI_Barrier(MPI_COMM_WORLD);
  printf("Process %2d now has value %2d\n",my_rank,buf);

  /* Shut down MPI */
  MPI_Finalize();

} /* end main */

!mpicc codes/openmpi/multi_send_recv.c -o ~/multi_send_recv
!mpirun -np 4 --map-by core:OVERSUBSCRIBE ~/multi_send_recv


I am working on the file "05-intro-to-mpi.ipynb".
The first task is:
Individual processes rely on communication (message passing) to enforce workflow
Point-to-point Communication
Collective Communication

<center> Point-to-Point: Send and Receive
 Original MPI C Syntax: MPI_Send
int MPI_Send(void *buf,
    int count,
    MPI_Datatype datatype,
    int dest,
    int tag,
    MPI_Comm comm)

MPI_Datatype may be MPI_BYTE, MPI_PACKED, MPI_CHAR, MPI_SHORT, MPI_INT, MPI_LONG, MPI_FLOAT, MPI_DOUBLE, MPI_LONG_DOUBLE, MPI_UNSIGNED_CHAR
dest is the rank of the process the message is sent to
tag is an integer identify the message. Programmer is responsible for managing tag

 Original MPI C Syntax: MPI_Recv
int MPI_Recv(
    void *buf,
    int count,
    MPI_Datatype datatype,
    int source,
    int tag,
    MPI_Comm comm,
    MPI_Status *status)

MPI_Datatype may be MPI_BYTE, MPI_PACKED, MPI_CHAR, MPI_SHORT, MPI_INT, MPI_LONG, MPI_FLOAT, MPI_DOUBLE, MPI_LONG_DOUBLE, MPI_UNSIGNED_CHAR
source is the rank of the process from which the message was sent.
tag is an integer identify the message. MPI_Recv will only place data in the buffer if the tag from MPI_Send matches. The constant MPI_ANY_TAG may be used when the source tag is unknown or not important.
Can you write Python code for it?

We want to write an MPI program that exchange the ranks of two processes, 0 and 1.

%%writefile codes/openmpi/send_recv.c
#include <mpi.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char** argv)
{
  int my_rank;
  int size;
  int tag=0;
  int buf,i;
  int des1,des2;
  MPI_Status status;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  /*  set up data */
  buf = my_rank;

  printf("Process %2d has original value %2d \n",my_rank,buf);

  if (my_rank == 0){
    MPI_Send(&buf,1,MPI_INT,1,tag,MPI_COMM_WORLD);
    MPI_Recv(&buf,1,MPI_INT,1,tag,MPI_COMM_WORLD,&status);
  }

  if (my_rank == 1){
    MPI_Recv(&buf,1,MPI_INT,0,tag,MPI_COMM_WORLD,&status);
    MPI_Send(&buf,1,MPI_INT,0,tag,MPI_COMM_WORLD);
  }
  printf("Process %2d now has value %2d\n",my_rank,buf);

  MPI_Finalize();
} /* end main */

!mpicc codes/openmpi/send_recv.c -o ~/send_recv
!mpirun -np 2 ~/send_recv


I am working on the file "05-intro-to-mpi.ipynb".
The first task is:
Original MPI C Syntax: MPI_Gather
int MPI_Gather(
    void *sendbuff,
    int sendcount,
    MPI_Datatype sendtype,
    void *recvbuff,
    int recvcnt,
    MPI_Datatype recvtype,
    int root,
    MPI_Comm comm);
Can you write Python code for it?

%%writefile codes/openmpi/gather.c
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  int size;
  int my_rank;
  int sendbuf[2];
  int recvbuf[8] = {-1,-1,-1,-1,-1,-1,-1,-1};
  int i;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

  for (i = 0; i < 2; i++){
    sendbuf[i] = my_rank;
  }
  MPI_Gather(&sendbuf, 2, MPI_INT, &recvbuf, 2, MPI_INT, 0, MPI_COMM_WORLD);
  for (i = 0; i < 8; i++){
    printf ("Process %d has element %d at index %d in its recvbuf \n",
               my_rank, recvbuf[i], i);
  }

  /* Finalize the MPI environment. */
  MPI_Finalize();
}

!mpicc codes/openmpi/gather.c -o ~/gather
!mpirun -np 4 --map-by core:OVERSUBSCRIBE ~/gather


I am working on the file "05-intro-to-mpi.ipynb".
The first task is:
Original MPI C Syntax: MPI_Reduce
int MPI_Reduce(
    void *sendbuf,
    void *recvbuff,
    int count,
    MPI_Datatype datatype,
    MPI_OP op,
    int root,
    MPI_Comm comm);
- MPI_Op may be MPI_MIN, MPI_MAX, MPI_SUM, MPI_PROD (twelve total)
- Programmer may add operations, must be commutative and associative
- If count > 1, then operation is performed element-wise
Can you write Python code for it?

%%writefile codes/openmpi/reduce.c
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
  int size;
  int my_rank;
  int rank_sum;
  int i;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

  rank_sum = my_rank;

  MPI_Reduce(&my_rank, &rank_sum, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
  printf ("The total sum of all ranks at process %d is %d \n", my_rank, rank_sum);

  /* Finalize the MPI environment. */
  MPI_Finalize();
}

!mpicc codes/openmpi/reduce.c -o ~/reduce
!mpirun -np 4 --map-by core:OVERSUBSCRIBE ~/reduce

!mpicc codes/openmpi/reduce.c -o ~/reduce
!mpirun -np 8 --map-by core:OVERSUBSCRIBE ~/reduce