I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If you want concat to work like numpy.concatanate, you may provide the ignore_index=True argument.
Exercise 1
In the data/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each. Write code that imports each of the data spreadsheets and combines them into a single DataFrame, adding the identifying information from the metadata spreadsheet as columns in the combined DataFrame.
Can you write Python code for it?

# Loading all the .xls files one by one
mb1 = pd.read_excel('Data/microbiome/MID1.xls', 'Sheet 1', index_col=0, header=None)
mb2 = pd.read_excel('Data/microbiome/MID2.xls', 'Sheet 1', index_col=0, header=None)
mb3 = pd.read_excel('Data/microbiome/MID3.xls', 'Sheet 1', index_col=0, header=None)
mb4 = pd.read_excel('Data/microbiome/MID4.xls', 'Sheet 1', index_col=0, header=None)
mb5 = pd.read_excel('Data/microbiome/MID5.xls', 'Sheet 1', index_col=0, header=None)
mb6 = pd.read_excel('Data/microbiome/MID6.xls', 'Sheet 1', index_col=0, header=None)
mb7 = pd.read_excel('Data/microbiome/MID7.xls', 'Sheet 1', index_col=0, header=None)
mb8 = pd.read_excel('Data/microbiome/MID8.xls', 'Sheet 1', index_col=0, header=None)
mb9 = pd.read_excel('Data/microbiome/MID9.xls', 'Sheet 1', index_col=0, header=None)
# Each of these files contain two column : the name of the taxon and a counter. So we name the second column as "count" to keep the same meaning.
mb1.columns = mb2.columns = mb3.columns = mb4.columns = mb5.columns = mb6.columns = mb7.columns = mb8.columns = mb9.columns = ['Count']
# Same here for the first column by adding the name of the taxon.
mb1.index.name = mb2.index.name = mb3.index.name = mb4.index.name = mb5.index.name = mb6.index.name = mb7.index.name = mb8.index.name = mb9.index.name = 'Taxon'
# Now we'll add three columns which are defined in the metadata file : the barcode, the group and the sample type of each excel file.
dataframe = pd.concat([mb1, mb2, mb3, mb4, mb5, mb6, mb7, mb8, mb9], axis=0)
dataframe['Barcode']=['MID1']*len(mb1) + ['MID2']*len(mb2) + ['MID3']*len(mb3) + ['MID4']*len(mb4)+ ['MID5']*len(mb5)+ ['MID6']*len(mb6)+ ['MID7']*len(mb7)+ ['MID8']*len(mb8)+ ['MID9']*len(mb9)
dataframe['Group']=['Extraction Control']*len(mb1) + ['NEC 1']*len(mb2) + ['Control 1']*len(mb3) + ['NEC 2']*len(mb4)+ ['Control 2']*len(mb5)+ ['NEC 1']*len(mb6)+ ['Control 1']*len(mb7)+ ['NEC 2']*len(mb8)+ ['Control 2']*len(mb9)
dataframe['Sample']=['NA']*len(mb1) + ['tissue']*len(mb2) + ['tissue']*len(mb3) +...


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
In such situations, we can replace the zero with a value so small that it makes no difference to the ensuing analysis. We can do this with replace.
Can you write Python code for it?

vals = vals.replace(0, 1e-6)
np.log(vals)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
In this section, we will manipulate data collected from ocean-going vessels on the eastern seaboard. Vessel operations are monitored using the Automatic Identification System (AIS), a safety at sea navigation technology which vessels are required to maintain and that uses transponders to transmit very high frequency (VHF) radio signals containing static information including ship name, call sign, and country of origin, as well as dynamic information unique to a particular voyage such as vessel location, heading, and speed.
The International Maritime Organization’s (IMO) International Convention for the Safety of Life at Sea requires functioning AIS capabilities on all vessels 300 gross tons or greater and the US Coast Guard requires AIS on nearly all vessels sailing in U.S. waters. The Coast Guard has established a national network of AIS receivers that provides coverage of nearly all U.S. waters. AIS signals are transmitted several times each minute and the network is capable of handling thousands of reports per minute and updates as often as every two seconds. Therefore, a typical voyage in our study might include the transmission of hundreds or thousands of AIS encoded signals. This provides a rich source of spatial data that includes both spatial and temporal information.
For our purposes, we will use summarized data that describes the transit of a given vessel through a particular administrative area. The data includes the start and end time of the transit segment, as well as information about the speed of the vessel, how far it travelled, etc.
Can you write Python code for it?

segments = pd.read_csv("Data/AIS/transit_segments.csv")
segments.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Inidcator variables
For some statistical analyses (e.g. regression models or analyses of variance), categorical or group variables need to be converted into columns of indicators--zeros and ones--to create a so-called design matrix. The Pandas function get_dummies (indicator variables are also known as dummy variables) makes this transformation straightforward.
Let's consider the DataFrame containing the ships corresponding to the transit segments on the eastern seaboard. The type variable denotes the class of vessel; we can create a matrix of indicators for this. For simplicity, lets filter out the 5 most common types of ships:
Can you write Python code for it?

top5 = vessels.type.isin(vessels.type.value_counts().index[:5])
top5.head(10)

vessels5 = vessels[top5]

pd.get_dummies(vessels5.type).head(10)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
It is easy to do column selection within groupby operations, if we are only interested split-apply-combine operations on a subset of columns:
Can you write Python code for it?

cdystonia_grouped['twstrs'].mean().head()

# This gives the same result as a DataFrame
cdystonia_grouped[['twstrs']].mean().head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Its also possible to group by one or more levels of a hierarchical index. Recall cdystonia2, which we created with a hierarchical index:
Can you write Python code for it?

cdystonia2.head(10)

cdystonia2.groupby(level='obs', axis=0)['twstrs'].mean()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Let's give the index and columns meaningful labels:
Can you write Python code for it?

mb1.columns = mb2.columns = ['Count']

mb1.index.name = mb2.index.name = 'Taxon'

mb1.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Merging and joining DataFrame objects
Now that we have the vessel transit information as we need it, we may want a little more information regarding the vessels themselves. In the data/AIS folder there is a second table that contains information about each of the ships that traveled the segments in the segments table.
Can you write Python code for it?

vessels = pd.read_csv("Data/AIS/vessel_information.csv", index_col='mmsi')
vessels.head()

[v for v in vessels.type.unique() if v.find('/')==-1]

vessels.type.value_counts()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Note that you can easily combine discretiztion with the generation of indicator variables shown above:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that additional arguments for the applied function can be passed via apply after the function name. It assumes that the DataFrame is the first argument.
Recall the microbiome data sets that we used previously for the concatenation example. Suppose that we wish to aggregate the data at a higher biological classification than genus. For example, we can identify samples down to class, which is the 3rd level of organization in each index.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that c_ and r_ are not really functions at all, since it is performing some sort of indexing operation, rather than being called. They are actually class instances, but they are here behaving mostly like functions. Don't think about this too hard; just know that they are there.

This operation is also called binding or stacking.
With Pandas' indexed data structures, there are additional considerations as the overlap in index values between two data structures affects how they are concatenate.
Lets import two microbiome datasets, each consisting of counts of microorganiams from a particular patient. We will use the first column of each dataset as the index.
Can you write Python code for it?

mb1 = pd.read_excel('Data/microbiome/MID1.xls', 'Sheet 1', index_col=0, header=None)
mb2 = pd.read_excel('Data/microbiome/MID2.xls', 'Sheet 1', index_col=0, header=None)
mb1.shape, mb2.shape

mb1.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that these are 8-bit integers, which are essentially single bytes of data, making memory usage lower.
There is also a performance benefit. Consider an operation such as calculating the total segment lengths for each ship in the segments table (this is also a preview of pandas' groupby operation!):
Can you write Python code for it?

%time segments.groupby(segments.name).seg_length.sum().sort_values(ascending=False, inplace=False).head()

segments['name'] = segments.name.astype('category')

%time segments.groupby(segments.name).seg_length.sum().sort_values(ascending=False, inplace=False).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that without any information about which column to use as a key, Pandas did the right thing and used the id column in both tables. Unless specified otherwise, merge will used any common column names as keys for merging the tables.
Notice also that id=3 from df1 was omitted from the merged table. This is because, by default, merge performs an inner join on the tables, meaning that the merged table represents an intersection of the two tables.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Our first order of business will be to convert these data to datetime. The strptime method parses a string representation of a date and/or time field, according to the expected format of this information.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Permutation and sampling
For some data analysis tasks, such as simulation, we need to be able to randomly reorder our data, or draw random values from it. Calling NumPy's permutation function with the length of the sequence you want to permute generates an array with a permuted sequence of integers, which can be used to re-order the sequence.
Can you write Python code for it?

new_order = np.random.permutation(len(segments))
new_order[:30]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Since the data are now ordinal, rather than numeric, we can give them labels:
Can you write Python code for it?

pd.cut(cdystonia.age, [20,40,60,80,90], labels=['young','middle-aged','old','really old'])[:30]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
The add_prefix and add_suffix methods can be used to give the columns of the resulting table labels that reflect the transformation:
Can you write Python code for it?

cdystonia_grouped.mean().add_suffix('_mean').head()

# The median of the `twstrs` variable
cdystonia_grouped['twstrs'].quantile(0.5)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
The challenge, however, is that several ships have travelled multiple segments, so there is not a one-to-one relationship between the rows of the two tables. The table of vessel information has a one-to-many relationship with the segments.
In Pandas, we can combine tables according to the value of one or more keys that are used to identify rows, much like an index. Using a trivial example:
Can you write Python code for it?

df1 = pd.DataFrame(dict(id=range(4), age=np.random.randint(18, 31, size=4)))
df2 = pd.DataFrame(dict(id=list(range(3))+list(range(3)),
                        score=np.random.random(size=6)))

df1

df2

pd.merge(df1, df2)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
The parentheses indicate an open interval, meaning that the interval includes values up to but not including the endpoint, whereas the square bracket is a closed interval, where the endpoint is included in the interval. We can switch the closure to the left side by setting the right flag to False:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
This behavior can be overridden by specifying a suffixes argument, containing a list of the suffixes to be used for the columns of the left and right columns, respectively.
Concatenation
A common data manipulation is appending rows or columns to a dataset that already conform to the dimensions of the exsiting rows or colums, respectively. In NumPy, this is done either with concatenate or the convenience "functions" c_ and r_:
Can you write Python code for it?

np.concatenate([np.random.random(5), np.random.random(5)])

np.r_[np.random.random(5), np.random.random(5)]

np.c_[np.random.random(5), np.random.random(5)]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
This can be used to easily filter rows by particular temporal attributes:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
This dataset includes repeated measurements of the same individuals (longitudinal data). Its possible to present such information in (at least) two ways: showing each repeated measurement in their own row, or in multiple columns representing multiple measurements.
The stack method rotates the data frame so that columns are represented in rows:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
This grouped dataset is hard to visualize
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
This illustrates the two formats for longitudinal data: long and wide formats. Its typically better to store data in long format because additional data can be included as additional rows in the database, while wide format requires that the entire database schema be altered by adding columns to every row as data are collected.
The preferable format for analysis depends entirely on what is planned for the data, so it is imporant to be able to move easily between them.
Pivoting
The pivot method allows a DataFrame to be transformed easily between long and wide formats in the same way as a pivot table is created in a spreadsheet. It takes three arguments: index, columns and values, corresponding to the DataFrame index (the row headers), columns and cell values, respectively.
For example, we may want the twstrs variable (the response variable) in wide format according to patient, as we saw with the unstacking method above:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Though most of the transits appear to be short, there are a few longer distances that make the plot difficult to read. This is where a transformation is useful:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
To complement this, unstack pivots from rows back to columns.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
To convert our "wide" format back to long, we can use the melt function, appropriately parameterized. This function is useful for DataFrames where one
or more columns are identifier variables (id_vars), with the remaining columns being measured variables (value_vars). The measured variables are "unpivoted" to
the row axis, leaving just two non-identifier columns, a variable and its corresponding value, which can both be renamed using optional arguments.
Can you write Python code for it?

pd.melt(cdystonia_wide, id_vars=['patient','site','id','treat','age','sex'],
        var_name='obs', value_name='twsters').head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
To see this in action, consider the vessel transit segments dataset (which we merged with the vessel information to yield segments_merged). Say we wanted to return the 3 longest segments travelled by each ship:
Can you write Python code for it?

top3segments = segments_merged.groupby('mmsi').apply(top, column='seg_length', n=3)[['names', 'seg_length']]
top3segments.head(15)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
We can also create a hierarchical index based on keys identifying the original tables.
Can you write Python code for it?

pd.concat([mb1, mb2], keys=['patient1', 'patient2']).head()

pd.concat([mb1, mb2], keys=['patient1', 'patient2']).index.is_unique


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
We can convert all the dates in a particular column by using the apply method.
Can you write Python code for it?

segments.st_time.apply(lambda d: datetime.strptime(d, '%m/%d/%y %H:%M'))


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
We can see that although there are date/time fields in the dataset, they are not in any specialized format, such as datetime.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
we see that there is a mmsi value (a vessel identifier) in each table, but it is used as an index for the vessels table. In this case, we have to specify to join on the index for this table, and on the mmsi column for the other.
Can you write Python code for it?

segments_merged = pd.merge(vessels, segments, left_index=True, right_on='mmsi')

segments_merged.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
$$corr(x,y) = \frac{cov(x,y)}{(n-1)s_x s_y} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_i (x_i - \bar{x})^2 \sum_i (y_i - \bar{y})^2}}$$
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
A DataFrame has a second index, representing the columns:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Above, Pandas recognized NA and an empty field as missing data.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Additionally, the order of the set of indices in a hierarchical MultiIndex can be changed by swapping them pairwise:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Alternately, if we want to process our data in reasonable chunks, the chunksize argument will return an iterable object that can be employed in a data processing loop. For example, our microbiome data are organized by bacterial phylum, with 14 patients represented in each:
Can you write Python code for it?

pd.read_csv("Data/microbiome.csv", chunksize=14)

data_chunks = pd.read_csv("Data/microbiome.csv", chunksize=14)

mean_tissue = pd.Series({chunk.Taxon[0]: chunk.Tissue.mean() for chunk in data_chunks})

mean_tissue


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Alternatively, you can break ties via one of several methods, such as by the order in which they occur in the dataset:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Calling the DataFrame's rank method results in the ranks of all columns:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Clearly, sum is more meaningful for some columns than others. For methods like mean for which application to string variables is not just meaningless, but impossible, these columns are automatically exculded:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Contrast this with NumPy arrays, where arrays of the same length will combine values element-wise; adding Series combined values with the same label in the resulting series. Notice also that the missing values were propogated by addition.
DataFrame
Inevitably, we want to be able to store, view and manipulate data that is multivariate, where for every index there are multiple fields or columns of data, often of varying data type.
A DataFrame is a tabular data structure, encapsulating multiple series like columns in a spreadsheet. Data are stored internally as a 2-dimensional object, but the DataFrame allows us to represent and manipulate higher-dimensional data.
Can you write Python code for it?

data = pd.DataFrame({'value':[623, 1638, 569, 115, 433, 1130, 754, 555],
                     'patient':[1, 1, 1, 1, 2, 2, 2, 2],
                     'phylum':['Firmicutes', 'Proteobacteria', 'Actinobacteria',
    'Bacteroidetes', 'Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes']})
data


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Critically, the labels are used to align data when used in operations with other Series objects:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Data can also be sorted by any index level, using sortlevel:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Exercise 3
You can use the isin method query a DataFrame based upon a list of values as follows:
data['phylum'].isin(['Firmacutes', 'Bacteroidetes'])

Use isin to find all players that played for the Los Angeles Dodgers (LAN) or the San Francisco Giants (SFN). How many records contain these values?
Can you write Python code for it?

baseball_newind[baseball_newind['team'].isin(['LAN','SFN'])]
#there are 15 players who are either LAN or SFN


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Exercise 4
Calculate on base percentage for each player, and return the ordered series of estimates.
$$OBP = \frac{H + BB + HBP}{AB + BB + HBP + SF}$$
Can you write Python code for it?

def onbasepercent(x):
    onbases = x['h'] + x['bb'] + x['hbp']
    basetotal = x['ab'] + x['bb'] + x['hbp'] + x['sf']+1e-6

    return onbases/basetotal

baseball.apply(onbasepercent, axis=1).round(3)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
For a DataFrame, we can sort according to the values of one or more columns using the by argument of sort_values:
Can you write Python code for it?

baseball[['player','sb','cs']].sort_values(ascending=[False,True],
                                           by=['sb', 'cs']).head(10)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Hierarchical indices can be created on either or both axes. Here is a trivial example:
Can you write Python code for it?

frame = pd.DataFrame(np.arange(12).reshape(( 4, 3)),
                  index =[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                  columns =[['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']])

frame


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
If we have a DataFrame with a hierarchical index (or indices), summary statistics can be applied with respect to any of the index levels:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
If we plan on modifying an extracted Series, its a good idea to make a copy.
Can you write Python code for it?

vals = data.value.copy()
vals[5] = 1000
vals


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Importing data
A key, but often under-appreciated, step in data analysis is importing the data that we wish to analyze. Though it is easy to load basic data structures into Python using built-in tools or those provided by packages like NumPy, it is non-trivial to import structured data well, and to easily convert this input into a robust data structure:
genes = np.loadtxt("genes.csv", delimiter=",", dtype=[('gene', '|S10'), ('value', '<f4')])

Pandas provides a convenient set of functions for importing tabular data in a number of formats directly into a DataFrame object. These functions include a slew of options to perform type inference, indexing, parsing, iterating and cleaning automatically as data are imported.
Let's start with some more bacteria data, stored in csv format.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
In addition to using loc to select rows and columns by label, pandas also allows indexing by position using the iloc attribute.
So, we can query rows and columns by absolute position, rather than by name:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Indexing and Selection
Indexing works analogously to indexing in NumPy arrays, except we can use the labels in the Index object to extract values in addition to arrays of integers.
Can you write Python code for it?

# Sample Series object
hits = baseball_newind.h
hits

# Numpy-style indexing
hits[:3]

# Indexing by label
hits[['womacto01CHN2006','schilcu01BOS2006']]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Its important to note that the Series returned when a DataFrame is indexted is merely a view on the DataFrame, and not a copy of the data itself. So you must be cautious when manipulating this data:
Can you write Python code for it?

vals = data.value
vals

vals[5] = 0
vals


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Keep in mind that reindex does not work if we pass a non-unique index series.
We can remove rows or columns via the drop method:
Can you write Python code for it?

baseball.shape

baseball.drop([89525, 89526])

baseball.drop(['ibb','hbp'], axis=1)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Lets use apply to calculate a meaningful baseball statistics, slugging percentage:
$$SLG = \frac{1B + (2 \times 2B) + (3 \times 3B) + (4 \times HR)}{AB}$$
And just for fun, we will format the resulting estimate.
Can you write Python code for it?

def slugging(x):
    bases = x['h']-x['X2b']-x['X3b']-x['hr'] + 2*x['X2b'] + 3*x['X3b'] + 4*x['hr']
    ab = x['ab']+1e-6

    return bases/ab

baseball.apply(slugging, axis=1).round(3)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Missing data
The occurence of missing data is so prevalent that it pays to use tools like Pandas, which seamlessly integrates missing data handling so that it can be dealt with easily, and in the manner required by the analysis at hand.
Missing data are represented in Series and DataFrame objects by the NaN floating point value. However, None is also treated as missing, since it is commonly used as such in other contexts (e.g. NumPy).
Can you write Python code for it?

foo = pd.Series([np.nan, -3, None, 'foobar'])
foo

foo.isnull()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Missing values can also be interpolated, using any one of a variety of methods:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Missing values can be filled as desired, either with selected values, or by rule:
Can you write Python code for it?

baseball.reindex(id_range, method='ffill', columns=['player','year']).head()

baseball.reindex(id_range, fill_value='charliebrown', columns=['player']).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice that fillna by default returns a new object with the desired filling behavior, rather than changing the Series or  DataFrame in place (in general, we like to do this, by the way!).
We can alter values in-place using inplace=True.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice that read_csv automatically considered the first row in the file to be a header row.
We can override default behavior by customizing some the arguments, like header, names or index_col.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice that the Series is created in key-sorted order.
If we pass a custom index to Series, it will select the corresponding values from the dict, and treat indices without corrsponding values as missing. Pandas uses the NaN (not a number) type for missing values.
Can you write Python code for it?

bacteria2 = pd.Series(bacteria_dict,
                      index=['Cyanobacteria','Firmicutes',
                             'Proteobacteria','Actinobacteria'])
bacteria2

bacteria2.isnull()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice that the id index is not sequential. Say we wanted to populate the table with every id value. We could specify and index that is a sequence from the first to the last id numbers in the database, and Pandas would fill in the missing data with NaN values:
Can you write Python code for it?

id_range = range(baseball.index.values.min(), baseball.index.values.max())
baseball.reindex(id_range).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice that the indexing operation preserved the association between the values and the corresponding indices.
We can still use positional indexing if we wish.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice that we specified the id column as the index, since it appears to be a unique identifier. We could try to create a unique index ourselves by combining player and year:
Can you write Python code for it?

player_id = baseball.player + baseball.year.astype(str)
baseball_newind = baseball.copy()
baseball_newind.index = player_id
baseball_newind.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice the DataFrame is sorted by column name. We can change the order by indexing them in the order we desire:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Notice this is different than with Series, where dict-like indexing retrieved a particular element (row).
If we want access to a row in a DataFrame, we index its loc attribute.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Now, let's add them together, in hopes of getting 2-year home run totals:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Operations can also be broadcast between rows or columns.
For example, if we subtract the maximum number of home runs hit from the hr column, we get how many fewer than the maximum were hit by each player:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Or, looking at things row-wise, we can see how a particular player compares with the rest of the group with respect to important statistics
Can you write Python code for it?

baseball.loc[89521, "player"]

stats = baseball[['h','X2b', 'X3b', 'hr']]
diff = stats - stats.loc[89521]
diff[:10]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Pandas uses a custom data structure to represent the indices of Series and DataFrames.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Pandas' data alignment places NaN values for labels that do not overlap in the two Series. In fact, there are only 6 players that occur in both years.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Passing axis=1 will summarize over rows instead of columns, which only makes sense in certain situations.
Can you write Python code for it?

extra_bases = baseball[['X2b','X3b','hr']].sum(axis=1)
extra_bases.sort_values(ascending=False)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Ranking does not re-arrange data, but instead returns an index that ranks each value relative to others in the Series.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Rather than omitting missing data from an analysis, in some cases it may be suitable to fill the missing value in, either with a default value (such as zero) or a value that is either imputed or carried forward/backward from similar data points. We can do this programmatically in Pandas with the fillna argument.
Can you write Python code for it?

bacteria2.fillna(0)

data.fillna({'year': 2013, 'treatment':2})


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
So, indices need not be unique. Our choice is not unique because some players change teams within years.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Specifying a Series as a new columns cause its values to be added according to the DataFrame's index:
Can you write Python code for it?

treatment = pd.Series([0]*4 + [1]*2)
treatment

data['treatment'] = treatment
data


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
The sep argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats:
sep='\s+'

For a more useful index, we can specify the first two columns, which together provide a unique index to the data.
Can you write Python code for it?

mb = pd.read_csv("Data/microbiome.csv", index_col=['Patient','Taxon'])
mb.head(5)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
There are several other data formats that can be imported into Python and converted into DataFrames, with the help of buitl-in or third-party libraries. These include JSON, XML, HDF5, relational and non-relational databases, and various web APIs. These are beyond the scope of this tutorial, but are covered in Python for Data Analysis.
Pandas Fundamentals
This section introduces the new user to the key functionality of Pandas that is required to use the software effectively.
For some variety, we will leave our digestive tract bacteria behind and employ some baseball data.
Can you write Python code for it?

baseball = pd.read_csv("Data/baseball.csv", index_col='id')
baseball.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
These can be specified on a column-wise basis using an appropriate dict as the argument for na_values.
Microsoft Excel
Since so much financial and scientific data ends up in Excel spreadsheets (regrettably), Pandas' ability to directly import Excel spreadsheets is valuable. This support is contingent on having one or two dependencies (depending on what version of Excel file is being imported) installed: xlrd and openpyxl (these may be installed with either pip or easy_install).
The read_excel convenience function in pandas imports a specific sheet from an Excel file
Can you write Python code for it?

mb = pd.read_excel('Data/microbiome/MID2.xls', sheetname='Sheet 1', header=None)
mb.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
These labels can be used to refer to the values in the Series.
Can you write Python code for it?

bacteria['Actinobacteria']

bacteria

bacteria[[name.endswith('bacteria') for name in bacteria.index]]

[name.endswith('bacteria') for name in bacteria.index]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
This can be customized further by specifying how many values need to be present before a row is dropped via the thresh argument.
Can you write Python code for it?

data.loc[7, 'year'] = np.nan
data

data.dropna(thresh=5)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
This can be overridden by passing the how='all' argument, which only drops a row when every field is a missing value.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
This index is a MultiIndex object that consists of a sequence of tuples, the elements of which is some combination of the three columns used to create the index. Where there are multiple repeated values, Pandas does not print the repeats, making it easy to identify groups of values.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
This looks okay, but let's check:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
This table can be read into a DataFrame using read_csv:
Can you write Python code for it?

mb = pd.read_csv("data/microbiome.csv")
mb


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Ties are assigned the mean value of the tied ranks, which may result in decimal values.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Try running corr on the entire baseball DataFrame to see what is returned:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Try sorting the columns instead of the rows, in ascending order:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Try using this hierarchical index to retrieve Julio Franco (francju01), who played for the Atlanta Braves (ATL) in 2007:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can also apply functions to each column or row of a DataFrame
Can you write Python code for it?

stats.apply(np.median)

def range_calc(x):
    return x.max() - x.min()

stat_range = lambda x: x.max() - x.min()
stats.apply(stat_range)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can also filter according to the values in the Series:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can also use sort_values to sort a Series by value, rather than by label.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can assign meaningful labels to the index, if they are available:
Can you write Python code for it?

bacteria = pd.Series([632, 1638, 569, 115],
    index=['Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes'])

bacteria


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can create meaningful indices more easily using a hierarchical index; for now, we will stick with the numeric id field as our index.
Manipulating indices
Reindexing allows users to manipulate the data labels in a DataFrame. It forces a DataFrame to conform to the new index, and optionally, fill in missing data if requested.
A simple use of reindex is to alter the order of the rows:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can create or modify columns by assignment:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can extract the underlying data as a simple ndarray by accessing the values attribute:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
We can give both the array of values and the index meaningful labels themselves:
Can you write Python code for it?

bacteria.name = 'counts'
bacteria.index.name = 'phylum'
bacteria


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
Writing Data to Files
As well as being able to read several data input formats, Pandas can also export data to a variety of storage formats. We will bring your attention to just a couple of these.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
data.head shows all of the data up to a certain number, if it is indicated in the parantheses
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
read_csv is just a convenience function for read_table, since csv is such a common format:
Can you write Python code for it?

mb = pd.read_table("Data/microbiome.csv", sep=',')


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
this gives you the number of rows of data (8) and columns (3)
An alternative way of initializing a DataFrame is with a list of dicts:
Can you write Python code for it?

data = pd.DataFrame([{'patient': 1, 'phylum': 'Firmicutes', 'value': 632},
                    {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},
                    {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},
                    {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},
                    {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},
                    {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},
                    {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},
                    {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}])

data


I am working on the file "Homework/01 - Pandas and Data Wrangling/Intro to Pandas.ipynb".
The first task is:
this shows the last 3 rows of data of the data row
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
A common data analysis procedure is the split-apply-combine operation, which groups subsets of data together, applies a function to each of the groups, then recombines them into a new data table.
For example, we may want to aggregate our data with with some function.

*(figure taken from "Python for Data Analysis", p.251)*

We can aggregate in Pandas using the aggregate (or agg, for short) method:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
A logical way to specify these numerically is to change them to integer values, perhaps using "Placebo" as a baseline value. If we create a dict with the original values as keys and the replacements as values, we can pass it to the map method to implement the changes.
Can you write Python code for it?

treatment_map = {'Placebo': 0, '5000U': 1, '10000U': 2}

cdystonia['treatment'] = cdystonia.treat.map(treatment_map)
cdystonia.treatment


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
A related function qcut uses empirical quantiles to divide the data. If, for example, we want the quartiles -- (0-25%], (25-50%], (50-70%], (75-100%] -- we can just specify 4 intervals, which will be equally-spaced by default:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
A slightly cleaner way of doing this is to set the patient-level information as an index before unstacking:
Can you write Python code for it?

(cdystonia.set_index(['patient','site','id','treat','age','sex','week'])['twstrs']
     .unstack('week').head())


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Alternately, if we simply want to replace particular values in a Series or DataFrame, we can use the replace method.
An example where replacement is useful is dealing with zeros in certain transformations. For example, if we try to take the log of a set of values:
Can you write Python code for it?

vals = pd.Series([float(i)**10 for i in range(10)])
vals

np.log(vals)


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Alternately, we can transform the data, using a function of our choice with the transform method:
Can you write Python code for it?

normalize = lambda x: (x - x.mean())/x.std()

cdystonia_grouped.transform(normalize).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Alternatively, one can specify custom quantiles to act as cut points:
Can you write Python code for it?

quantiles = pd.qcut(segments.seg_length, [0, 0.01, 0.05, 0.95, 0.99, 1])
quantiles[:30]


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Alternatively, you can pass keys to the concatenation by supplying the DataFrames (or Series) as a dict, resulting in a "wide" format table.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Apply
We can generalize the split-apply-combine methodology by using apply function. This allows us to invoke any function we wish on a grouped dataset and recombine them into a DataFrame.
The function below takes a DataFrame and a column name, sorts by the column, and takes the n largest values of that column. We can use this with apply to return the largest values from every group in a DataFrame in a single call.
Can you write Python code for it?

def top(df, column, n=5):
    return df.sort_values(by=column, ascending=False)[:n]


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
As a convenience, Pandas has a to_datetime method that will parse and convert an entire Series of formatted strings into datetime objects.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
By default the Categorical type represents an unordered categorical.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
By default, groupby groups by row, but we can specify the axis argument to change this. For example, we can group our columns by dtype this way:
Can you write Python code for it?

grouped_by_type = cdystonia.groupby(cdystonia.dtypes, axis=1)
{g:grouped_by_type.get_group(g) for g in grouped_by_type.groups}


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Columns of the datetime type have an accessor to easily extract properties of the data type. This will return a Series, with the same row index as the DataFrame. For example:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Compare this ordering with the original:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Concatenating along axis=1 will concatenate column-wise, but respecting the indices of the two DataFrames.
Can you write Python code for it?

pd.concat([mb1, mb2], axis=1).shape

pd.concat([mb1, mb2], axis=1).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Data aggregation and GroupBy operations
One of the most powerful features of Pandas is its GroupBy functionality. On occasion we may want to perform operations on groups of observations within a dataset. For exmaple:

aggregation, such as computing the sum of mean of each group, which involves applying a function to each group and returning the aggregated results
slicing the DataFrame into groups and then doing something with the resulting slices (e.g. plotting)
group-wise transformation, such as standardization/normalization
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Date/Time data handling
Date and time data are inherently problematic. There are an unequal number of days in every month, an unequal number of days in a year (due to leap years), and time zones that vary over space. Yet information about time is essential in many analyses, particularly in the case of time series analysis.
The datetime built-in library handles temporal information down to the nanosecond.
Can you write Python code for it?

from datetime import datetime

now = datetime.now()
now

now.day

now.weekday()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Exercise 2
Load the dataset in titanic.xls. It contains data on all the passengers that travelled on the Titanic.
Can you write Python code for it?

from IPython.core.display import HTML
HTML(filename='Data/titanic.html')


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
For example, we might be interested in the distribution of transit lengths, so we can plot them as a histogram:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
For this dataset, it makes sense to create a hierarchical index based on the patient and observation:
Can you write Python code for it?

cdystonia2 = cdystonia.set_index(['patient','obs'])
cdystonia2.head()

cdystonia2.index.is_unique


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Having a custom data type for dates and times is convenient because we can perform operations on them easily. For example, we may want to calculate the difference between two times:
Can you write Python code for it?

my_age = now - datetime(1970, 1, 1)
my_age

print(type(my_age))
my_age.days/365


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
However, an ordering can be imposed. The order is lexical by default, but will assume the order of the listed categories to be the desired order.
Can you write Python code for it?

cdystonia.treat.cat.categories = ['Placebo', '5000U', '10000U']

cdystonia.treat.cat.as_ordered().head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
However, since there are multiple taxonomic units with the same class, our index is no longer unique:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
However, the index is no longer unique, due to overlap between the two DataFrames.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If we are only interested in taxa that are included in both DataFrames, we can specify a join=inner argument.
Can you write Python code for it?

pd.concat([mb1, mb2], axis=1, join='inner').head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If we omit the values argument, we get a DataFrame with hierarchical columns, just as when we applied unstack to the hierarchically-indexed table:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If we want to transform this data so that repeated measurements are in columns, we can unstack the twstrs measurements according to obs.
Can you write Python code for it?

twstrs_wide = cdystonia2['twstrs'].unstack('obs')
twstrs_wide.head()

cdystonia_wide = (cdystonia[['patient','site','id','treat','age','sex']]
                  .drop_duplicates()
                  .merge(twstrs_wide, right_index=True, left_on='patient', how='inner')
                  .head())
cdystonia_wide


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If we wanted to use the second table to fill values absent from the first table, we could use combine_first.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If we wish, we can easily aggregate according to multiple keys:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If you simply want to divide your DataFrame into chunks for later use, its easy to convert them into a dict so that they can be easily indexed out as needed:
Can you write Python code for it?

chunks = dict(list(cdystonia_grouped))

chunks[4]


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
If you want concat to work like numpy.concatanate, you may provide the ignore_index=True argument.
Exercise 1
In the data/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each. Write code that imports each of the data spreadsheets and combines them into a single DataFrame, adding the identifying information from the metadata spreadsheet as columns in the combined DataFrame.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
In addition to datetime there are simpler objects for date and time information only, respectively.
Can you write Python code for it?

from datetime import date, time

time(3, 24)

date(1970, 9, 3)


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
In addition, time zone information can be applied:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
In such situations, we can replace the zero with a value so small that it makes no difference to the ensuing analysis. We can do this with replace.
Can you write Python code for it?

vals = vals.replace(0, 1e-6)
np.log(vals)


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
In this case, the default inner join is suitable; we are not interested in observations from either table that do not have corresponding entries in the other.
Notice that mmsi field that was an index on the vessels table is no longer an index on the merged table.
Here, we used the merge function to perform the merge; we could also have used the merge method for either of the tables:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
In this section, we will manipulate data collected from ocean-going vessels on the eastern seaboard. Vessel operations are monitored using the Automatic Identification System (AIS), a safety at sea navigation technology which vessels are required to maintain and that uses transponders to transmit very high frequency (VHF) radio signals containing static information including ship name, call sign, and country of origin, as well as dynamic information unique to a particular voyage such as vessel location, heading, and speed.
The International Maritime Organization’s (IMO) International Convention for the Safety of Life at Sea requires functioning AIS capabilities on all vessels 300 gross tons or greater and the US Coast Guard requires AIS on nearly all vessels sailing in U.S. waters. The Coast Guard has established a national network of AIS receivers that provides coverage of nearly all U.S. waters. AIS signals are transmitted several times each minute and the network is capable of handling thousands of reports per minute and updates as often as every two seconds. Therefore, a typical voyage in our study might include the transmission of hundreds or thousands of AIS encoded signals. This provides a rich source of spatial data that includes both spatial and temporal information.
For our purposes, we will use summarized data that describes the transit of a given vessel through a particular administrative area. The data includes the start and end time of the transit segment, as well as information about the speed of the vessel, how far it travelled, etc.
Can you write Python code for it?

segments = pd.read_csv("Data/AIS/transit_segments.csv")
segments.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
It is easy to do column selection within groupby operations, if we are only interested split-apply-combine operations on a subset of columns:
Can you write Python code for it?

cdystonia_grouped['twstrs'].mean().head()

# This gives the same result as a DataFrame
cdystonia_grouped[['twstrs']].mean().head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Let's give the index and columns meaningful labels:
Can you write Python code for it?

mb1.columns = mb2.columns = ['Count']

mb1.index.name = mb2.index.name = 'Taxon'

mb1.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Note that you can easily combine discretiztion with the generation of indicator variables shown above:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that additional arguments for the applied function can be passed via apply after the function name. It assumes that the DataFrame is the first argument.
Recall the microbiome data sets that we used previously for the concatenation example. Suppose that we wish to aggregate the data at a higher biological classification than genus. For example, we can identify samples down to class, which is the 3rd level of organization in each index.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that c_ and r_ are not really functions at all, since it is performing some sort of indexing operation, rather than being called. They are actually class instances, but they are here behaving mostly like functions. Don't think about this too hard; just know that they are there.

This operation is also called binding or stacking.
With Pandas' indexed data structures, there are additional considerations as the overlap in index values between two data structures affects how they are concatenate.
Lets import two microbiome datasets, each consisting of counts of microorganiams from a particular patient. We will use the first column of each dataset as the index.
Can you write Python code for it?

mb1 = pd.read_excel('Data/microbiome/MID1.xls', 'Sheet 1', index_col=0, header=None)
mb2 = pd.read_excel('Data/microbiome/MID2.xls', 'Sheet 1', index_col=0, header=None)
mb1.shape, mb2.shape

mb1.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that these are 8-bit integers, which are essentially single bytes of data, making memory usage lower.
There is also a performance benefit. Consider an operation such as calculating the total segment lengths for each ship in the segments table (this is also a preview of pandas' groupby operation!):
Can you write Python code for it?

%time segments.groupby(segments.name).seg_length.sum().sort_values(ascending=False, inplace=False).head()

segments['name'] = segments.name.astype('category')

%time segments.groupby(segments.name).seg_length.sum().sort_values(ascending=False, inplace=False).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Notice that without any information about which column to use as a key, Pandas did the right thing and used the id column in both tables. Unless specified otherwise, merge will used any common column names as keys for merging the tables.
Notice also that id=3 from df1 was omitted from the merged table. This is because, by default, merge performs an inner join on the tables, meaning that the merged table represents an intersection of the two tables.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Occasionally, there will be fields with the same in both tables that we do not wish to use to join the tables; they may contain different information, despite having the same name. In this case, Pandas will by default append suffixes _x and _y to the columns to uniquely identify them.
Can you write Python code for it?

segments['type'] = 'foo'
pd.merge(vessels, segments, left_index=True, right_on='mmsi').head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Our first order of business will be to convert these data to datetime. The strptime method parses a string representation of a date and/or time field, according to the expected format of this information.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Reshaping DataFrame objects
In the context of a single DataFrame, we are often interested in re-arranging the layout of our data.
This dataset is from Table 6.9 of Statistical Methods for the Analysis of Repeated Measurements by Charles S. Davis, pp. 161-163 (Springer, 2002). These data are from a multicenter, randomized controlled trial of botulinum toxin type B (BotB) in patients with cervical dystonia from nine U.S. sites.

Randomized to placebo (N=36), 5000 units of BotB (N=36), 10,000 units of BotB (N=37)
Response variable: total score on Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), measuring severity, pain, and disability of cervical dystonia (high scores mean more impairment)
TWSTRS measured at baseline (week 0) and weeks 2, 4, 8, 12, 16 after treatment began
Can you write Python code for it?

cdystonia = pd.read_csv("Data/cdystonia.csv", index_col=None)
cdystonia.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
The add_prefix and add_suffix methods can be used to give the columns of the resulting table labels that reflect the transformation:
Can you write Python code for it?

cdystonia_grouped.mean().add_suffix('_mean').head()

# The median of the `twstrs` variable
cdystonia_grouped['twstrs'].quantile(0.5)


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
The challenge, however, is that several ships have travelled multiple segments, so there is not a one-to-one relationship between the rows of the two tables. The table of vessel information has a one-to-many relationship with the segments.
In Pandas, we can combine tables according to the value of one or more keys that are used to identify rows, much like an index. Using a trivial example:
Can you write Python code for it?

df1 = pd.DataFrame(dict(id=range(4), age=np.random.randint(18, 31, size=4)))
df2 = pd.DataFrame(dict(id=list(range(3))+list(range(3)),
                        score=np.random.random(size=6)))

df1

df2

pd.merge(df1, df2)


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
The important difference between the category type and the object type is that category is represented by an underlying array of integers, which is then mapped to character labels.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
The index of these data is the unique biological classification of each organism, beginning with domain, phylum, class, and for some organisms, going all the way down to the genus level.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
This behavior can be overridden by specifying a suffixes argument, containing a list of the suffixes to be used for the columns of the left and right columns, respectively.
Concatenation
A common data manipulation is appending rows or columns to a dataset that already conform to the dimensions of the exsiting rows or colums, respectively. In NumPy, this is done either with concatenate or the convenience "functions" c_ and r_:
Can you write Python code for it?

np.concatenate([np.random.random(5), np.random.random(5)])

np.r_[np.random.random(5), np.random.random(5)]

np.c_[np.random.random(5), np.random.random(5)]


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
This can be used to easily filter rows by particular temporal attributes:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
This dataset includes repeated measurements of the same individuals (longitudinal data). Its possible to present such information in (at least) two ways: showing each repeated measurement in their own row, or in multiple columns representing multiple measurements.
The stack method rotates the data frame so that columns are represented in rows:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
This illustrates the two formats for longitudinal data: long and wide formats. Its typically better to store data in long format because additional data can be included as additional rows in the database, while wide format requires that the entire database schema be altered by adding columns to every row as data are collected.
The preferable format for analysis depends entirely on what is planned for the data, so it is imporant to be able to move easily between them.
Pivoting
The pivot method allows a DataFrame to be transformed easily between long and wide formats in the same way as a pivot table is created in a spreadsheet. It takes three arguments: index, columns and values, corresponding to the DataFrame index (the row headers), columns and cell values, respectively.
For example, we may want the twstrs variable (the response variable) in wide format according to patient, as we saw with the unstacking method above:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Though most of the transits appear to be short, there are a few longer distances that make the plot difficult to read. This is where a transformation is useful:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
To complement this, unstack pivots from rows back to columns.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
To convert our "wide" format back to long, we can use the melt function, appropriately parameterized. This function is useful for DataFrames where one
or more columns are identifier variables (id_vars), with the remaining columns being measured variables (value_vars). The measured variables are "unpivoted" to
the row axis, leaving just two non-identifier columns, a variable and its corresponding value, which can both be renamed using optional arguments.
Can you write Python code for it?

pd.melt(cdystonia_wide, id_vars=['patient','site','id','treat','age','sex'],
        var_name='obs', value_name='twsters').head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
To see this in action, consider the vessel transit segments dataset (which we merged with the vessel information to yield segments_merged). Say we wanted to return the 3 longest segments travelled by each ship:
Can you write Python code for it?

top3segments = segments_merged.groupby('mmsi').apply(top, column='seg_length', n=3)[['names', 'seg_length']]
top3segments.head(15)


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Using the string methods split and join we can create an index that just uses the first three classifications: domain, phylum and class.
Can you write Python code for it?

class_index = mb1.index.map(lambda x: ' '.join(x.split(' ')[:3]))

mb_class = mb1.copy()
mb_class.index = class_index


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Using this sequence as an argument to the take method results in a reordered DataFrame:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Value replacement
Frequently, we get data columns that are encoded as strings that we wish to represent numerically for the purposes of including it in a quantitative analysis. For example, consider the treatment variable in the cervical dystonia dataset:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
We can also create a hierarchical index based on keys identifying the original tables.
Can you write Python code for it?

pd.concat([mb1, mb2], keys=['patient1', 'patient2']).head()

pd.concat([mb1, mb2], keys=['patient1', 'patient2']).index.is_unique


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
We can also perform the same replacement that we used map for with replace:
Can you write Python code for it?

cdystonia2.treat.replace({'Placebo': 0, '5000U': 1, '10000U': 2})


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
We can convert all the dates in a particular column by using the apply method.
Can you write Python code for it?

segments.st_time.apply(lambda d: datetime.strptime(d, '%m/%d/%y %H:%M'))


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
We can convert this to a category type either by the Categorical constructor, or casting the column using astype:
Can you write Python code for it?

pd.Categorical(cdystonia.treat)

cdystonia['treat'] = cdystonia.treat.astype('category')

cdystonia.treat.describe()


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
We can re-establish a unique index by summing all rows with the same class, using groupby:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
We can see that although there are date/time fields in the dataset, they are not in any specialized format, such as datetime.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/temp/Data Wrangling with Pandas.ipynb".
The first task is:
Women and children first?

Describe each attribute, both with basic statistics and plots. State clearly your assumptions and discuss your findings.
Use the groupby method to calculate the proportion of passengers that survived by sex.
Calculate the same proportion, but by class and sex.
Create age categories: children (under 14 years), adolescents (14-20), adult (21-64), and senior(65+), and calculate survival proportions by age category, class and sex.
Can you write Python code for it?

titanic_df = pd.read_excel('Data/titanic.xls', 'titanic', index_col=None, header=0)

titanic_df

titanic_nameduplicate = titanic_df.duplicated(subset='name')

#titanic_nameduplicate

titanic_df.drop_duplicates(['name'])

gender_map = {'male':0, 'female':1}

titanic_df['sex'] = titanic_df.sex.map(gender_map)

titanic_df

titanic_grouped = titanic_df.groupby(titanic_df.sex)

titanic_grouped

for sex, survived in titanic_grouped:
    print('sex', sex)
    print('survived', survived)

titanic_grouped.agg(survived.mean).head()


I am working on the file "Homework/02 - Data from the Web/Question 2.ipynb".
The first task is:
Now we need to make other requests to IS Academia, which specify every parameter : computer science students, all the years, and all bachelor semester (which are a couple of two values : pedagogic period and semester type). Thus, we're going to get all the parameters we need to make the next request :
Can you write Python code for it?

# We first get the "Computer science" value
computerScienceField = htmlContent.find('option', text='Informatique')
computerScienceField

computerScienceValue = computerScienceField.get('value')
computerScienceValue

# Then, we're going to need all the academic years values.
academicYearsField = htmlContent.find('select', attrs={'name':'ww_x_PERIODE_ACAD'})
academicYearsSet = academicYearsField.findAll('option')

# Since there are several years to remember, we're storing all of them in a table to use them later
academicYearValues = []
# We'll put the textual content in a table aswell ("Master semestre 1", "Master semestre 2"...)
academicYearContent = []

for option in academicYearsSet:
    value = option.get('value')
    # However, we don't want any "null" value
    if value != 'null':
        academicYearValues.append(value)
        academicYearContent.append(option.text)

# Now, we have all the academic years that might interest us. We wrangle them a little bit so be able to make request more easily later.
academicYearValues_series = pd.Series(academicYearValues)
academicYearContent_series = pd.Series(academicYearContent)
academicYear_df = pd.concat([academicYearContent_series, academicYearValues_series], axis = 1)
academicYear_df.columns= ['Academic_year', 'Value']
academicYear_df = academicYear_df.sort_values(['Academic_year', 'Value'], ascending=[1, 0])
academicYear_df

# Then, let's get all the pedagogic periods we need. It's a little bit more complicated here because we need to link the pedagogic period with a season (eg : Bachelor 1 is autumn, Bachelor 2 is spring etc.)
# Thus, we need more than the pedagogic values. For doing some tests to associate them with the right season, we need the actual textual value ("Bachelor semestre 1", "Bachelor semestre 2" etc.)
pedagogicPeriodsField = htmlContent.find('select', attrs={'name':'ww_x_PERIODE_PEDAGO'})
pedagogicPeriodsSet = pedagogicPeriodsField.findAll('option')

# Same as above, we'll store the values in a table
pedagogicPeriodValues = []
# We'll put the...


I am working on the file "Homework/03 - Interactive Viz/.ipynb_checkpoints/HW3_Interactive_Viz-checkpoint.ipynb".
The first task is:
Now let's start by creating the indexes for universities and institutions :
Can you write Python code for it?

try:
    university_canton_dict = json.loads(open('university_canton_dict.json').read())
except FileNotFoundError:
    print('The dictionary for universities has not been saved yet. Let''s create a new dictionary.')
    university_canton_dict = {}

try:
    institution_canton_dict = json.loads(open('institution_canton_dict.json').read())
except FileNotFoundError:
    print('The dictionary for institutions has not been saved yet. Let''s create a new dictionary.')
    institution_canton_dict = {}


I am working on the file "Homework/03 - Interactive Viz/.ipynb_checkpoints/HW3_Interactive_Viz-checkpoint.ipynb".
The first task is:
Now we have the cantons associated with the universities/institutions :)
We save the dataframe into several formats, just in case, in order to use them in another notebook.
Can you write Python code for it?

try:
    p3_grant_cantons.to_csv('P3_Cantons.csv', encoding='utf-8')
except PermissionError:
    print("Couldn't access to the file. Maybe close Excel and try again :)")

p3_grant_cantons_json = p3_grant_cantons.to_json()
with open('P3_cantons.json', 'w') as fp:
    json.dump(p3_grant_cantons_json, fp, indent=4)

# The pickle format seems convenients to works with in Python, we're going to use it for transfering data to another notebook
p3_grant_cantons.to_pickle('P3_Cantons.pickle')


I am working on the file "Homework/03 - Interactive Viz/.ipynb_checkpoints/HW3_Interactive_Viz-checkpoint.ipynb".
The first task is:
Using only university names as a parametrer for geolocators isn't enough, because we get about half of the results.
A better idea would be using university name, and if there is no result, using the institution name as a second chance.
We will then take both universities and institution names into account in order to do our research:
1) Create initial data containers :
    - a key-value (name-canton) dictionary for universities and institutions :
        -['University', 'Canton'] and ['Institution', 'Canton']
    - a table that contains all cantons that have been found
2) Go trough the dataframe:
  - Check if the university name exists in our index. If not, geolocate the address
  - Check if the address is in Switzerland, otherwise canton will be considered as 'None'
  - If the university address is not found, try to find it with institution name, the same way as above
  - Extract the canton of the address (if it found an address and if it's in Switzerland). If no canton, let's say 'None' canton
  - Add the canton name to a the dictionary (or add something like 'None' if no canton has been found), so next university or institution that has been already found won't have to be geolocated again
  - Add the canton to the canton table
3) Add the canton table to the above dataframe in a way that they match with the universities or institutions
Can you write Python code for it?

# Let's start by creating our geolocator. We will use Google Maps API :
googlemapsapikeyjson = json.loads(open('google_maps_api_keys.json').read())
# We might need several API keys, to make a potentially huge number of requests
googlemapsapikeys = googlemapsapikeyjson['keys']

# Specifying the region for the geolocator: for instance, University of Geneva might be localized in the US !
test_geolocator =  geopy.geocoders.GoogleV3(api_key=googlemapsapikeys[0])
test_university_geneva = test_geolocator.geocode("University of Geneva", region='ch')
test_university_geneva


I am working on the file "Homework/03 - Interactive Viz/.ipynb_checkpoints/HW3_Interactive_Viz-checkpoint.ipynb".
The first task is:
We will need to log the next steps in order de debug easily the part of code related to geolocation...
It seems like it's hard to create a log file in iPython, so we adapted the following of code. Basically, it writes to a file named geolocation.log
Can you write Python code for it?

# set root logger level
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)

# setup custom logger
logger = logging.getLogger(__name__)
handler = logging.FileHandler('geolocation.log')

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

# log
logger.info('This file is used to debug the next code part related to geolocation of universities/institutions')


I am working on the file "Homework/03 - Interactive Viz/HW3_Interactive_Viz.ipynb".
The first task is:
First, we will locate projcets according to the University name.
We will ignore all project in which the University is not mentioned : we assume that if it's not, the project is probably outside Switzerland.
If we have the time, a better solution would be taking the institution's location into account as well.
Can you write Python code for it?

# Removing rows in which University is not mentioned
# p3_grant_export_data = p3_grant_export_data.dropna(subset=['University'])
# p3_grant_export_data.size

p3_grant_export_data


I am working on the file "Homework/03 - Interactive Viz/HW3_Interactive_Viz.ipynb".
The first task is:
It's rather dirty, but we will need more than one API key to make all the requests we need for our data.
So we created several Google API keys and switch the key each time the current one cannot be used anymore !
Here is the main code to get all the cantons that we will associate with our dataframe:
Can you write Python code for it?

# We create tables that will contains every canton we find, so we'll be able to match it with the dataframe at the end.
logger.debug('Beginning of geolocation : creating canton tables')
canton_shortname_table = [] # eg: VD
canton_longname_table = []# eg: Vaud

# number of rows analysed. Can be limited for debuging (eg : 10) because the number of requests to Google Maps API is limited !
MAX_ROWS = math.inf # values between 0 and math.inf
row_counter = 0 # will be incremented each time we iterate over a row

# maximum duration of a query to the geocoder, in seconds
geocoder_timeout = 5

# We're going to use more than one API key if we want to make all the requests !! :@
# Keys are referenced in a table, se we start with the first key:
APIkeynumber = 0

# This function definition makes the geolocator "stubborn" : it uses all the keys that are available and if it gets a timeout error, it just tries again !
def stubborn_geocode(geolocator, address):
    global APIkeynumber

    try:
        geolocator = geopy.geocoders.GoogleV3(api_key=googlemapsapikeys[APIkeynumber])
        return geolocator.geocode(address, region='ch', timeout=geocoder_timeout)

    except geopy.exc.GeocoderTimedOut:
        print("Error : the geocoder timed out. Let's try again...")
        return stubborn_geocode(geolocator, address)

    except geopy.exc.GeocoderQuotaExceeded:
        print("Error : The given key has gone over the requests limit in the 24 hour period or has submitted too many requests in too short a period of time. Let's try again with a different key...")
        APIkeynumber = APIkeynumber + 1

        try:
            print("Trying API key n°" + str(APIkeynumber) + "...")
            return stubborn_geocode(geolocator, address)

        except IndexError:
            print("Error : Out of API keys ! We need to request another API key from Google :(")
            print("When you get a new API key, add it to the json file containing the others keys.")
            # We have to...


I am working on the file "Homework/03 - Interactive Viz/HW3_Interactive_Viz.ipynb".
The first task is:
Now let's start by creating the indexes for universities and institutions :
Can you write Python code for it?


I am working on the file "Homework/03 - Interactive Viz/HW3_Interactive_Viz.ipynb".
The first task is:
This is the end of the first part. Now that we have linked universities and institutions to cantons, we can start working with the map !
(2) In this part of the exercise, we now need to put the data which we have procured about the funding levels of the different universities that are located in different cantons onto a canton map. We will do so using Folio and take the example TopoJSON mapping which they use.
Can you write Python code for it?

import folium

# Import the Switzerland map (from the folio pylib notebook)
topo_geo = r'ch-cantons.topojson.json'

# Import our csv file with all of the values for the amounts of the grants
grants_data = pd.read_csv('P3_Cantons_Sum.csv')
#grants_data['Approved Amount'] = (grants_data['Approved Amount']).astype(int)

missing_cantons = pd.Series(['UR','OW','NW','GL','BL','AR','AI','JU'], name='Canton Shortname')
missing_cantons_zeros = pd.Series([0,0,0,0,0,0,0,0], name='Approved Amount')
missing_cantons_df = pd.DataFrame([missing_cantons, missing_cantons_zeros]).T
grants_data_all_cantons = grants_data.append(missing_cantons_df)
grants_data_all_cantons = grants_data_all_cantons.reset_index(drop=True)

grants_data_all_cantons['Approved Amount'] = grants_data_all_cantons['Approved Amount']/10000000

grants_data_all_cantons

# Need to be able to extract the id of the canton from the topo file
from pprint import pprint

with open(topo_geo) as data_file:
    data = json.load(data_file)
#pprint(data)

data['objects']['cantons']['geometries'][25]['id']

#len(data['objects']['cantons']['geometries'])

# Need to overlay the Swiss topo file on the generic Folio map
ch_map = folium.Map(location=[46.9, 8.3], tiles='Mapbox Bright', zoom_start=7)
#folium.TopoJson(open(topo_geo), 'objects.cantons', name = 'topojson').add_to(ch_map)
#folium.LayerControl().add_to(ch_map)

ch_map.geo_json(geo_path=topo_geo,
        data=grants_data_all_cantons,
        columns=['Canton Shortname', 'Approved Amount'],
        key_on='feature.id',
        topojson='objects.cantons',
        fill_color='YlGnBu',
        fill_opacity=0.7,
        line_opacity=0.5,
        legend_name='Total Grant Amount by Canton (tens millions CHF) (1970+)',
        threshold_scale=[0,0.01,10,150,300,400],
        reset = True)
ch_map.save('swiss.html')
ch_map


I am working on the file "Homework/04 - Applied ML/.ipynb_checkpoints/Homework04_Referees_teamawesome-checkpoint.ipynb".
The first task is:
1) Peeking into the Data
Can you write Python code for it?


I am working on the file "Homework/04 - Applied ML/.ipynb_checkpoints/Homework04_Referees_teamawesome-checkpoint.ipynb".
The first task is:
2) Disaggregate the data so each row is 1 game
Can you write Python code for it?

#for i in Data.iterrows():
#    if Data.at[i, 'games'] > 1:
#        df = Data[i]

# Testing - divide # cards (all types) by # games column
Data_hasImage['fractionYellow'] = Data_hasImage['yellowCards']/Data_hasImage['games']
Data_hasImage['fractionYellowRed'] = Data_hasImage['yellowReds']/Data_hasImage['games']
Data_hasImage['fractionRed'] = Data_hasImage['redCards']/Data_hasImage['games']

# Get the average of the raters
Data_hasImage['raterAvg'] = (Data_hasImage['rater1']+Data_hasImage['rater2'])/2

#Data_hasImage.head()


I am working on the file "Homework/04 - Applied ML/.ipynb_checkpoints/Homework04_Referees_teamawesome-checkpoint.ipynb".
The first task is:
3) Create the Training and Testing Datframes with only select data
Can you write Python code for it?

# Removing columns that we do not need
Data_Simple1 = Data_hasImage[['games', 'fractionYellow', 'fractionYellowRed', 'fractionRed',
                              'refNum', 'refCountry', 'raterAvg']]

#cols = ['playerShort', 'games', 'fractionYellow', 'fractionYellowRed', 'fractionRed',
cols = ['games', 'fractionYellow', 'fractionYellowRed', 'fractionRed', 'refNum', 'refCountry']
colsRes = ['raterAvg']

# Take a random 80% sample of the Data for the Training Sample
Data_Training = Data_Simple1.sample(frac=0.8)

# Need to split this into the data and the results columns
# http://stackoverflow.com/questions/34246336/python-randomforest-unknown-label-error
Input_Data_Training = Data_Training.drop(colsRes, axis=1)
#Results_Data_Training = list(Data_Training.raterAvg.values)
Results_Data_Training = Data_Training[colsRes]

# Take a random 20% sample of the Data for the Testing Sample
#Data_Testing = Data_Simple1.loc[~Data_Simple1.index.isin(Data_Training.index)]

# Need to split this into the data and the results columns
# http://stackoverflow.com/questions/34246336/python-randomforest-unknown-label-error
#Input_Data_Testing = Data_Testing.drop(colsRes, axis=1)
#Results_Data_Testing = list(Data_Testing.raterAvg.values)

# Need to make arrays
# http://www.analyticbridge.com/profiles/blogs/random-forest-in-python

trainArr = Input_Data_Training.as_matrix(cols) #training array
trainRes = Results_Data_Training.as_matrix(colsRes) #training results
#trainRes = np.asarray(Data_Training['raterAvg'], dtype="|S6")

trainRes


# Removing columns that we do not need
Data_Simple1 = Data_OneGamePerRow[['playerShort', 'yellowCards', 'yellowReds', 'redCards',
                              'refNum', 'refCountry', 'mean_rater']]

# Take a random 80% sample of the Data for the Training Sample
Data_Training = Data_Simple1.sample(frac=0.8)

# Take a random 20% sample of the Data for the Testing Sample
Data_Testing = Data_Simple1.loc[~Data_Simple1.index.isin(Data_Training.index)]

Data_Simple1


I am working on the file "Homework/04 - Applied ML/.ipynb_checkpoints/Homework04_Referees_teamawesome-checkpoint.ipynb".
The first task is:
II. Preparing the training & test data : Fraction version
Can you write Python code for it?

# 1) Remove the players without rater 1 / 2 rating because we won't be
# able to train or test the values (this can be done as bonus later)

Data_hasImage = Data[pd.notnull(Data['photoID'])]
#Data_hasImage.ix[:10,13:28]


I am working on the file "Homework/04 - Applied ML/.ipynb_checkpoints/Homework04_Referees_teamawesome-checkpoint.ipynb".
The first task is:
II. Preparing the training & test data : Unique Game Row version
1) Keep only players that have a Rater Image
Can you write Python code for it?


I am working on the file "Homework/04 - Applied ML/.ipynb_checkpoints/Homework04_Referees_teamawesome-checkpoint.ipynb".
The first task is:
III. Random Forest
Can you write Python code for it?

#Initialize
forest = RandomForestClassifier(n_estimators = 100)

# Fit the training data and create the decision trees
forest.fit(trainArr,trainRes)

# Take the same decision trees and run it on the test data
#testArr = test.as_matrix(cols)
#results = rf.predict(testArr)

#test['predictions'] = results
#test.head()


I am working on the file "Homework/04 - Applied ML/Homework04_Referees_teamawesome_Q2.ipynb".
The first task is:
(We show only skin color and number of "red cards" because it's a 2D plot, but we actually used 5 parameters: position, yellowCards, yellowReds, redCards and number of games. So this graph doesn't really represent how our data has been clustered.
This is only to check if some clustering has ben done. Here we don't really see two distincts clusters. It looks like more random coloring ! :x
Now, let's add the result to each player:
Can you write Python code for it?

cluster = pd.DataFrame(pd.Series(model.labels_, name='cluster'))
Data_Clustered = Data_aggregated
Data_Clustered['cluster'] = cluster
Data_Clustered


I am working on the file "Homework/04 - Applied ML/Homework04_Referees_teamawesome_Q2.ipynb".
The first task is:
2) Getting rif of referees and grouping data by soccer player
We need to aggregate the information about referees and group the result by soccer player. It means that each line will correspond to a soccer player, with the sum of all the cards he got, and we won't know anymore who gaves the cards.
Can you write Python code for it?

# Group by player and do the sum of every column, except for mean_rater (skin color) that we need to move away during the calculation (we don't want to sum skin color values !)
Data_aggregated = Data_hasImage.drop(['refNum', 'refCountry'], 1)
Data_aggregated = Data_aggregated.groupby(['playerShort', 'position'])['games','yellowCards', 'yellowReds', 'redCards'].sum()
Data_aggregated = Data_aggregated.reset_index()

# Take information of skin color for each player
Data_nbGames_skinColor = Data_hasImage
Data_nbGames_skinColor.drop_duplicates('playerShort')
Data_nbGames_skinColor['skinColor']=(Data_nbGames_skinColor['rater1']+Data_hasImage['rater2'])/2
Data_nbGames_skinColor = pd.DataFrame(Data_nbGames_skinColor[['playerShort','skinColor']])
Data_aggregated = pd.merge(left=Data_aggregated,right=Data_nbGames_skinColor, how='left', left_on='playerShort', right_on='playerShort')
Data_aggregated = Data_aggregated.drop_duplicates('playerShort')
Data_aggregated = Data_aggregated.reset_index(drop=True)
Data_aggregated


I am working on the file "Homework/04 - Applied ML/Homework04_Referees_teamawesome_Q2.ipynb".
The first task is:
II. Preparing data
1) Keep only players that have a Rater Image
Can you write Python code for it?

# Remove the players without rater 1 / 2 (ie: without photo) because we won't be
# able to train or test the values (this can be done as bonus later)

Data_hasImage = Data[pd.notnull(Data['photoID'])]


I am working on the file "Homework/04 - Applied ML/Homework04_Referees_teamawesome_Q2.ipynb".
The first task is:
III. Unsupervized machine learning
The first idea we got is to start an unsupervized learning kept as simple as possible.
We will have to take player position, the three types of cards and the skin color: that makes 5 dimensions to deal with !
Instead, let say we only look at the total number of cards the players got, and their skin color. Then we would be able to display something in 2 dimensions only:

Then, we would try to obtain two clusters that might lead to really simple conclusion such as "dark people slightly tend to get more cards":

Again, this is totally hypothetical. So let's give it a try.
We try to use a K means clustering methode to obtain 2 distinct clusters, with the help of this website:
http://stamfordresearch.com/k-means-clustering-in-python/
Can you write Python code for it?

# Input
x = Data_aggregated
x = x.drop(['playerShort'], 1)

# We have to convert every columns to floats, to be able to train our model
mapping = {'Center Back': 1, 'Attacking Midfielder': 2, 'Right Midfielder': 3, 'Center Midfielder': 4, 'Defensive Midfielder': 5, 'Goalkeeper':6, 'Left Fullback':7, 'Left Midfielder':8, 'Right Fullback':9, 'Center Forward':10, 'Left Winger':11, 'Right Winger':12}
x = x.replace({'position': mapping})
x

# Output with the same length as the input, that will contains the associated cluster
y = pd.DataFrame(index=x.index, columns=['targetCluster'])
y.head()

# K Means Cluster
model = KMeans(n_clusters=2)
model = model.fit(x)
model

# We got a model with two clusters
model.labels_

# View the results
# Set the size of the plot
plt.figure(figsize=(14,7))

# Create a colormap for the two clusters
colormap = np.array(['blue', 'lime'])

# Plot the Model Classification PARTIALLY
plt.scatter((0.5*x.yellowCards + x.yellowReds + x.redCards)/x.games, x.skinColor, c=colormap[model.labels_], s=40)
plt.xlabel('Red cards per game (yellow = half a red card)')
plt.ylabel('Skin color')
plt.title('K Mean Classification')
plt.show()