I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Above, Pandas recognized NA and an empty field as missing data.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Additionally, the order of the set of indices in a hierarchical MultiIndex can be changed by swapping them pairwise:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Alternately, if we want to process our data in reasonable chunks, the chunksize argument will return an iterable object that can be employed in a data processing loop. For example, our microbiome data are organized by bacterial phylum, with 15 patients represented in each:
Can you write Python code for it?

data_chunks = pd.read_csv("data/microbiome.csv", chunksize=15)

mean_tissue = {chunk.Taxon[0]:chunk.Tissue.mean() for chunk in data_chunks}

mean_tissue


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Alternatively, we can create a DataFrame with a dict of dicts:
Can you write Python code for it?

data = pd.DataFrame({0: {'patient': 1, 'phylum': 'Firmicutes', 'value': 632},
                    1: {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},
                    2: {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},
                    3: {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},
                    4: {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},
                    5: {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},
                    6: {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},
                    7: {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}})

data


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Alternatively, you can break ties via one of several methods, such as by the order in which they occur in the dataset:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
But note, we cannot use the attribute indexing method to add a new column:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
By default, dropna drops entire rows in which one or more values are missing.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Calling the DataFrame's rank method results in the ranks of all columns:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Clearly, sum is more meaningful for some columns than others. For methods like mean for which application to string variables is not just meaningless, but impossible, these columns are automatically exculded:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Contrast this with NumPy arrays, where arrays of the same length will combine values element-wise; adding Series combined values with the same label in the resulting series. Notice also that the missing values were propogated by addition.
DataFrame
Inevitably, we want to be able to store, view and manipulate data that is multivariate, where for every index there are multiple fields or columns of data, often of varying data type.
A DataFrame is a tabular data structure, encapsulating multiple series like columns in a spreadsheet. Data are stored internally as a 2-dimensional object, but the DataFrame allows us to represent and manipulate higher-dimensional data.
Can you write Python code for it?

data = pd.DataFrame({'value':[632, 1638, 569, 115, 433, 1130, 754, 555],
                     'patient':[1, 1, 1, 1, 2, 2, 2, 2],
                     'phylum':['Firmicutes', 'Proteobacteria', 'Actinobacteria',
    'Bacteroidetes', 'Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes']})
data


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Conversely, if we only want to import a small number of rows from, say, a very large data file we can use nrows:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Data can also be sorted by any index level, using sortlevel:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Data summarization
We often wish to summarize data in Series or DataFrame objects, so that they can more easily be understood or compared with similar data. The NumPy package contains several functions that are useful here, but several summarization or reduction methods are built into Pandas data structures.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Exercise
Calculate on base percentage for each player, and return the ordered series of estimates.
$$OBP = \frac{H + BB + HBP}{AB + BB + HBP + SF}$$
Hierarchical indexing
In the baseball example, I was forced to combine 3 fields to obtain a unique index that was not simply an integer value. A more elegant way to have done this would be to create a hierarchical index from the three fields.
Can you write Python code for it?

baseball_h = baseball.set_index(['year', 'team', 'player'])
baseball_h.head(10)


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
For a DataFrame, we can sort according to the values of one or more columns using the by argument of sort_index:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
If an index is not specified, a default sequence of integers is assigned as the index. A NumPy array comprises the values of the Series, while the index is a pandas Index object.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
If we wish to access columns, we can do so either by dict-like indexing or by attribute:
Can you write Python code for it?

data['value']

data.value

type(data.value)

type(data[['value']])


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
If you want to get fancy, both the row and column indices themselves can be given names:
Can you write Python code for it?

frame.index.names = ['key1', 'key2']
frame.columns.names = ['state', 'color']
frame


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Importing data
A key, but often under-appreciated, step in data analysis is importing the data that we wish to analyze. Though it is easy to load basic data structures into Python using built-in tools or those provided by packages like NumPy, it is non-trivial to import structured data well, and to easily convert this input into a robust data structure:
genes = np.loadtxt("genes.csv", delimiter=",", dtype=[('gene', '|S10'), ('value', '<f4')])

Pandas provides a convenient set of functions for importing tabular data in a number of formats directly into a DataFrame object. These functions include a slew of options to perform type inference, indexing, parsing, iterating and cleaning automatically as data are imported.
Let's start with some more bacteria data, stored in csv format.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Indexing and Selection
Indexing works analogously to indexing in NumPy arrays, except we can use the labels in the Index object to extract values in addition to arrays of integers.
Can you write Python code for it?

# Sample Series object
hits = baseball_newind.h
hits

# Numpy-style indexing
hits[:3]

# Indexing by label
hits[['womacto01CHN2006','schilcu01BOS2006']]


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Keep in mind that reindex does not work if we pass a non-unique index series.
We can remove rows or columns via the drop method:
Can you write Python code for it?

baseball.shape

baseball.drop([89525, 89526])

baseball.drop(['ibb','hbp'], axis=1)


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Lets use apply to calculate a meaningful baseball statistics, slugging percentage:
$$SLG = \frac{1B + (2 \times 2B) + (3 \times 3B) + (4 \times HR)}{AB}$$
And just for fun, we will format the resulting estimate.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Missing values can also be interpolated, using any one of a variety of methods:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Missing values can be filled as desired, either with selected values, or by rule:
Can you write Python code for it?

baseball.reindex(id_range, method='ffill', columns=['player','year']).head()

baseball.reindex(id_range, fill_value='mr.nobody', columns=['player']).head()


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Missing values may be dropped or indexed out:
Can you write Python code for it?

bacteria2

bacteria2.dropna()

bacteria2[bacteria2.notnull()]


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Most real-world data is incomplete, with values missing due to incomplete observation, data entry or transcription error, or other reasons. Pandas will automatically recognize and parse common missing data indicators, including NA and NULL.
Can you write Python code for it?

!cat data/microbiome_missing.csv

pd.read_csv("data/microbiome_missing.csv").head(20)


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Notice that because of the mix of string and integer (and NaN) values, the dtype of the array is object. The dtype will automatically be chosen to be as general as needed to accomodate all the columns.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Notice that read_csv automatically considered the first row in the file to be a header row.
We can override default behavior by customizing some the arguments, like header, names or index_col.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Notice that the Series is created in key-sorted order.
If we pass a custom index to Series, it will select the corresponding values from the dict, and treat indices without corrsponding values as missing. Pandas uses the NaN (not a number) type for missing values.
Can you write Python code for it?

bacteria2 = pd.Series(bacteria_dict, index=['Cyanobacteria','Firmicutes','Proteobacteria','Actinobacteria'])
bacteria2

bacteria2.isnull()


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Notice that we specified the id column as the index, since it appears to be a unique identifier. We could try to create a unique index ourselves by combining player and year:
Can you write Python code for it?

player_id = baseball.player + baseball.year.astype(str)
baseball_newind = baseball.copy()
baseball_newind.index = player_id
baseball_newind.head()


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Notice this is different than with Series, where dict-like indexing retrieved a particular element (row). If we want access to a row in a DataFrame, we index its ix attribute.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Operations
DataFrame and Series objects allow for several operations to take place either on a single object, or between two or more objects.
For example, we can perform arithmetic on the elements of two objects, such as combining baseball statistics across years:
Can you write Python code for it?

hr2006 = baseball[baseball.year==2006].xs('hr', axis=1)
hr2006.index = baseball.player[baseball.year==2006]

hr2007 = baseball[baseball.year==2007].xs('hr', axis=1)
hr2007.index = baseball.player[baseball.year==2007]

hr2006 = pd.Series(baseball.hr[baseball.year==2006].values, index=baseball.player[baseball.year==2006])
hr2007 = pd.Series(baseball.hr[baseball.year==2007].values, index=baseball.player[baseball.year==2007])

hr_total = hr2006 + hr2007
hr_total


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Operations can also be broadcast between rows or columns.
For example, if we subtract the maximum number of home runs hit from the hr column, we get how many fewer than the maximum were hit by each player:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Or, looking at things row-wise, we can see how a particular player compares with the rest of the group with respect to important statistics
Can you write Python code for it?

baseball.ix[89521]["player"]

stats = baseball[['h','X2b', 'X3b', 'hr']]
diff = stats - stats.xs(89521)
diff[:10]


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Pandas uses a custom data structure to represent the indices of Series and DataFrames.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Pandas' data alignment places NaN values for labels that do not overlap in the two Series. In fact, there are only 6 players that occur in both years.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Passing axis=1 will summarize over rows instead of columns, which only makes sense in certain situations.
Can you write Python code for it?

extra_bases = baseball[['X2b','X3b','hr']].sum(axis=1)
extra_bases.order(ascending=False)


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Rather than omitting missing data from an analysis, in some cases it may be suitable to fill the missing value in, either with a default value (such as zero) or a value that is either imputed or carried forward/backward from similar data points. We can do this programmatically in Pandas with the fillna argument.
Can you write Python code for it?

bacteria2.fillna(0)

data.fillna({'year': 2013, 'treatment':2})


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Similarly, the cross-section method xs (not a field) extracts a single column or row by label and returns it as a Series:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Specifying a Series as a new columns cause its values to be added according to the DataFrame's index:
Can you write Python code for it?

treatment = pd.Series([0]*4 + [1]*2)
treatment

data['treatment'] = treatment
data


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
The important difference between NumPy's functions and Pandas' methods is that the latter have built-in support for handling missing data.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
The most important consequence of a non-unique index is that indexing by label will return multiple values for some labels:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
The sep argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats:
sep='\s+'

For a more useful index, we can specify the first two columns, which together provide a unique index to the data.
Can you write Python code for it?

mb = pd.read_csv("data/microbiome.csv", index_col=['Taxon','Patient'])
mb.head()


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
The to_csv method writes a DataFrame to a comma-separated values (csv) file. You can specify custom delimiters (via sep argument), how missing values are written (via na_rep argument), whether the index is writen (via index argument), whether the header is included (via header argument), among other options.
An efficient way of storing data to disk is in binary format. Pandas supports this using Python’s built-in pickle serialization.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
There are several other data formats that can be imported into Python and converted into DataFrames, with the help of buitl-in or third-party libraries. These include JSON, XML, HDF5, relational and non-relational databases, and various web APIs. These are beyond the scope of this tutorial, but are covered in Python for Data Analysis.
Pandas Fundamentals
This section introduces the new user to the key functionality of Pandas that is required to use the software effectively.
For some variety, we will leave our digestive tract bacteria behind and employ some baseball data.
Can you write Python code for it?

baseball = pd.read_csv("data/baseball.csv", index_col='id')
baseball.head()


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
There is now a read_excel convenience function in Pandas that combines these steps into a single call:
Can you write Python code for it?

mb2 = pd.read_excel('data/microbiome/MID2.xls', sheetname='Sheet 1', header=None)
mb2.head()


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
This can be customized further by specifying how many values need to be present before a row is dropped via the thresh argument.
Can you write Python code for it?

data.ix[7, 'year'] = nan
data

data.dropna(thresh=4)


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
This can be overridden by passing the how='all' argument, which only drops a row when every field is a missing value.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
This index is a MultiIndex object that consists of a sequence of tuples, the elements of which is some combination of the three columns used to create the index. Where there are multiple repeated values, Pandas does not print the repeats, making it easy to identify groups of values.
Can you write Python code for it?

baseball_h.index[:10]

baseball_h.index.is_unique

baseball_h.ix[(2007, 'ATL', 'francju01')]


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
This is so that Index objects can be shared between data structures without fear that they will be changed.
Can you write Python code for it?

bacteria2.index = bacteria.index

bacteria2


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
This table can be read into a DataFrame using read_csv:
Can you write Python code for it?

mb = pd.read_csv("data/microbiome.csv")
mb


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Ties are assigned the mean value of the tied ranks, which may result in decimal values.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Unfortunately, there will sometimes be inconsistency with the conventions for missing data. In this example, there is a question mark "?" and a large negative number where there should have been a positive integer. We can specify additional symbols with the na_values argument:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can also apply functions to each column or row of a DataFrame
Can you write Python code for it?

stats.apply(np.median)

stat_range = lambda x: x.max() - x.min()
stats.apply(stat_range)


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can also calculate summary statistics across multiple columns, for example, correlation and covariance.
$$cov(x,y) = \sum_i (x_i - \bar{x})(y_i - \bar{y})$$
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can also filter according to the values in the Series:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can also use order to sort a Series by value, rather than by label.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can create meaningful indices more easily using a hierarchical index; for now, we will stick with the numeric id field as our index.
Manipulating indices
Reindexing allows users to manipulate the data labels in a DataFrame. It forces a DataFrame to conform to the new index, and optionally, fill in missing data if requested.
A simple use of reindex is to alter the order of the rows:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can create or modify columns by assignment:
Can you write Python code for it?

data.value[3] = 14
data

data['year'] = 2013
data


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We can give both the array of values and the index meaningful labels themselves:
Can you write Python code for it?

bacteria.name = 'counts'
bacteria.index.name = 'phylum'
bacteria


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We probably want this transposed:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
We will learn more about indexing below.
We can create a truly unique index by combining player, team and year:
Can you write Python code for it?

player_unique = baseball.player + baseball.team + baseball.year.astype(str)
baseball_newind = baseball.copy()
baseball_newind.index = player_unique
baseball_newind.head()

baseball_newind.index.is_unique


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
While we do want the operation to honor the data labels in this way, we probably do not want the missing values to be filled with NaN. We can use the add method to calculate player home run totals by using the fill_value argument to insert a zero for home runs where labels do not overlap:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
With this, we can do all sorts of custom indexing:
Can you write Python code for it?

frame.ix['a']['Ohio']

frame.ix['b', 2]['Colorado']


I am working on the file "Day_01/01_Pandas/1. Introduction to Pandas.ipynb".
The first task is:
Writing Data to Files
As well as being able to read several data input formats, Pandas can also export data to a variety of storage formats. We will bring your attention to just a couple of these.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
A common data analysis procedure is the split-apply-combine operation, which groups subsets of data together, applies a function to each of the groups, then recombines them into a new data table.
For example, we may want to aggregate our data with with some function.

*(figure taken from "Python for Data Analysis", p.251)*

We can aggregate in Pandas using the aggregate (or agg, for short) method:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
A related function qcut uses empirical quantiles to divide the data. If, for example, we want the quartiles -- (0-25%], (25-50%], (50-70%], (75-100%] -- we can just specify 4 intervals, which will be equally-spaced by default:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
A slightly cleaner way of doing this is to set the patient-level information as an index before unstacking:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Alternately, if we simply want to replace particular values in a Series or DataFrame, we can use the replace method.
An example where replacement is useful is dealing with zeros in certain transformations. For example, if we try to take the log of a set of values:
Can you write Python code for it?

vals = pd.Series([float(i)**10 for i in range(10)])
vals

np.log(vals)


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Apply
We can generalize the split-apply-combine methodology by using apply function. This allows us to invoke any function we wish on a grouped dataset and recombine them into a DataFrame.
The function below takes a DataFrame and a column name, sorts by the column, and takes the n largest values of that column. We can use this with apply to return the largest values from every group in a DataFrame in a single call.
Can you write Python code for it?

def top(df, column, n=5):
    return df.sort_index(by=column, ascending=False)[:n]


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
By default, groupby groups by row, but we can specify the axis argument to change this. For example, we can group our columns by type this way:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Compare this ordering with the original:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Concatenating along axis=1 will concatenate column-wise, but respecting the indices of the two DataFrames.
Can you write Python code for it?

pd.concat([mb1, mb2], axis=1).shape

pd.concat([mb1, mb2], axis=1).head()

pd.concat([mb1, mb2], axis=1).values[:5]


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Data transformation
There are a slew of additional operations for DataFrames that we would collectively refer to as "transformations" that include tasks such as removing duplicate values, replacing values, and grouping values.
Dealing with duplicates
We can easily identify and remove duplicate values from DataFrame objects. For example, say we want to removed ships from our vessels dataset that have the same name:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Discretization
Pandas' cut function can be used to group continuous or countable data in to bins. Discretization is generally a very bad idea for statistical analysis, so use this function responsibly!
Lets say we want to bin the ages of the cervical dystonia patients into a smaller number of groups:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Exercise
Load the dataset in titanic.xls. It contains data on all the passengers that travelled on the Titanic.
Can you write Python code for it?

from IPython.core.display import HTML
HTML(filename='data/titanic.html')


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
For a simple cross-tabulation of group frequencies, the crosstab function (not a method) aggregates counts of data according to factors in rows and columns. The factors may be hierarchical if desired.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
For example, we might be interested in the distribution of transit lengths, so we can plot them as a histogram:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
For this dataset, it makes sense to create a hierarchical index based on the patient and observation:
Can you write Python code for it?

cdystonia2 = cdystonia.set_index(['patient','obs'])
cdystonia2.head()

cdystonia2.index.is_unique


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Having a custom data type for dates and times is convenient because we can perform operations on them easily. For example, we may want to calculate the difference between two times:
Can you write Python code for it?

my_age = now - datetime(1970, 9, 3)
my_age

my_age.days/365.


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
If we omit the values argument, we get a DataFrame with hierarchical columns, just as when we applied unstack to the hierarchically-indexed table:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
If we want to transform this data so that repeated measurements are in columns, we can unstack the twstrs measurements according to obs.
Can you write Python code for it?

twstrs_wide = cdystonia2['twstrs'].unstack('obs')
twstrs_wide.head()

cdystonia_long = cdystonia[['patient','site','id','treat','age','sex']].drop_duplicates().merge(
                    twstrs_wide, right_index=True, left_on='patient', how='inner').head()
cdystonia_long


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
If we wanted to use the second table to fill values absent from the first table, we could use combine_first.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
If we wish, we can easily aggregate according to multiple keys:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
If you simply want to divide your DataFrame into chunks for later use, its easy to convert them into a dict so that they can be easily indexed out as needed:
Can you write Python code for it?

chunks = dict(list(cdystonia_grouped))

chunks[4]


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
If you want concat to work like numpy.concatanate, you may provide the ignore_index=True argument.
Exercise
In the data/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each. Write code that imports each of the data spreadsheets and combines them into a single DataFrame, adding the identifying information from the metadata spreadsheet as columns in the combined DataFrame.
Reshaping DataFrame objects
In the context of a single DataFrame, we are often interested in re-arranging the layout of our data.
This dataset in from Table 6.9 of Statistical Methods for the Analysis of Repeated Measurements by Charles S. Davis, pp. 161-163 (Springer, 2002). These data are from a multicenter, randomized controlled trial of botulinum toxin type B (BotB) in patients with cervical dystonia from nine U.S. sites.

Randomized to placebo (N=36), 5000 units of BotB (N=36), 10,000 units of BotB (N=37)
Response variable: total score on Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), measuring severity, pain, and disability of cervical dystonia (high scores mean more impairment)
TWSTRS measured at baseline (week 0) and weeks 2, 4, 8, 12, 16 after treatment began
Can you write Python code for it?

cdystonia = pd.read_csv("data/cdystonia.csv", index_col=None)
cdystonia.head()


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
In addition to datetime there are simpler objects for date and time information only, respectively.
Can you write Python code for it?

from datetime import date, time

time(3, 24)

date(1970, 9, 3)


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
In this case, the default inner join is suitable; we are not interested in observations from either table that do not have corresponding entries in the other.
Notice that mmsi field that was an index on the vessels table is no longer an index on the merged table.
Here, we used the merge function to perform the merge; we could also have used the merge method for either of the tables:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
In this section, we will manipulate data collected from ocean-going vessels on the eastern seaboard. Vessel operations are monitored using the Automatic Identification System (AIS), a safety at sea navigation technology which vessels are required to maintain and that uses transponders to transmit very high frequency (VHF) radio signals containing static information including ship name, call sign, and country of origin, as well as dynamic information unique to a particular voyage such as vessel location, heading, and speed.
The International Maritime Organization’s (IMO) International Convention for the Safety of Life at Sea requires functioning AIS capabilities on all vessels 300 gross tons or greater and the US Coast Guard requires AIS on nearly all vessels sailing in U.S. waters. The Coast Guard has established a national network of AIS receivers that provides coverage of nearly all U.S. waters. AIS signals are transmitted several times each minute and the network is capable of handling thousands of reports per minute and updates as often as every two seconds. Therefore, a typical voyage in our study might include the transmission of hundreds or thousands of AIS encoded signals. This provides a rich source of spatial data that includes both spatial and temporal information.
For our purposes, we will use summarized data that describes the transit of a given vessel through a particular administrative area. The data includes the start and end time of the transit segment, as well as information about the speed of the vessel, how far it travelled, etc.
Can you write Python code for it?

segments = pd.read_csv("data/AIS/transit_segments.csv")
segments.head()


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Inidcator variables
For some statistical analyses (e.g. regression models or analyses of variance), categorical or group variables need to be converted into columns of indicators--zeros and ones--to create a so-called design matrix. The Pandas function get_dummies (indicator variables are also known as dummy variables) makes this transformation straightforward.
Let's consider the DataFrame containing the ships corresponding to the transit segments on the eastern seaboard. The type variable denotes the class of vessel; we can create a matrix of indicators for this. For simplicity, lets filter out the 5 most common types of ships:
Can you write Python code for it?

top5 = vessels.type.apply(lambda s: s in vessels.type.value_counts().index[:5])
vessels5 = vessels[top5]

pd.get_dummies(vessels5.type).head(10)


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Its also possible to group by one or more levels of a hierarchical index. Recall cdystonia2, which we created with a hierarchical index:
Can you write Python code for it?

cdystonia2.head(10)

cdystonia2.groupby(level='obs', axis=0)['twstrs'].mean()


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Let's transform these data into decades, beginnnig with individuals in their 20's and ending with those in their 90's:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Note that you can easily combine discretiztion with the generation of indicator variables shown above:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Notice that additional arguments for the applied function can be passed via apply after the function name. It assumes that the DataFrame is the first argument.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Notice that the treat and sex variables are not included in the aggregation. Since it does not make sense to aggregate non-string variables, these columns are simply ignored by the method.
Some aggregation functions are so common that Pandas has a convenience method for them, such as mean:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Notice that without any information about which column to use as a key, Pandas did the right thing and used the id column in both tables. Unless specified otherwise, merge will used any common column names as keys for merging the tables.
Notice also that id=3 from df1 was omitted from the merged table. This is because, by default, merge performs an inner join on the tables, meaning that the merged table represents an intersection of the two tables.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Occasionally, there will be fields with the same in both tables that we do not wish to use to join the tables; they may contain different information, despite having the same name. In this case, Pandas will by default append suffixes _x and _y to the columns to uniquely identify them.
Can you write Python code for it?

segments['type'] = 'foo'
pd.merge(vessels, segments, left_index=True, right_on='mmsi').head()


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Our first order of business will be to convert these data to datetime. The strptime method parses a string representation of a date and/or time field, according to the expected format of this information.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Pandas also has a custom NA value for missing datetime objects, NaT.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Permutation and sampling
For some data analysis tasks, such as simulation, we need to be able to randomly reorder our data, or draw random values from it. Calling NumPy's permutation function with the length of the sequence you want to permute generates an array with a permuted sequence of integers, which can be used to re-order the sequence.
Can you write Python code for it?

new_order = np.random.permutation(len(segments))
new_order[:30]


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Recall the microbiome data sets that we used previously for the concatenation example. Suppose that we wish to aggregate the data at a higher biological classification than genus. For example, we can identify samples down to class, which is the 3rd level of organization in each index.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Since the data are now ordinal, rather than numeric, we can give them labels:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
The add_prefix and add_suffix methods can be used to give the columns of the resulting table labels that reflect the transformation:
Can you write Python code for it?

cdystonia_grouped.mean().add_suffix('_mean').head()

# The median of the `twstrs` variable
cdystonia_grouped['twstrs'].quantile(0.5)


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
The dateutil package includes a parser that attempts to detect the format of the date strings, and convert them automatically.
Can you write Python code for it?

from dateutil.parser import parse

parse(segments.st_time.ix[0])


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
The outer join above yields the union of the two tables, so all rows are represented, with missing values inserted as appropriate. One can also perform right and left joins to include all rows of the right or left table (i.e. first or second argument to merge), but not necessarily the other.
Looking at the two datasets that we wish to merge:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
The parentheses indicate an open interval, meaning that the interval includes values up to but not including the endpoint, whereas the square bracket is a closed interval, where the endpoint is included in the interval. We can switch the closure to the left side by setting the right flag to False:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
This grouped dataset is hard to visualize
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
This operation is also called binding or stacking.
With Pandas' indexed data structures, there are additional considerations as the overlap in index values between two data structures affects how they are concatenate.
Lets import two microbiome datasets, each consisting of counts of microorganiams from a particular patient. We will use the first column of each dataset as the index.
Can you write Python code for it?

mb1 = pd.read_excel('data/microbiome/MID1.xls', 'Sheet 1', index_col=0, header=None)
mb2 = pd.read_excel('data/microbiome/MID2.xls', 'Sheet 1', index_col=0, header=None)
mb1.shape, mb2.shape

mb1.head()


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
To complement this, unstack pivots from rows back to columns.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
To convert our "wide" format back to long, we can use the melt function, appropriately parameterized:
Can you write Python code for it?

pd.melt(cdystonia_long, id_vars=['patient','site','id','treat','age','sex'],
        var_name='obs', value_name='twsters').head()


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
To see this in action, consider the vessel transit segments dataset (which we merged with the vessel information to yield segments_merged). Say we wanted to return the 3 longest segments travelled by each ship:
Can you write Python code for it?

top3segments = segments_merged.groupby('mmsi').apply(top, column='seg_length', n=3)[['names', 'seg_length']]
top3segments


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Using the string methods split and join we can create an index that just uses the first three classifications: domain, phylum and class.
Can you write Python code for it?

class_index = mb1.index.map(lambda x: ' '.join(x.split(' ')[:3]))

mb_class = mb1.copy()
mb_class.index = class_index


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Using this sequence as an argument to the take method results in a reordered DataFrame:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
Value replacement
Frequently, we get data columns that are encoded as strings that we wish to represent numerically for the purposes of including it in a quantitative analysis. For example, consider the treatment variable in the cervical dystonia dataset:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
We can also perform the same replacement that we used map for with replace:
Can you write Python code for it?

cdystonia2.treat.replace({'Placebo': 0, '5000U': 1, '10000U': 2})


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
We can convert all the dates in a particular column by using the apply method.
Can you write Python code for it?

segments.st_time.apply(lambda d: datetime.strptime(d, '%m/%d/%y %H:%M'))


I am working on the file "Day_01/01_Pandas/2. Data Wrangling with Pandas.ipynb".
The first task is:
We can see that although there are date/time fields in the dataset, they are not in any specialized format, such as datetime.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
A density plot is similar to a histogram in that it describes the distribution of the underlying data, but rather than being a pure empirical representation, it is an estimate of the underlying "true" distribution. As a result, it is smoothed into a continuous line plot. We create them in Pandas using the plot method with kind='kde', where kde stands for kernel density estimate.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Another way of comparing the groups is to look at the survival rate, by adjusting for the number of people in each group.
Can you write Python code for it?

death_counts.div(death_counts.sum(1).astype(float), axis=0).plot(kind='barh', stacked=True, color=['black','gold'])


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
As an illustration of the high-level nature of Pandas plots, we can split multiple series into subplots with a single argument for plot:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Bar plots
Bar plots are useful for displaying and comparing measurable quantities, such as counts or volumes. In Pandas, we just use the plot method with a kind='bar' argument.
For this series of examples, let's load up the Titanic dataset:
Can you write Python code for it?

titanic = pd.read_excel("data/titanic.xls", "titanic")
titanic.head()

titanic.groupby('pclass').survived.sum().plot(kind='bar')

titanic.groupby(['sex','pclass']).survived.sum().plot(kind='barh')

death_counts = pd.crosstab([titanic.pclass, titanic.sex], titanic.survived.astype(bool))
death_counts.plot(kind='bar', stacked=True, color=['black','gold'], grid=False)


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Here, we had to normalize the histogram (normed=True), since the kernel density is normalized by definition (it is a probability distribution).
We will explore kernel density estimates more in the next section.
Boxplots
A different way of visualizing the distribution of data is the boxplot, which is a display of common quantiles; these are typically the quartiles and the lower and upper 5 percent values.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
If we would like a little more control, we can use matplotlib's subplots function directly, and manually assign plots to its axes:
Can you write Python code for it?

fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i,var in enumerate(['normal','gamma','poisson']):
    variables[var].cumsum(0).plot(ax=axes[i], title=var)
axes[0].set_ylabel('cumulative sum')


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Notice that by default a line plot is drawn, and a light grid is included. All of this can be changed, however:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Or, we may want to have some series displayed on the secondary y-axis, which can allow for greater detail and less empty space:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Scatterplots are useful for data exploration, where we seek to uncover relationships among variables. There are no scatterplot methods for Series or DataFrame objects; we must instead use the matplotlib function scatter.
Can you write Python code for it?

plt.scatter(baseball.ab, baseball.h)
xlim(0, 700); ylim(0, 200)


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
The above plot simply shows two sets of random numbers taken from a normal distribution plotted against one another. The 'ro' argument is a shorthand argument telling matplotlib that I wanted the points represented as red circles.
This plot was expedient. We can exercise a little more control by breaking the plotting into a workflow:
Can you write Python code for it?

with mpl.rc_context(rc={'font.family': 'serif', 'font.weight': 'bold', 'font.size': 8}):
    fig = plt.figure(figsize=(6,3))
    ax1 = fig.add_subplot(121)
    ax1.set_xlabel('some random numbers')
    ax1.set_ylabel('more random numbers')
    ax1.set_title("Random scatterplot")
    plt.plot(np.random.normal(size=100), np.random.normal(size=100), 'r.')
    ax2 = fig.add_subplot(122)
    plt.hist(np.random.normal(size=100), bins=15)
    ax2.set_xlabel('sample')
    ax2.set_ylabel('cumulative sum')
    ax2.set_title("Normal distrubution")
    plt.tight_layout()
    plt.savefig("normalvars.png", dpi=150)


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
There are algorithms for determining an "optimal" number of bins, each of which varies somehow with the number of observations in the data series.
Can you write Python code for it?

sturges = lambda n: int(log2(n) + 1)
square_root = lambda n: int(sqrt(n))
from scipy.stats import kurtosis
doanes = lambda data: int(1 + log(len(data)) + log(1 + kurtosis(data) * (len(data) / 6.) ** 0.5))

n = len(titanic)
sturges(n), square_root(n), doanes(titanic.fare.dropna())

titanic.fare.hist(bins=doanes(titanic.fare.dropna()))


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Trellis Plots
One of the enduring strengths of carrying out statistical analyses in the R language is the quality of its graphics. In particular, the addition of Hadley Wickham's ggplot2 package allows for flexible yet user-friendly generation of publication-quality plots. Its srength is based on its implementation of a powerful model of graphics, called the Grammar of Graphics (GofG). The GofG is essentially a theory of scientific graphics that allows the components of a graphic to be completely described. ggplot2 uses this description to build the graphic component-wise, by adding various layers.
Pandas recently added functions for generating graphics using a GofG approach. Chiefly, this allows for the easy creation of trellis plots, which are a faceted graphic that shows relationships between two variables, conditioned on particular values of other variables. This allows for the representation of more than two dimensions of information without having to resort to 3-D graphics, etc.
Let's use the titanic dataset to create a trellis plot that represents 4 variables at a time. This consists of 4 steps:

Create a RPlot object that merely relates two variables in the dataset
Add a grid that will be used to condition the variables by both passenger class and sex
Add the actual plot that will be used to visualize each comparison
Draw the visualization
Can you write Python code for it?

from pandas.tools.rplot import *

titanic = titanic[titanic.age.notnull() & titanic.fare.notnull()]

tp = RPlot(titanic, x='age')
tp.add(TrellisGrid(['pclass', 'sex']))
tp.add(GeomDensity())
_ = tp.render(gcf())


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Using the cervical dystonia dataset, we can simultaneously examine the relationship between age and the primary outcome variable as a function of both the treatment received and the week of the treatment by creating a scatterplot of the data, and fitting a polynomial relationship between age and twstrs:
Can you write Python code for it?

cdystonia = pd.read_csv("data/cdystonia.csv", index_col=None)
cdystonia.head()

plt.figure(figsize=(12,12))
bbp = RPlot(cdystonia, x='age', y='twstrs')
bbp.add(TrellisGrid(['week', 'treat']))
bbp.add(GeomScatter())
bbp.add(GeomPolyFit(degree=2))
_ = bbp.render(gcf())


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
We can add additional information to scatterplots by assigning variables to either the size of the symbols or their colors.
Can you write Python code for it?

plt.scatter(baseball.ab, baseball.h, s=baseball.hr*10, alpha=0.5)
xlim(0, 700); ylim(0, 200)

plt.scatter(baseball.ab, baseball.h, c=baseball.hr, s=40, cmap='hot')
xlim(0, 700); ylim(0, 200);


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
When data are dense, a couple of tricks used above help the visualization:

reducing the alpha level to make the points partially transparent
adding random "jitter" along the x-axis to avoid overstriking

A related but inferior cousin of the box plot is the so-called dynamite plot, which is just a bar chart with half of an error bar.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
Why is this plot a poor choice?

bar charts should be used for measurable quantities (e.g. raw data), not estimates. The area of the bar does not represent anything, since these are estimates derived from the data.
the "data-ink ratio" (sensu Edward Tufte) is very high. There are only 6 values represented here (3 means and 3 standard deviations).
the plot hides the underlying data.

A boxplot is always a better choice than a dynamite plot.
Can you write Python code for it?

data1 = [150, 155, 175, 200, 245, 255, 395, 300, 305, 320, 375, 400, 420, 430, 440]
data2 = [225, 380]

fake_data = pd.DataFrame([data1, data2]).transpose()
p = fake_data.mean().plot(kind='bar', yerr=fake_data.std(), grid=False)

fake_data = pd.DataFrame([data1, data2]).transpose()
p = fake_data.mean().plot(kind='bar', yerr=fake_data.std(), grid=False)
x1, x2 = p.xaxis.get_majorticklocs()
plot(np.random.normal(x1, 0.01, size=len(data1)), data1, 'ro')
plot([x2]*len(data2), data2, 'ro')


I am working on the file "Day_01/01_Pandas/3. Plotting and Visualization.ipynb".
The first task is:
You can think of the box plot as viewing the distribution from above. The blue crosses are "outlier" points that occur outside the extreme quantiles.
One way to add additional information to a boxplot is to overlay the actual data; this is generally most suitable with small- or moderate-sized data series.
Can you write Python code for it?

bp = titanic.boxplot(column='age', by='pclass', grid=False)
for i in [1,2,3]:
    y = titanic.age[titanic.pclass==i].dropna()
    # Add some random "jitter" to the x-axis
    x = np.random.normal(i, 0.04, size=len(y))
    plot(x, y, 'r.', alpha=0.2)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
... and fit the model.
Can you write Python code for it?

b0,b1 = fmin(logistic_like, [0.5,0], args=(x,y))
b0, b1

jitter = np.random.normal(scale=0.01, size=len(x))
plot(x, y+jitter, 'r.', alpha=0.3)
yticks([0,.25,.5,.75,1])
xvals = np.linspace(0, 600)
plot(xvals, invlogit(b0+b1*xvals))


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Although polynomial model characterizes a nonlinear relationship, it is a linear problem in terms of estimation. That is, the regression model $f(y | x)$ is linear in the parameters.
For some data, it may be reasonable to consider polynomials of order>2. For example, consider the relationship between the number of home runs a baseball player hits and the number of runs batted in (RBI) they accumulate; clearly, the relationship is positive, but we may not expect a linear relationship.
Can you write Python code for it?

ss3 = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2) - theta[3]*(x**3)) ** 2)

bb = pd.read_csv("data/baseball.csv", index_col=0)
plot(bb.hr, bb.rbi, 'r.')
b0,b1,b2,b3 = fmin(ss3, [0,1,-1,0], args=(bb.hr, bb.rbi))
xvals = arange(40)
plot(xvals, b0 + b1*xvals + b2*(xvals**2) + b3*(xvals**3))


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
And here's the logit function:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
And now plug this back into the solution for beta:
<div style="font-size: 120%;">
$$ \beta  = \frac{\alpha}{\bar{X}} $$
Can you write Python code for it?

beta_mle = alpha_mle/precip.mean()[-1]
beta_mle


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Bootstrap Estimates
From our bootstrapped samples, we can extract estimates of the expectation and its variance:
$$\bar{T}^ = \hat{E}(T^) = \frac{\sum_i T_i^*}{R}$$
$$\hat{\text{Var}}(T^) = \frac{\sum_i (T_i^ - \bar{T}^*)^2}{R-1}$$
Can you write Python code for it?

boot_mean = np.sum(s)/R
boot_mean

boot_var = ((np.array(s) - boot_mean) ** 2).sum() / (R-1)
boot_var


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Exercise: Cervical dystonia analysis
Recall the cervical dystonia database, which is a clinical trial of botulinum toxin type B (BotB) for patients with cervical dystonia from nine U.S. sites. The response variable is measurements on the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), measuring severity, pain, and disability of cervical dystonia (high scores mean more impairment). One way to check the efficacy of the treatment is to compare the distribution of TWSTRS for control and treatment patients at the end of the study.
Use the method of moments or MLE to calculate the mean and variance of TWSTRS at week 16 for one of the treatments and the control group. Assume that the distribution of the twstrs variable is normal:
$$f(x \mid \mu, \sigma^2) = \sqrt{\frac{1}{2\pi\sigma^2}} \exp\left{ -\frac{1}{2} \frac{(x-\mu)^2}{\sigma^2} \right}$$
Can you write Python code for it?

cdystonia = pd.read_csv("data/cdystonia.csv")
cdystonia[cdystonia.obs==6].hist(column='twstrs', by=cdystonia.treat, bins=8)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Exercise: Polynomial function
Write a function that specified a polynomial of arbitrary degree.
Model Selection
How do we choose among competing models for a given dataset? More parameters are not necessarily better, from the standpoint of model fit. For example, fitting a 9-th order polynomial to the sample data from the above example certainly results in an overfit.
Can you write Python code for it?

def calc_poly(params, data):
        x = np.c_[[data**i for i in range(len(params))]]
        return np.dot(params, x)

ssp = lambda theta, x, y: np.sum((y - calc_poly(theta, x)) ** 2)
betas = fmin(ssp, np.zeros(10), args=(x,y), maxiter=1e6)
plot(x, y, 'ro')
xvals = np.linspace(0, max(x), 100)
plot(xvals, calc_poly(betas, xvals))


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Exercise: multivariate logistic regression
Which other variables might be relevant for predicting the probability of surviving the Titanic? Generalize the model likelihood to include 2 or 3 other covariates from the dataset.
Bootstrapping
Parametric inference can be non-robust:

inaccurate if parametric assumptions are violated
if we rely on asymptotic results, we may not achieve an acceptable level of accuracy

Parmetric inference can be difficult:

derivation of sampling distribution may not be possible

An alternative is to estimate the sampling distribution of a statistic empirically without making assumptions about the form of the population.
We have seen this already with the kernel density estimate.
Non-parametric Bootstrap
The bootstrap is a resampling method discovered by Brad Efron that allows one to approximate the true sampling distribution of a dataset, and thereby obtain estimates of the mean and variance of the distribution.
Bootstrap sample:
<div style="font-size: 120%;">
$$S_1^* = \{x_{11}^*, x_{12}^*, \ldots, x_{1n}^*\}$$
</div>

$S_i^$ is a sample of size $n$, with* replacement.
In Python, we have already seen the NumPy function permutation that can be used in conjunction with Pandas' take method to generate a random sample of some data without replacement:
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
For some common distributions, SciPy includes methods for fitting via MLE:
Can you write Python code for it?

from scipy.stats import gamma

gamma.fit(precip.Dec)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Hence, we would select the 2-parameter (linear) model.
Logistic Regression
Fitting a line to the relationship between two variables using the least squares approach is sensible when the variable we are trying to predict is continuous, but what about when the data are dichotomous?

male/female
pass/fail
died/survived

Let's consider the problem of predicting survival in the Titanic disaster, based on our available information. For example, lets say that we want to predict survival as a function of the fare paid for the journey.
Can you write Python code for it?

titanic = pd.read_excel("data/titanic.xls", "titanic")
titanic.name

jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
yticks([0,1])
ylabel("survived")
xlabel("log(fare)")


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
How is the likelihood function different than the probability distribution function (PDF)? The likelihood is a function of the parameter(s) given the data, whereas the PDF returns the probability of data given a particular parameter value. Here is the PDF of the Poisson for $\lambda=5$.
Can you write Python code for it?

lam = 5
xvals = arange(15)
plt.bar(xvals, [poisson_like(x, lam) for x in xvals])
xlabel('x')
ylabel('Pr(X|$\lambda$=5)')


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
I have added random jitter on the y-axis to help visualize the density of the points, and have plotted fare on the log scale.
Clearly, fitting a line through this data makes little sense, for several reasons. First, for most values of the predictor variable, the line would predict values that are not zero or one. Second, it would seem odd to choose least squares (or similar) as a criterion for selecting the best line.
Can you write Python code for it?

x = np.log(titanic.fare[titanic.fare>0])
y = titanic.survived[titanic.fare>0]
betas_titanic = fmin(ss, [1,1], args=(x,y))

jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
yticks([0,1])
ylabel("survived")
xlabel("log(fare)")
plt.plot([0,7], [betas_titanic[0], betas_titanic[0] + betas_titanic[1]*7.])


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
If we look at this data, we can see that for most values of fare, there are some individuals that survived and some that did not. However, notice that the cloud of points is denser on the "survived" (y=1) side for larger values of fare than on the "died" (y=0) side.
Stochastic model
Rather than model the binary outcome explicitly, it makes sense instead to model the probability of death or survival in a stochastic model. Probabilities are measured on a continuous [0,1] scale, which may be more amenable for prediction using a regression line. We need to consider a different probability model for this exerciese however; let's consider the Bernoulli distribution as a generative model for our data:

$$f(y|p) = p^{y} (1-p)^{1-y}$$

where $y = {0,1}$ and $p \in [0,1]$. So, this model predicts whether $y$ is zero or one as a function of the probability $p$. Notice that when $y=1$, the $1-p$ term disappears, and when $y=0$, the $p$ term disappears.
So, the model we want to fit should look something like this:

$$p_i = \beta_0 + \beta_1 x_i + \epsilon_i$$

However, since $p$ is constrained to be between zero and one, it is easy to see where a linear (or polynomial) model might predict values outside of this range. We can modify this model sligtly by using a **link function** to transform the probability to have an unbounded range on a new scale. Specifically, we can use a **logit transformation** as our link function:

$$\text{logit}(p) = \log\left[\frac{p}{1-p}\right] = x$$

Here's a plot of $p/(1-p)$
Can you write Python code for it?

logit = lambda p: np.log(p/(1.-p))
unit_interval = np.linspace(0,1)
plt.plot(unit_interval/(1-unit_interval), unit_interval)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
In general, simulating data is a terrific way of testing your model before using it with real data.
Kernel density estimates
In some instances, we may not be interested in the parameters of a particular distribution of data, but just a smoothed representation of the data at hand. In this case, we can estimate the disribution non-parametrically (i.e. making no assumptions about the form of the underlying distribution) using kernel density estimation.
Can you write Python code for it?

# Some random data
y = np.random.random(15) * 10
y

x = np.linspace(0, 10, 100)
# Smoothing parameter
s = 0.4
# Calculate the kernels
kernels = np.transpose([norm.pdf(x, yi, s) for yi in y])
plot(x, kernels, 'k:')
plot(x, kernels.sum(1))
plot(y, np.zeros(len(y)), 'ro', ms=10)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Looping over all months, we can create a grid of plots for the distribution of rainfall, using the gamma distribution:
Can you write Python code for it?

axs = precip.hist(normed=True, figsize=(12, 8), sharex=True, sharey=True, bins=15, grid=False)

for ax in axs.ravel():

    # Get month
    m = ax.get_title()

    # Plot fitted distribution
    x = linspace(*ax.get_xlim())
    ax.plot(x, gamma.pdf(x, alpha_mom[m], beta_mom[m]))

    # Annotate with parameter estimates
    label = 'alpha = {0:.2f}\nbeta = {1:.2f}'.format(alpha_mom[m], beta_mom[m])
    ax.annotate(label, xy=(10, 0.2))

tight_layout()


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Of course, we need not fit least squares models by hand. The statsmodels package implements least squares models that allow for model fitting in a single line:
Can you write Python code for it?

import statsmodels.api as sm

straight_line = sm.OLS(y, sm.add_constant(x)).fit()
straight_line.summary()

from statsmodels.formula.api import ols as OLS

data = pd.DataFrame(dict(x=x, y=y))
cubic_fit = OLS('y ~ x + I(x**2)', data).fit()

cubic_fit.summary()


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
One approach is to use an information-theoretic criterion to select the most appropriate model. For example Akaike's Information Criterion (AIC) balances the fit of the model (in terms of the likelihood) with the number of parameters required to achieve that fit. We can easily calculate AIC as:
$$AIC = n \log(\hat{\sigma}^2) + 2p$$
where $p$ is the number of parameters in the model and $\hat{\sigma}^2 = RSS/(n-p-1)$.
Notice that as the number of parameters increase, the residual sum of squares goes down, but the second term (a penalty) increases.
To apply AIC to model selection, we choose the model that has the lowest AIC value.
Can you write Python code for it?

n = len(x)

aic = lambda rss, p, n: n * np.log(rss/(n-p-1)) + 2*p

RSS1 = ss(fmin(ss, [0,1], args=(x,y)), x, y)
RSS2 = ss2(fmin(ss2, [1,1,-1], args=(x,y)), x, y)

print aic(RSS1, 2, n), aic(RSS2, 3, n)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Regression models
A general, primary goal of many statistical data analysis tasks is to relate the influence of one variable on another. For example, we may wish to know how different medical interventions influence the incidence or duration of disease, or perhaps a how baseball player's performance varies as a function of age.
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Remove null values from variables
Can you write Python code for it?

x, y = titanic[titanic.fare.notnull()][['fare', 'survived']].values.T


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Since we have estimated the expectation of the bootstrapped statistics, we can estimate the bias of T:
$$\hat{B}^ = \bar{T}^ - T$$
Can you write Python code for it?


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
The product $\prod_{i=1}^n Pr(y_i | \theta)$ gives us a measure of how likely it is to observe values $y_1,\ldots,y_n$ given the parameters $\theta$. Maximum likelihood fitting consists of choosing the appropriate function $l= Pr(Y|\theta)$ to maximize for a given set of observations. We call this function the likelihood function, because it is a measure of how likely the observations are if the model is true.

Given these data, how likely is this model?

In the above model, the data were drawn from a Poisson distribution with parameter $\lambda =5$.
$$L(y|\lambda=5) = \frac{e^{-5} 5^y}{y!}$$
So, for any given value of $y$, we can calculate its likelihood:
Can you write Python code for it?

poisson_like = lambda x, lam: np.exp(-lam) * (lam**x) / (np.arange(x)+1).prod()

lam = 6
value = 10
poisson_like(value, lam)

np.sum(poisson_like(yi, lam) for yi in y)

lam = 8
np.sum(poisson_like(yi, lam) for yi in y)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
This fit is not directly comparable to our estimates, however, because SciPy's gamma.fit method fits an odd 3-parameter version of the gamma distribution.
Example: truncated distribution
Suppose that we observe $Y$ truncated below at $a$ (where $a$ is known). If $X$ is the distribution of our observation, then:
$$ P(X \le x) = P(Y \le x|Y \gt a) = \frac{P(a \lt Y \le x)}{P(Y \gt a)}$$
(so, $Y$ is the original variable and $X$ is the truncated variable)
Then X has the density:
$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$
Suppose $Y \sim N(\mu, \sigma^2)$ and $x_1,\ldots,x_n$ are independent observations of $X$. We can use maximum likelihood to find $\mu$ and $\sigma$.
First, we can simulate a truncated distribution using a while statement to eliminate samples that are outside the support of the truncated distribution.
Can you write Python code for it?

x = np.random.normal(size=10000)
a = -1
x_small = x < a
while x_small.sum():
    x[x_small] = np.random.normal(size=x_small.sum())
    x_small = x < a

_ = hist(x, bins=100)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
Time to optimize!
Can you write Python code for it?

# Alpha MLE for December
alpha_mle = newton(dlgamma, 2, dl2gamma, args=(log_mean[-1], mean_log[-1]))
alpha_mle


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
To apply the Newton-Raphson algorithm, we need a function that returns a vector containing the first and second derivatives of the function with respect to the variable of interest. In our case, this is:
Can you write Python code for it?

from scipy.special import psi, polygamma

dlgamma = lambda m, log_mean, mean_log: np.log(m) - psi(m) - log_mean + mean_log
dl2gamma = lambda m, *args: 1./m - polygamma(1, m)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
We can construct a log likelihood for this function using the conditional form:
$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$
Can you write Python code for it?

from scipy.stats.distributions import norm

trunc_norm = lambda theta, a, x: -(np.log(norm.pdf(x, theta[0], theta[1])) -
                                      np.log(1 - norm.cdf(a, theta[0], theta[1]))).sum()


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
We regard S as an "estimate" of population P

population : sample :: sample : bootstrap sample

The idea is to generate replicate bootstrap samples:

$$S^* = \{S_1^*, S_2^*, \ldots, S_R^*\}$$

Compute statistic $t$ (estimate) for each bootstrap sample:

$$T_i^* = t(S^*)$$

Can you write Python code for it?

n = 10
R = 1000
# Original sample (n=10)
x = np.random.normal(size=n)
# 1000 bootstrap samples of size 10
s = [x[np.random.randint(0,n,n)].mean() for i in range(R)]
_ = hist(s, bins=30)


I am working on the file "Day_01/01_Pandas/4. Statistical Data Modeling.ipynb".
The first task is:
where log_mean and mean_log are $\log{\bar{x}}$ and $\overline{\log(x)}$, respectively. psi and polygamma are complex functions of the Gamma function that result when you take first and second derivatives of that function.
Can you write Python code for it?

# Calculate statistics
log_mean = precip.mean().apply(log)
mean_log = precip.apply(log).mean()


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
And this same type of file can then be read with the matching np.loadtxt function:
Can you write Python code for it?

arr2 = np.loadtxt('test.out')
print(arr2)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Array Computations
Can you write Python code for it?

b = 3*a - 1    # a is array, b becomes array


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Array creation
Above we created an array from an existing list; now let us now see other ways in which we can create arrays, which we'll illustrate next.  A common need is to have an array initialized with a constant value, and very often this value is 0 or 1 (suitable as starting value for additive and multiplicative loops respectively); zeros creates arrays of all zeros, with any desired dtype:
Can you write Python code for it?

np.zeros((5, 5), dtype=np.float64)

np.zeros((2, 3), dtype=np.int64)

np.zeros(3, dtype=complex)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Array memory representation
The information about the type of an array is contained in its dtype attribute:
Can you write Python code for it?

x = np.array([[1, 2], [3, 4]], dtype=np.uint8)
print(x)

[b for b in bytes(x.data)]

arr = np.array([10, 20, 123123])
arr.dtype


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Arrays with more than one dimension
Most of the array creation methods can be used to construct >1D arrays:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
As you can see in this example, the value of the axis parameter is the dimension which will be consumed once the operation has been carried out.  This is why to sum along the rows we use axis=0.
Another widely used property of arrays is the .T attribute, which allows you to access the transpose of the array (NumPy does this without making a copy of the array, by manipulating its strides):
Can you write Python code for it?

print('Array:\n', arr)
print('Transpose:\n', arr.T)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Elements of a one-dimensional array are accessed with the same syntax as a list:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Exercise 1: Access the final element in the arr array
The first difference to note between lists and arrays is that arrays are homogeneous; i.e. all elements of an array must be of the same type.  In contrast, lists can contain elements of arbitrary type. For example, we can change the last element in our list above to be a string:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Exercise 2: Create a new array of 11 elements logarithmically spaced from 1 to 100:
Finally, it is often useful to create arrays with random numbers that follow a specific distribution.  The np.random module provides several random number generators.  For example, here we produce an array of 5 random samples taken from a standard normal distribution (0 mean and variance 1):
Can you write Python code for it?

rng = np.random.RandomState(0)  # <-- seed value, not needed but useful for reproducibility

rng.normal(loc=0, scale=1, size=5)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Exercise 6: Numerically integrate y = x^2 from x = 0 to 10.  How many samples do you need to reach convergence to 2 decimal places?
Hint: You can use whatever simple method you like: from the left, right or the trapezoid rule. Just recall the approximation for a definite integral from calculus:
$$\int^b_a f(x) \ dx \approx \sum^n_{i=1} f(x_i^*) \ \Delta x_i$$
Universal Functions and Loops
Universal functions run much faster than for loops, which should be avoided whenever possible
Can you write Python code for it?

def mult1(a,b):
    return a * b

def mult2(a,b):
    c = np.empty(a.shape)
    for i in range(a.shape[0]):
        for j in range(a.shape[1]):
            c[i,j] = a[i,j] * b[i,j]
    return c

a = np.random.random((400,400))
b = np.random.random((400,400))

%timeit mult1(a,b)

%timeit mult2(a,b)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
For matrix-matrix multiplication, the same dimension-matching rules must be satisfied, e.g. consider the difference between $A \times A^T$:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
For these methods, the above operations area all computed on all the elements of the array.  But for a multidimensional array, it's possible to do the computation along a single dimension, by passing the axis parameter; for example:
Can you write Python code for it?

print('For the following array:\n', arr)
print('The sum of elements along the rows is    :', np.sum(arr, axis=1))
print('The sum of elements along the columns is :', np.sum(arr, axis=0))


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
In-place operations:
a = 3.0     # multiply a's elements by 3
   a -= 1.0     # subtract 1 from each element
   a /= 3.0     # divide each element by 3
   a += 1.0     # add 1 to each element
   a *= 2.0    # square all elements
Math Functions and Array Arguments
Can you write Python code for it?

# let b be an array

c = np.sin(b)
c = np.arcsin(c)
c = np.sinh(b)
# same functions for the cos and tan families
c = b**2.5  # power function
c = np.log(b)
c = np.exp(b)
c = np.sqrt(b)

# a is an array
a.clip(min=3, max=12)  	# clip elements
a.mean(); np.mean(a)     	# mean value
a.var();  np.var(a)       	# variance
a.std();  np.std(a)        	# standard deviation
np.median(a)
np.trapz(a)              	# Trapezoidal integration
np.diff(a)               	# finite differences (da/dx)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Indexing with other arrays
Above we saw how to index arrays with single numbers and slices, just like Python lists.  But arrays allow for a more sophisticated kind of indexing which is very powerful: you can index an array with another array, and in particular with an array of boolean values.  This is particluarly useful to extract information from an array that matches a certain condition.
Consider for example that in the array uni we want to replace all values above 0 with the value 10.  We can do so by first finding the mask that indicates where this condition is true or false:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Linear algebra in numpy
Numpy ships with a basic linear algebra library, and all arrays have a dot method whose behavior is that of the scalar dot product when its arguments are vectors (one-dimensional arrays) and the traditional matrix multiplication when one or both of its arguments are two-dimensional arrays:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Note, that, unlike in MATLAB, operations are performed element-wise:
Can you write Python code for it?

print(arr1, '*', arr2, '=', arr1 * arr2)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Now let us see how the np.savez_compressed function works.
Can you write Python code for it?

np.savez_compressed('test.npz', first=arr, second=arr2)
arrays = np.load('test.npz')
arrays.files


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Now that we have this mask, we can use it to either read those values or to reset them to 0:
Can you write Python code for it?

print('Array:', uni)
print('Masked array:', uni[mask])

uni[mask] = 0
print(uni)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Once an array has been created, its dtype is fixed and it can only store elements of the same type.  For this example where the dtype is integer, if we store a floating point number it will be automatically converted into an integer:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Or the same, but from a uniform distribution:
Can you write Python code for it?

# 5 random numbers, picked from a uniform distribution between -10 and 10
uni = rng.uniform(-10, 10, size=5)
print(uni)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Other numpy functions and array properties
Now that we have seen how to create arrays with more than one dimension, let's take a look at some other properties.
Can you write Python code for it?

print('Data type                :', arr.dtype)
print('Total number of elements :', arr.size)
print('Number of dimensions     :', arr.ndim)
print('Shape (dimensionality)   :', arr.shape)
print('Memory used (in bytes)   :', arr.nbytes)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Some other example of quick random number generation using numpy
Can you write Python code for it?

print np.random.rand(2,2)
print np.random.randint(2,100,5)
print np.random.randn(5)    # samples 5 times from the standard normal distribution


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
The above operation generates a temporary array:
<OL>
<LI> tb = 3*a
<LI> b = tb - 1
</OL>
As far as possible, we want to avoid the creation
  of temporary arrays to limit the memory usage and
  to decrease the computational time associated with
  with array computations.
In-Place Array Arithmetics
Can you write Python code for it?

b = a
b *= 3  # or multiply(b, 3, b)
b -= 1  # or subtract(b, 1, b)


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
The broadcasting rules
This broadcasting behavior is powerful, especially because when numpy broadcasts to create new dimensions or to 'stretch' existing ones, it doesn't replicate the data.  In the example above the operation is carried as if the 3 was a 1-d array with 3 in all of its entries, but no actual array was ever created.  This can save memory in cases when the arrays in question are large, with significant performance implications.
The general rule is: when operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward, creating dimensions of length 1 as needed. Two dimensions are considered compatible when

they are equal or either is None or one
either dimension is 1 or None, or if dimensions are equal

If these conditions are not met, a ValueError: frames are not aligned exception is thrown, indicating that the arrays have incompatible shapes. The size of the resulting array is the maximum size along each dimension of the input arrays.
Examples below:
```
(9, 5)   (9, 5)   (9, 5)   (9, 1)
   ( )   (9, 1)   (   5)   (   5)
------   ------   ------   ------
(9, 5)   (9, 5)   (9, 5)   (9, 5)
```
<img src="Broadcasting.png"/>
Sketch from Nicolas Rougier's NumPy tutorial
Exercise 5: Use broadcasting to generate a 6x6 array from two one-dimensional arrays.
Sometimes, it is necessary to modify arrays before they can be used together.  Numpy allows you to add new axes to an array by indexing with np.newaxis:
Can you write Python code for it?

c = np.arange(5)
d = np.arange(6)

print(c.shape)
print(d.shape)

c + d

c = np.arange(5)
d = np.arange(6)

print "Before adding a new axis:"
print c

c = c[:, np.newaxis]

print "After adding a new axis:"
print c

print(c.shape)
print('  ', d.shape)
print('-------')
print((c + d).shape)

#   d d d d d d
#
# c              c c c c c c   d d d d d d
# c              c c c c c c   d d d d d d
# c     +      = c c c c c c + d d d d d d
# c              c c c c c c   d d d d d d
# c              c c c c c c   d d d d d d

c + d


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
The primary building block of the numpy module is the class "ndarray". A ndarray object represents a multidimensional, homogeneous array of fixed-sized items. An associated date-type object describes the format of each element in the array. An ndarray object is (almost) never instantiated directly, but instead using a method that returns an instance of the class.  Here is a pictoral representation of an "ndarray":
<img src="http://docs.scipy.org/doc/numpy-1.10.1/_images/threefundamental.png">
Arrays vs. Lists
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
There are also many useful functions in numpy that operate on arrays, e.g.:
Can you write Python code for it?

print('Minimum and maximum             :', np.min(arr), np.max(arr))
print('Sum and product of all elements :', np.sum(arr), np.prod(arr))
print('Mean and standard deviation     :', np.mean(arr), np.std(arr))


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
We don't have time here to look at all the numpy functions that operate on arrays, but here's a complete list.  Simply try exploring some of these IPython to learn more, or read their description in their docstrings or the Numpy documentation:
np.ALLOW_THREADS              np.compress                   np.irr                        np.pv
np.BUFSIZE                    np.concatenate                np.is_busday                  np.r_
np.CLIP                       np.conj                       np.isclose                    np.rad2deg
np.ComplexWarning             np.conjugate                  np.iscomplex                  np.radians
np.DataSource                 np.convolve                   np.iscomplexobj               np.random
np.ERR_CALL                   np.copy                       np.isfinite                   np.rank
np.ERR_DEFAULT                np.copysign                   np.isfortran                  np.rate
np.ERR_IGNORE                 np.copyto                     np.isinf                      np.ravel
np.ERR_LOG                    np.core                       np.isnan                      np.ravel_multi_index
np.ERR_PRINT                  np.corrcoef                   np.isneginf                   np.real
np.ERR_RAISE                  np.correlate                  np.isposinf                   np.real_if_close
np.ERR_WARN                   np.cos                        np.isreal                     np.rec
np.FLOATING_POINT_SUPPORT     np.cosh                       np.isrealobj                  np.recarray
np.FPE_DIVIDEBYZERO           np.count_nonzero              np.isscalar                   np.recfromcsv
np.FPE_INVALID                np.cov                        np.issctype                   np.recfromtxt
np.FPE_OVERFLOW               np.cross                      np.issubclass_                np.reciprocal
np.FPE_UNDERFLOW              np.csingle                    np.issubdtype                 np.record
np.False_                     np.ctypeslib                  np.issubsctype          ...
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
While this means that, in principle, arrays must always match in their dimensionality in order for an operation to be valid, numpy will broadcast dimensions when possible.  For example, suppose that you want to add the number 3 to arr1, this works:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
Why is a homogeneous data type required for arrays? Speed
Can you write Python code for it?

x = range(50000)
y = np.arange(50000)

%timeit [e**2  for e in x]
%timeit y**2


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
and $A^T \times A$:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/00_Numpy.ipynb".
The first task is:
and similarly for ones:
Can you write Python code for it?


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Annotating Text
Default options
Can you write Python code for it?

t = np.arange(0.0, 5.0, 0.01)
s = np.cos(2*np.pi*t)
# use 'lw' instead of 'linewidth'
plt.plot(t, s, lw=1.5)

# put annotation on to plot
plt.annotate('local max of $\cos(t)$', \
             xy=(2, 1), \
             xytext=(3, 1.5),\
             arrowprops=dict(facecolor='black'))
# change y limits for y-axis
plt.ylim(-2,2);


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Change Axes Limits
Can you write Python code for it?

# add labels to the curves, can use LaTeX
plt.plot(x, y1, label=r'$\sin(x)$')
plt.plot(x, y2, label=r'$\sin(x^2)$')
plt.title('functions: $\sin(x)$ and $\sin(x^2)$')

plt.xlabel('x')
plt.ylabel('y')
plt.grid()

# change axes limits
plt.xlim(-0.1,6.5)
plt.ylim(-1.1,1.1)

# add legend; loc=0 "best" placement
plt.legend(loc=0)


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Change linestyle
help for plot()
function - line style options listed here
Can you write Python code for it?

# short form of linestyle specification
plt.plot(t, s1,'k-', linewidth=1)

# long form of linestyle specification
plt.plot(t, s2, linestyle='dashed',color='r', linewidth=1)

plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('About as simple as it gets, folks')
plt.grid(True)


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Fine controll of text and arrow appearance
Can you write Python code for it?

t = np.arange(0.0, 5.0, 0.01)
s = np.cos(2*np.pi*t)
# use 'lw' instead of 'linewidth'
plt.plot(t, s, lw=1.5)

# put annotation on to plot
# fine control of text and arrow properties
plt.annotate('local max of $\cos(t)$', \
             xy=(2, 1), \
             xytext=(3, 1.5), \
             arrowprops=dict(facecolor='black', shrink=0.05,width=1,headwidth=6), \
            fontsize='x-large',fontname="serif")

# change y limits for y-axis
plt.ylim(-2,2);


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Log Plot
Can you write Python code for it?

plt.subplots_adjust(hspace=0.4)
t = np.arange(0.01, 20.0, 0.01)

# log y axis
plt.subplot(221)
plt.semilogy(t, np.exp(-t/5.0))
plt.title('semilogy')
plt.grid(True)

# log x axis
plt.subplot(222)
plt.semilogx(t, np.sin(2*np.pi*t))
plt.ylim(-1.1,1.1)
plt.title('semilogx')
plt.grid(True)

# log x and y axis
plt.subplot(223)
plt.loglog(t, 20*np.exp(-t/10.0), basex=2)
plt.grid(True)
plt.title('loglog base 4 on x')

# with errorbars: clip non-positive values
plt.subplot(224)
# set logarithmic axes
plt.xscale("log", nonposx='clip')
plt.yscale("log", nonposy='clip')

x = 10.0**np.linspace(0.0, 2.0, 20)
y = x**2.0
plt.errorbar(x, y, xerr=0.1*x, yerr=5.0+0.75*y)
plt.ylim(ymin=0.1)
plt.title('Errorbars go negative')

# will need it later
fig_log_plots=plt.gcf()


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Matplotlib API
Can you write Python code for it?

%matplotlib inline
import matplotlib.pylab as plt
import numpy as np

def f(t):
    return np.exp(-t) * np.cos(2*np.pi*t)

t1 = np.arange(0.0, 5.0, 0.1)
t2 = np.arange(0.0, 5.0, 0.02)

# create figure
fig=plt.figure(1)
# add subplot to the figure, retunrns Axes object
ax1=fig.add_subplot(2,1,1)
# plot both data with a single plot function call
# first plot filled circles, then solid black line
lines11,lines12,=ax1.plot(t1, f(t1), 'bo', t2, f(t2), 'k')

# can use a simpler form, without commas - 212: 2 rows, 1 column, second plot
ax2=fig.add_subplot(212)
lines22,=ax2.plot(t2, np.cos(2*np.pi*t2), 'r--')
plt.show()

lines22.set_linewidth(1)
lines22.set_linestyle(':')
ax2.set_ylim(-1.5,1.5)
ax1.grid(True)
fig

fig_log_plots.savefig('log_plots.png')

fig.savefig('two_plots.png',dpi=600)


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Multiple Figures on the Same Plot
subplot(nrows, ncols, plot_number) creates a subplot axes positioned by the given grid definition  (help)
Can you write Python code for it?

def f(t):
    return np.exp(-t) * np.cos(2*np.pi*t)

t1 = np.arange(0.0, 5.0, 0.1)
t2 = np.arange(0.0, 5.0, 0.02)

# create figure
plt.figure(1)
# first subplot - 2,1,1: 2 rows, 1 column, first plot
plt.subplot(2,1,1)
# plot both data with a single plot function call
# first plot filled circles, then solid black line
plt.plot(t1, f(t1), 'bo', t2, f(t2), 'k')

# can use a simpler form, without commas - 212: 2 rows, 1 column, second plot
plt.subplot(212)
plt.plot(t2, np.cos(2*np.pi*t2), 'r--')


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Pie Chart
Can you write Python code for it?

# make a square figure and axes
plt.figure(1, figsize=(6,6))
plt.axes([0.1, 0.1, 0.8, 0.8])
labels = 'Frogs', 'Hogs', 'Dogs', 'Logs'
fracs = [15,30,45, 10]
explode=(0, 0.1, 0, 0)

plt.pie(fracs, explode=explode, labels=labels, autopct='%1.1f%%', shadow=True)
plt.title('Raining Hogs and Dogs', bbox={'facecolor':'0.9', 'pad':5})


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Plot with Fill
Can you write Python code for it?

t = np.arange(0.0, 1.01, 0.01)
s = np.sin(2*2*np.pi*t)

plt.fill(t, s*np.exp(-5*t), 'r')
plt.grid(True)


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Save Figure to File
savefig(filename.ext) saves current figure into a file (help)
Can you write Python code for it?

# add labels to the curves, can use LaTeX
plt.plot(x, y1, label=r'$\sin(x)$')
plt.plot(x, y2, label=r'$\sin(x^2)$')
plt.title('functions: $\sin(x)$ and $\sin(x^2)$')

plt.xlabel('x')
plt.ylabel('y')
plt.grid()

# change axes limits
plt.xlim(-0.1,6.5)
plt.ylim(-1.1,1.1)

# add legend; loc=0 "best" placement
plt.legend(loc=0)

# save figure to a PDF file
plt.savefig('sin_legend_plot.pdf')


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Two plots on the same axes with labels
Can you write Python code for it?

plt.plot(t, s1)
plt.plot(t, s2)

# add labels
plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('About as simple as it gets, folks')
# add grid
plt.grid(True)


I am working on the file "Day_01/02_Numpy_Matplotlib/01_Matplotlib.ipynb".
The first task is:
Using Matplotlib Gallery
If you do not know how to make a specific plot, the best place to go is matplotlib gallery.
It contains a large number of plots together with the Python souces files which you can download and adjust to your needs.
Can you write Python code for it?

# I just picked one to try:
%load http://matplotlib.org/mpl_examples/api/sankey_demo_rankine.py