I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
Writing Out Data to CSV
We can use the to_csv command to do export a DataFrame in CSV format. Note that the code
below will by default save the data into the current working directory. We can
save it to a different folder by adding the foldername and a slash to the file
vertical_stack.to_csv('foldername/out.csv'). We use the 'index=False' so that
pandas doesn't include the index number for each line.
Can you write Python code for it?

df_1 = pd.DataFrame({"A":[1,2,3,4], "B":[3,4,5,6]})
df_2 = pd.DataFrame({"C":[1,2,3,4], "B":[3,4,5,6]})
df_1

df_2

pd.concat([df_1, df_2], axis=0)

# Write DataFrame to CSV
vertical_stack.to_csv('data/vertical_stack.csv', index=False)


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
which gives the output
alpha-3 region  sub-region
0   AFG Asia    Southern Asia
1   ALA Europe  Northern Europe
2   ALB Europe  Southern Europe
Notice that we asked for a slice from 0:3. This yielded 3 rows of data. When you
ask for 0:3, you are actually telling Python to start at index 0 and select rows
0, 1, 2 up to but not including 3.
Let's explore some other ways to index and select subsets of data:
Can you write Python code for it?

# Select all columns for rows of index values 0 and 10
df_regions.loc[[0, 10], :]

# What does this do?
df_regions.loc[0, ['name', 'alpha-3']]

# What happens when you type the code below?
df_regions.loc[[0, 10, 149], :]


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
.plot()
When used without any parameters, df.plot() plots the index against every column, adding every numerical column:
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
A third, optional argument in plot() is a string of characters that indicates
the line type and color for the plot. The default value is a continuous blue
line. For example, we can make the line red ('r'), with circles at every data
point ('o'), and a dot-dash pattern ('-.'). Look through the matplotlib
gallery for more examples.
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
As you can see, this is not a particularly useful plot in our case! It has plotted irrelevant data (LGHC Indicator ID), plotted all columns on the same scale, and the x-axis places all values sequentially as they appear in the table (including the year!).
In many cases (including ours), the index will not provide any meaningful relation to the data contained in that row. A meaningful index is most likely to exist in the following two cases:

The dataframe was produced using .value_counts() or .groupby()
You’ve used .set_index(), probably with dates

So except in the above cases, we will usually want to specify one column as the independent variable for the x-axis, and a second variable as the dependent variable for the y-axis. To do this, we specify these variables using the x and y parameters.
Can you write Python code for it?

df_asthma.plot(x='Year', y='Rate')
plt.axis([2010,2017,0,1000])
plt.show()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
By default, matplotlib will create the figure in a separate window. When using
ipython notebooks, we can make figures appear in-line within the notebook by
writing:
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Challenge question
Try creating a 2x2 grid, and placing the graphs in the top left and bottom right positions
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Getting better. To really get a feel for the underlying data, Seaborn can generate violin plots:
Can you write Python code for it?

fig, ax = fig, ax = plt.subplots(figsize=(13,4))
ax = sns.violinplot(data=df_subset2,
                     x='Year', y='Rate', hue='Age Group',
                     ax=ax,
                    )


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Here is a useful post about reshaping data in pandas (pivot tables, stacking and unstacking columns): http://nikgrozev.com/2015/07/01/reshaping-in-pandas-pivot-pivot-table-stack-and-unstack-explained-with-pictures/
Make a line plot of your data
Matplotlib is a Python library that can be used to visualize data. The
toolbox matplotlib.pyplot is a collection of functions that make matplotlib
work like MATLAB. In most cases, this is all that you will need to use, but
there are many other useful tools in matplotlib that you should explore.
We will cover a few basic commands for formatting plots in this lesson. A great
resource for help styling your figures is the matplotlib gallery
(http://matplotlib.org/gallery.html), which includes plots in many different
styles and the source code that creates them. The simplest of plots is the 2
dimensional line plot. These examples walk through the basic commands for making
line plots using pyplots.
Using pyplot:
First, import the pyplot toolbox:
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
It is also easy to produce nice looking histograms:
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
It is also possible to further break down a seaborn boxplot using the 'hue' parameter (to demonstrate this, we'll create another subset that is the total population strata, which includes three different age group breakdowns):
Can you write Python code for it?

df_subset2 = df_asthma[df_asthma['Strata']=='Total Population']
ax = sns.boxplot(data=df_subset2,
                 x='Year', y='Rate', hue='Age Group',
                )


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Make other types of plots:
Matplotlib can make many other types of plots in much the same way that it makes
2 dimensional line plots. Look through the examples in
http://matplotlib.org/users/screenshots.html and try a few of them (click on the
"Source code" link and copy and paste into a new cell in ipython notebook or
save as a text file with a .py extension and run in the command line).

Challenge - Final Plot
Display your data using one or more plot types from the example gallery. Which
ones to choose will depend on the content of your own data file. If you are
using the streamgage file, you could make a histogram of the number of days with
a given mean discharge, use bar plots to display daily discharge statistics, or
explore the different ways matplotlib can handle dates and times for figures.

Exporting/saving a figure
If you have a figure that you would like to save externally from the notebook, this can be achieved using the plt.savefig() command. At its most basic, call this function after producing your figure, but before calling plt.show(). In the parentheses, enter the full filename and extension you would like to save the image to. Matplotlib will look at the extension, and then save the image as the appropriate filetype. It can handle .png, .jpg, and .pdf, among others. Note that you will not get a warning if the file already exists - it will just overwrite the existing file with the new one, so make sure you check first! (or write some code to check...)
Can you write Python code for it?

import os

myfilename = "myfile.png"  # change this to whatever you want the file to be called. Include any directories in the name too (eg. "results/myimage.jpg")

if os.path.exists(myfilename):
    print("File already exists!")
else:
    plt.savefig(myfilename)


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Now for a particularly complex version, we will add the data for 2013 and 2014:
Can you write Python code for it?

# create a new pivot table showing the rates per county according to age group:
df_pivot3 = df_subset2[df_subset2['Year']==2013].pivot(index='Geography', values='Rate', columns='Strata Name')
df_pivot4 = df_subset2[df_subset2['Year']==2014].pivot(index='Geography', values='Rate', columns='Strata Name')

# add a column with the year:
df_pivot2['Year']=2012
df_pivot3['Year']=2013
df_pivot4['Year']=2014

# stack the tables vertically
df_concat = pd.concat([df_pivot2, df_pivot3, df_pivot4], axis=0)

# view first 5 rows:
df_concat.head()

sns.lmplot(data=df_concat,
           x='18 and Over', y='Under 18',
           hue='Year',                 # we want to display each year separately
           markers=['x', '+', '*'],    # change how each year's marker is displayed
           ci=68,                      # change the confidence interval for the shading
           fit_reg=True,               # change to False if you want to remove the regression
           col='Year',                 # split the graph into subgraphs (can also use 'row')
          )
plt.show()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Plotting in Pandas
The official Pandas plotting documentation helps show the extent of the types of plots and basic presentation options available.
The exercises here are based on the notes by Jonathan Soma, for the Algorithms course for the Lede Program at Columbia University Graduate School of Journalism.
First, let's refresh ourselves on what our dataframe looks like:
Can you write Python code for it?


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Seaborn
Seaborn improves on the default styling and the ease of use compared with Matplotlib. The main thing to note here, is that seaborn is not natively integrated with Pandas, so you can't get its functionality just using the .plot() function. Instead, you will need to import Seaborn separately, then feed it your Pandas dataframe. As you will soon see, this small extra step is worth the trouble.
The Seaborn website has a lovely gallery of plots, to show you the scope of what it can do. The documentation for each plot type can be found here.
Can you write Python code for it?

import seaborn as sns

ax = sns.boxplot(data=df_pivot)


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Second, we will create a pivot table, to arrange the ED visit rate by year and geographical location:
Can you write Python code for it?

df_pivot = df_subset.pivot( values='Rate',
                            columns='Year',
                            index='Geography'
                          )
df_pivot.head(10)


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
The command plt.axis() sets the limits of the axes from a list of [xmin,
xmax, ymin, ymax] values (the square brackets are needed because the argument
for the function axis() is one list of values, not four separate numbers!).
The functions xlabel() and ylabel() will label the axes, and title() will
write a title above the figure.
A single figure can include multiple lines, and they can be plotted using the
same plt.plot() command by adding more pairs of x values and y values (and
optionally line styles):
Can you write Python code for it?

import numpy as np

# Create a numpy array between 0 and 10, with values evenly spaced every 0.5
t = np.arange(0., 10., 0.5)

# Red dashes with no symbols, blue squares with a solid line, and green triangles with a dotted line
plt.plot(t, t, 'r--', t, t**2, 'bs-', t, t**3, 'g^:')

plt.xlabel('This is the x axis')
plt.ylabel('This is the y axis')
plt.title('This is the figure title')

plt.show()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
This graph is more more ugly than the default, but you can see how the values can be set for each of the different elements using a dictionary of values. If you want to set the same value for all elements, you can just specify that value once: color="green".
Colors can be specified by single letters:
* k - black
* w - white
* c - cyan
* m - magenta
* b - blue
* r - red
* y - yellow
or by html color codes ('red', 'orange', 'steelblue', etc)
or by hex color codes (#BDA493, #63C638, etc -  A very handy color picker can be found here)

Challenge
Have a go at making a chart that is actually pleasant to look at!

Multiple plots on one figure?
We can also use Matplotlib's ability to plot multiple graphs onto a single figure. Using the same construction as in the first section, we can assign subplots (using the add_subplot() command) to an axis.
When we pass ax=ax to our plot, we’re saying “hey, we already have a graph made up! Please just use it instead” and then pandas/matplotlib does, instead of using a brand-new image for each.
So what’s the difference between a figure and an axis/subplot? Figures are made up of subplots. Figures are the table that the dinner plates of subplots go on. So far we’ve only seen them made up of just one, but we can do more!
Let’s create a 2x1 grid and put something in the first subplot and something in the second subplot. In the example below, you will see we also use the shortcut 2,1,1 and 2,1,2 in the add_subplot function. Remember this is the same as saying nrows=2, ncols=1, index=1. You can be even more terse and enter it as a three-digit number: 211
Can you write Python code for it?

fig = plt.figure()
ax1 = fig.add_subplot(2,1,1)
df_pivot.plot(kind='box', ax=ax1)

ax2 = fig.add_subplot(2,1,2)
df_pivot.plot(ax=ax2, legend=None)

plt.show()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
This is a rather complicated table, as it stacks a number of different demographic categories in the same column (Strata). We can appreciate this better by looking at all the unique values in some of our columns.
Can you write Python code for it?

df_asthma['Age Group'].unique()

df_asthma['Strata'].unique()

df_asthma['Strata Name'].unique()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
This is better, but with a high density of points, a box plot would be a more helpful choice:
Can you write Python code for it?

df_asthma.plot(x='Year', y='Rate', kind='box')
plt.show()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
Thus, we will probably want to work with subsets of our dataframe for analysis and visualization. For this example, we will first create a subset that only includes the All Ages Strata group:
Can you write Python code for it?

df_subset = df_asthma[df_asthma['Strata Name']=='All Ages']
df_subset.head()

df_asthma.size

df_subset.size


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
We can include a legend by adding the optional keyword argument label='' in
plot(). Caution: We cannot add labels to multiple lines that are plotted
simultaneously by the plt.plot() command like we did above because Python
won't know to which line to assign the value of the argument label. Multiple
lines can also be plotted in the same figure by calling the plot() function
several times:
Can you write Python code for it?

# Red dashes with no symbols, blue squares with a solid line, and green triangles with a dotted line
plt.plot(t, t, 'r--', label='linear')
plt.plot(t, t**2, 'bs-', label='square')
plt.plot(t, t**3, 'g^:', label='cubic')

plt.legend(loc='upper left', shadow=True, fontsize='x-large')

plt.xlabel('This is the x axis')
plt.ylabel('This is the y axis')
plt.title('This is the figure title')

plt.show()


I am working on the file "Pandas/Pandas plotting.ipynb".
The first task is:
When pandas plots, it assumes every single data point should be connected. Naturally, a line graph is a poor choice to represent this kind of data. We can specify the type of chart to generate using the kind parameter.
Can you write Python code for it?

df_asthma.plot(x='Year', y='Rate', kind='scatter')
plt.axis([2010,2017,0,1000])
plt.show()


I am working on the file "Pandas/Pandas-motm.ipynb".
The first task is:
DataFrame
Represents a tabular, spreadsheet like data sturcture.
It has both row and column index.
You can think about it as a dict of Series but sharing the same index.
If you work with pandas, you are mostly working in dataframes.
Can you write Python code for it?

df_1 = pd.DataFrame(np.random.randint(0,100,size=(100,4)), columns=list('ABCD'))

print(type(df_1))
df_1.head()

df_1.tail(3)

df_1.index

df_1.columns

df_1.values

df_1.describe()

df_1.mean()

df_1

df_1['A']

df_1[0:3]


I am working on the file "Pandas/Pandas-motm.ipynb".
The first task is:
Reading data from File
Pandas can read and write to a multitude of file formats
Can you write Python code for it?

import time
t_start = time.time()
apple_df = pd.read_csv('https://raw.githubusercontent.com/matplotlib/sample_data/master/aapl.csv')
t_end = time.time()
print (t_end - t_start)
apple_df.head()

apple_describe_time_start = time.time()
apple_df.describe()
apple_describe_time_end = time.time()

print (apple_describe_time_end - apple_describe_time_start)

# pd.set_eng_float_format(accuracy=2, use_eng_prefix=True)
apple_df.mean()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
A single column from a dataframe is called a Series
Can you write Python code for it?

type(emissions)

type(emissions['Operating Time'])

emissions['Operating Time']


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Access parts of the dataframe
Can you write Python code for it?


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Boolean filtering
Can you write Python code for it?

PA_cap = capacity.loc[capacity['Plant State']=='PA',:]
PA_cap

PA_NGCC_cap = capacity.loc[(capacity['Plant State']=='PA') &
             (capacity['Technology']=='Natural Gas Fired Combined Cycle'),:]
PA_NGCC_cap


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
If I want to keep the NERC Region, I can do that in the groupby
Can you write Python code for it?

facility_gen = gen_stack.loc[gen_stack['Month']=='May',:].groupby(['Plant Id', 'NERC Region']).sum()
facility_gen.head()

facility_gen.reset_index('NERC Region', inplace=True)
facility_gen.head()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Index into a dataframe using .loc or .iloc with square brackets and row,column notation
Can you write Python code for it?

emissions.loc[0:5,'Operating Time']

emissions.iloc[0:5,:3]


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Load capacity data
Can you write Python code for it?

capacity = pd.read_excel(fn2, sheetname='Operating', header=1)

capacity.head()

capacity.tail()

capacity.drop(20187, inplace=True)

capacity.loc[:,'Plant ID'] = capacity.loc[:,'Plant ID'].astype(int)

capacity.head()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Load emissions data
Can you write Python code for it?

emissions = pd.read_csv(fn1)

emissions.head()

emissions = pd.read_csv(fn1, index_col=False)

emissions.head()

emissions.tail()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Merge data from all three sources
Can you write Python code for it?

merged = facility_gen.merge(facility_cap, how='inner', left_index=True, right_index=True)

merged.describe()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Not all columns sum well
Can you write Python code for it?

facility_emiss = emissions.groupby('Facility ID (ORISPL)').sum()
# facility_emiss = facility_emiss.iloc[:,2:]
facility_emiss


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Not sure why this is returning a float. Go back to the function and return an int instead.
Can you write Python code for it?

facility_emiss.apply(correct_region, axis=1)

facility_emiss.loc[:,'EPA Region'] = facility_emiss.apply(correct_region, axis=1)
facility_emiss = facility_emiss.iloc[:,2:]
facility_emiss


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Notice that most of the columns have a leading space?
Can you write Python code for it?

columns_strip = [name.strip() for name in emissions.columns]
columns_strip

emissions.columns = columns_strip
emissions.columns

emissions.dtypes


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Now concat the two dataframes
Can you write Python code for it?

final = pd.concat([merged, non_combust])
final

final.index.rename('Plant ID', inplace=True)

final.describe()

final['CO2 (short tons)'].sum() * 2000 * 2.2046 / final['Netgen'].sum()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Repeat groupby and sum to get capacity of facilities
Can you write Python code for it?

cols = ['Plant ID', 'Nameplate Capacity (MW)']
facility_cap = capacity.loc[:,cols].groupby('Plant ID').sum()
facility_cap


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Stack data by month rather than having multiple columns
Not sure if I'll have time for this section
I'm lazy and want to get a list of month names without typing them all
Can you write Python code for it?

# could have done this as a list comprehension, but it would have been harder to read
months = []
for name in generation.columns:
    if 'Netgen' in name:
        month = name.split()[-1]
        months.append(month)
months

id_cols = ['Plant Id', 'Plant State', 'NERC Region', 'AER Fuel Type Code']
monthly_cols = []
def find_col_names(cols):
    for col in cols:
        if 'January' in col:
            monthly_cols.append(col.split()[0])

find_col_names(generation.columns)
id_cols + monthly_cols

pd.DataFrame(columns=id_cols + monthly_cols + ['Month'])

gen_list = []
for month in months:
    gen_df = pd.DataFrame(columns=id_cols + monthly_cols)

    # Took me a few tries to figure out that I couldn't use .loc for gen_df
    gen_df[id_cols] = generation.loc[:,id_cols]
    gen_df['Month'] = month

    for col in monthly_cols:
        gen_df.loc[:,col] = generation.loc[:,col + ' ' + month]

    gen_list.append(gen_df)

gen_stack = pd.concat(gen_list)
gen_stack.describe()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Sum unit emissions for each facility using groupby
Can you write Python code for it?


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Tag lines as using a combustion fuel or not
Can you write Python code for it?

gen_stack['AER Fuel Type Code'].unique()

non_combust = ['HYC', 'NUC', 'SUN', 'GEO', 'WND'] # might be incomplete

def tag_combust(row):
    if row['AER Fuel Type Code'] in non_combust:
        return 0
    else:
        return 1

gen_stack['Combust'] = gen_stack.apply(tag_combust, axis=1)

gen_stack.head()


I am working on the file "Pandas/Pre-written Pandas example.ipynb".
The first task is:
Use apply to apply a function to every row of the dataframe
If we want to keep the EPA Region, there are probably better ways to do it than this. We will write a little function that divides the month by 5 (May) and then divides the region by that result.
Can you write Python code for it?

def correct_region(row):
    num_units = row['Month'] / 5
    region = row['EPA Region'] / num_units
    return region


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Average BasePay
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Finding some basic info about the data through the info() function
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Head of dataframe
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Highest amount of overtime pay in the dataset
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
How many people have the word Chief in their job title?
Can you write Python code for it?

def find_chief(job_title):
    if 'chief' in job_title.lower().split():
        return True
    else:
        return False

sal = pd.read_csv('Salaries.csv')

sum(sal['JobTitle'].apply(lambda x: find_chief(x)))


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
How much does JOSEPH DRISCOLL make (including benefits)?
Can you write Python code for it?

sal[sal['EmployeeName'] == 'JOSEPH DRISCOLL']['TotalPayBenefits']


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Job titles represented by only one person
Can you write Python code for it?

copied_sal = sal[sal['Year'] == 2013]
group = copied_sal.groupby('JobTitle').count()
count = group[group['Id'] == 1]
count.count()['Id']


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Number of unique job titles
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
Read CSV into dataframe 'sal'
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
The average (mean) BasePay of all employees per year? (2011-2014)?
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
The name of highest paid person (including benefits)?
Can you write Python code for it?


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
The name of lowest paid person
Can you write Python code for it?

sal[sal['TotalPayBenefits']  == sal['TotalPayBenefits'].min()]


I am working on the file "Pandas/SF Salaries Exercise.ipynb".
The first task is:
What is the job title of JOSEPH DRISCOLL?
Can you write Python code for it?

sal[sal['EmployeeName'] == 'JOSEPH DRISCOLL']['JobTitle']


I am working on the file "PandasPractice.ipynb".
The first task is:
Adding/Merging two series with common indices
Can you write Python code for it?

ser1 = pd.Series([1,2,3,4],['CA', 'OR', 'CO', 'AZ'])
ser2 = pd.Series([1,2,5,4],['CA', 'OR', 'NV', 'AZ'])
ser3 = ser1+ser2

print ("\nAfter adding the two series, the result looks like this...\n",'-'*59, sep='')
print(ser3)
print("\nPython tries to add values where it finds common index name, and puts NaN where indices are missing\n")

print ("\nThe idea works even for multiplication...\n",'-'*43, sep='')
print (ser1*ser2)

print ("\nOr even for combination of mathematical operations!\n",'-'*53, sep='')
print (np.exp(ser1)+np.log10(ser2))


I am working on the file "PandasPractice.ipynb".
The first task is:
Area plot
Can you write Python code for it?

import seaborn as sns #Plot style will change to Seaborn package style from now on
df2.plot.area(alpha=0.4)


I am working on the file "PandasPractice.ipynb".
The first task is:
Bar plot (with and without stacking)
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Boxplot
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Combine 'apply' with lambda expession for in-line calculations
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Conditional selection, index (re)setting, multi-index
Basic idea of conditional check and Boolean DataFrame
Can you write Python code for it?

print("\nThe DataFrame\n",'-'*45, sep='')
print(df)
print("\nBoolean DataFrame(s) where we are checking if the values are greater than 0\n",'-'*75, sep='')
print(df>0)
print("\n")
print(df.loc[['A','B','C']]>0)
booldf = df>0
print("\nDataFrame indexed by boolean dataframe\n",'-'*45, sep='')
print(df[booldf])


I am working on the file "PandasPractice.ipynb".
The first task is:
Creating a Series (Pandas class)

From numerical data only
From numerical data and corresponding index (row labels)
From NumPy array as the source of numerical data
Just using a pre-defined dictionary
Can you write Python code for it?

pd.Series(data=my_data) # Output looks very similar to a NumPy array

pd.Series(data=my_data, index=labels) # Note the extra information about index

# Inputs are in order of the expected parameters (not explicitly named), NumPy array is used for data
pd.Series(arr, labels)

pd.Series(d) # Using a pre-defined Dictionary object


I am working on the file "PandasPractice.ipynb".
The first task is:
Creating and accessing DataFrame

Indexing
Adding and deleting rows and columns
Subsetting DataFrame
Can you write Python code for it?

np.random.seed(101)
matrix_data = rn(5,4)
row_labels = ['A','B','C','D','E']
column_headings = ['W','X','Y','Z']

df = pd.DataFrame(data=matrix_data, index=row_labels, columns=column_headings)
print("\nThe data frame looks like\n",'-'*45, sep='')
print(df)


I am working on the file "PandasPractice.ipynb".
The first task is:
Deletion by 'del' command # This affects the dataframe immediately, unlike drop method.
Can you write Python code for it?

print("\nDeleting last column by 'del' command\n",'-'*50, sep='')
del df['col3length']
print(df)
df['col3length']= df['col3'].apply(len)


I am working on the file "PandasPractice.ipynb".
The first task is:
Deletion, sorting, list of column and row names
Getting the names of the columns
Can you write Python code for it?

print("\nName of columns\n",'-'*20, sep='')
print(df.columns)
l = list(df.columns)
print("\nColumn names in a list of strings for later manipulation:",l)


I am working on the file "PandasPractice.ipynb".
The first task is:
Find Null Values or Check for Null Values
Can you write Python code for it?

df = pd.DataFrame({'col1':[1,2,3,np.nan],
                   'col2':[np.nan,555,666,444],
                   'col3':['abc','def','ghi','xyz']})
df.head()

df.isnull()

df.fillna('FILL')


I am working on the file "PandasPractice.ipynb".
The first task is:
GroupBy method
Can you write Python code for it?

# Create dataframe
data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],
       'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],
       'Sales':[200,120,340,124,243,350]}
df = pd.DataFrame(data)
df

byComp = df.groupby('Company')
print("\nGrouping by 'Company' column and listing mean sales\n",'-'*55, sep='')
print(byComp.mean())
print("\nGrouping by 'Company' column and listing sum of sales\n",'-'*55, sep='')
print(byComp.sum())
# Note dataframe conversion of the series and transpose
print("\nAll in one line of command (Stats for 'FB')\n",'-'*65, sep='')
print(pd.DataFrame(df.groupby('Company').describe().loc['FB']).transpose())
print("\nSame type of extraction with little different command\n",'-'*68, sep='')
print(df.groupby('Company').describe().loc[['GOOG', 'MSFT']])


I am working on the file "PandasPractice.ipynb".
The first task is:
Hexagonal bin plot for bivariate data
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Histogram of a single column
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Indexing and slicing (columns)

By bracket method
By DOT method (NOT recommended)
Can you write Python code for it?

print("\nThe 'X' column\n",'-'*25, sep='')
print(df['X'])
print("\nType of the column: ", type(df['X']), sep='')
print("\nThe 'X' and 'Z' columns indexed by passing a list\n",'-'*55, sep='')
print(df[['X','Z']])
print("\nType of the pair of columns: ", type(df[['X','Z']]), sep='')
print ("\nSo, for more than one column, the object turns into a DataFrame")

print("\nThe 'X' column accessed by DOT method (NOT recommended)\n",'-'*55, sep='')
print(df.X)


I am working on the file "PandasPractice.ipynb".
The first task is:
Joining
Joining is a convenient method for combining the columns of two potentially differently-indexed DataFrames into a single DataFrame based on 'index keys'.
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Kernel density estimation
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Lineplot
Can you write Python code for it?

df1.plot.line(x=df1.index,y=['B','C'],figsize=(12,4),lw=1) # Note matplotlib arguments like 'lw' and 'figsize'


I am working on the file "PandasPractice.ipynb".
The first task is:
Merging by a common 'key'
The merge function allows you to merge DataFrames together using a similar logic as merging SQL Tables together.
Can you write Python code for it?

left = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                     'A': ['A0', 'A1', 'A2', 'A3'],
                     'B': ['B0', 'B1', 'B2', 'B3']})

right = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                          'C': ['C0', 'C1', 'C2', 'C3'],
                          'D': ['D0', 'D1', 'D2', 'D3']})

print("\nThe DataFrame 'left'\n",'-'*30, sep='')
print(left)
print("\nThe DataFrame 'right'\n",'-'*30, sep='')
print(right)

merge1= pd.merge(left,right,how='inner',on='key')
print("\nAfter simple merging with 'inner' method\n",'-'*50, sep='')
print(merge1)


I am working on the file "PandasPractice.ipynb".
The first task is:
Merging, Joining, Concatenating
Concatenation
Can you write Python code for it?


I am working on the file "PandasPractice.ipynb".
The first task is:
Missing Values
Can you write Python code for it?

df = pd.DataFrame({'A':[1,2,np.nan],'B':[5,np.nan,np.nan],'C':[1,2,3]})
df['States']="CA NV AZ".split()
df.set_index('States',inplace=True)
print(df)


I am working on the file "PandasPractice.ipynb".
The first task is:
Multi-indexing
Can you write Python code for it?

# Index Levels
outside = ['G1','G1','G1','G2','G2','G2']
inside = [1,2,3,1,2,3]
hier_index = list(zip(outside,inside))

print("\nTuple pairs after the zip and list command\n",'-'*45, sep='')
print(hier_index)
hier_index = pd.MultiIndex.from_tuples(hier_index)
print("\nIndex hierarchy\n",'-'*25, sep='')
print(hier_index)
print("\nIndex hierarchy type\n",'-'*25, sep='')
print(type(hier_index))

print("\nCreating DataFrame with multi-index\n",'-'*37, sep='')
np.random.seed(101)
df1 = pd.DataFrame(data=np.round(rn(6,3),2), index= hier_index, columns= ['A','B','C'])
print(df1)

print("\nSubsetting multi-index DataFrame using two 'loc' methods\n",'-'*60, sep='')
print(df1.loc['G2'].loc[[1,3]][['B','C']])

print("\nNaming the indices by 'index.names' method\n",'-'*45, sep='')
df1.index.names=['Outer', 'Inner']
print(df1)


I am working on the file "PandasPractice.ipynb".
The first task is:
Pandas 'dropna' method
Can you write Python code for it?

print("\nDropping any rows with a NaN value\n",'-'*35, sep='')
print(df.dropna(axis=0))
print("\nDropping any column with a NaN value\n",'-'*35, sep='')
print(df.dropna(axis=1))
print("\nDropping a row with a minimum 2 NaN value using 'thresh' parameter\n",'-'*68, sep='')
print(df.dropna(axis=0, thresh=2))


I am working on the file "PandasPractice.ipynb".
The first task is:
Pandas 'fillna' method
Can you write Python code for it?

print("\nFilling values with a default value\n",'-'*35, sep='')
print(df.fillna(value='FILL VALUE'))
print("\nFilling values with a computed value (mean of column A here)\n",'-'*60, sep='')
print(df.fillna(value=df['A'].mean()))


I am working on the file "PandasPractice.ipynb".
The first task is:
Pivot Table
Can you write Python code for it?

data = {'A':['foo','foo','foo','bar','bar','bar'],
     'B':['one','one','two','two','one','one'],
       'C':['x','y','x','y','x','y'],
       'D':[1,3,2,5,4,1]}

df = pd.DataFrame(data)
df

# Index out of 'A' and 'B', columns from 'C', actual numerical values from 'D'
df.pivot_table(values='D',index=['A', 'B'],columns=['C'])

# Index out of 'A' and 'B', columns from 'C', actual numerical values from 'D'
df.pivot_table(values='D',index=['A', 'B'],columns=['C'], fill_value='FILLED')


I am working on the file "PandasPractice.ipynb".
The first task is:
Read in the CSV data file
Can you write Python code for it?

df1=pd.read_csv('df1.csv', index_col=0)
df1.head()

df2=pd.read_csv('df2')
df2.head()


I am working on the file "PandasPractice.ipynb".
The first task is:
Standard statistical functions directly apply to columns
Can you write Python code for it?

print("\nSum of the column 'FuncApplied' is: ",df['FuncApplied'].sum())
print("Mean of the column 'FuncApplied' is: ",df['FuncApplied'].mean())
print("Std dev of the column 'FuncApplied' is: ",df['FuncApplied'].std())
print("Min and max of the column 'FuncApplied' are: ",df['FuncApplied'].min(),"and",df['FuncApplied'].max())


I am working on the file "PandasPractice.ipynb".
The first task is:
Subsetting DataFrame
Can you write Python code for it?

np.random.seed(101)
matrix_data = rn(5,4)
row_labels = ['A','B','C','D','E']
column_headings = ['W','X','Y','Z']
df = pd.DataFrame(data=matrix_data, index=row_labels, columns=column_headings)

print("\nThe DatFrame\n",'-'*45, sep='')
print(df)
print("\nElement at row 'B' and column 'Y' is\n")
print(df.loc['B','Y'])
print("\nSubset comprising of rows B and D, and columns W and Y, is\n")
df.loc[['B','D'],['W','Y']]


I am working on the file "PandasPractice.ipynb".
The first task is:
Useful operations
head() and unique values

head()
unique()
nunique()
value_count()
Can you write Python code for it?

import pandas as pd
df = pd.DataFrame({'col1':[1,2,3,4,5,6,7,8,9,10],
                   'col2':[444,555,666,444,333,222,666,777,666,555],
                   'col3':'aaa bb c dd eeee fff gg h iii j'.split()})
df

print("\nMethod head() is for showing first few entries\n",'-'*50, sep='')
df.head()

print("\nFinding unique values in 'col2'\n",'-'*40, sep='') # Note 'unique' method applies to pd.series only
print(df['col2'].unique())

print("\nFinding number of unique values in 'col2'\n",'-'*45, sep='')
print(df['col2'].nunique())

print("\nTable of unique values in 'col2'\n",'-'*40, sep='')
t1=df['col2'].value_counts()
print(t1)


I am working on the file "PandasPractice.ipynb".
The first task is:
What type of values can a Pandas Series hold?
Can you write Python code for it?

print ("\nHolding numerical data\n",'-'*25, sep='')
print(pd.Series(arr))
print ("\nHolding text labels\n",'-'*20, sep='')
print(pd.Series(labels))
print ("\nHolding functions\n",'-'*20, sep='')
print(pd.Series(data=[sum,print,len]))
print ("\nHolding objects from a dictionary\n",'-'*40, sep='')
print(pd.Series(data=[d.keys, d.items, d.values]))


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
A where operation for getting.
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
A where operation with setting.
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Applying

Aggregation: computing a summary statistic (or statistics) about each group. Some examples:

Compute group sums or means
Compute group sizes / counts

Transformation: perform some group-specific computations and return a like-indexed. Some examples:

Standardizing data (zscore) within group
Filling NAs within groups with a value derived from each group

Filtration: discard some groups, according to a group-wise computation that evaluates True or False. Some examples:

Discarding data that belongs to groups with only a few members
Filtering out data based on the group sum or mean

Aggregation
"reduces" the DataFrame, meaning the df_original.shape > df_aggregated.shape
Can you write Python code for it?

grouped.aggregate(["sum", "count", "median", "mean"]) # see `Operations` above for more built-in methods


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Axis

default: axis=0
axis=0 is "column-wise", for each column along the rows, horizontal
axis=1 is "row-wise", for each row along the columns, vertical
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
df


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Concise summary of a DataFrame
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Creating a DataFrame by passing a dict of objects that can be converted to series-like.
Can you write Python code for it?

df2 = pd.DataFrame({'A' : 1.,
                    'B' : pd.Timestamp('20130102'),
                    'C' : pd.Series(1, index=list(range(4)), dtype='float32'),
                    'D' : np.array([3] * 4, dtype='int32'),
                    'E' : pd.Categorical(["test","train","test","train"]),
                    'F' : 'foo',
                    'tinyRick': pd.Series([2000])})
df2


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Creating a DataFrame by passing a numpy array, with labeled columns:
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Display the index, columns, and the underlying numpy data
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Drop only rows where all values are missing.
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Duplicates
Can you write Python code for it?

dfx = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar', 'foo', 'foo'],
                   'B' : ['one', 'one', 'two', 'three',
                          'two', 'two', 'one', 'three'],
                   'C' : np.random.randn(8),
                   'D' : np.random.randn(8)})
dfx


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Fill missing values
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Filtration
returns a subset of the original object.
Can you write Python code for it?

df = pd.DataFrame({'A': np.arange(8), 'B': list('aabbbbcc')})
df

df.groupby('B').filter(lambda x: len(x) > 2)


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Frequent use case: combining a Series of Bools with specific column names to select data.
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Grouping
By “group by” we are referring to a process involving one or more of the following steps

Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure
Can you write Python code for it?

df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar', 'foo', 'foo'],
                   'B' : ['one', 'one', 'two', 'three',
                          'two', 'two', 'one', 'three'],
                   'C' : np.random.randn(8),
                   'D' : np.random.randn(8)})
df


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Join
SQL style merges.
Can you write Python code for it?

left = pd.DataFrame({'key': ['foo', 'bar'], 'lval': [1, 2]})
right = pd.DataFrame({'key': ['bar', 'foo'], 'rval': [4, 5]})
left

right

pd.merge(left, right, on='key')


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Merge
pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.
Concat
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(10, 4))
df

# break it into multiple pieces
pieces = [df[3:7], df[:3], df[7:]]
pd.concat(pieces)


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Miscellaneous
zip example
Can you write Python code for it?

xls = pd.ExcelFile('data/microbiome/MID1.xls')
df = xls.parse("Sheet 1", header=None)
df.columns = ["Taxon", "Count"]
df.head(3)

df["superkingdom"], df["rest"] = zip(*df["Taxon"].apply(lambda x: x.split(" ", 1)))
df.head(3)

df.superkingdom.unique()


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Missing Data
pandas primarily uses the value np.nan to represent missing data.
Can you write Python code for it?

df.loc[0, ] = np.nan
df.loc[2, ["A", "C"]] = np.nan
df


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Navigation

Esc and Enter to toggle between "Command" and "Edit" mode --> see colour indicator of frame surrounding the cell

when in "Command" mode:

use cursor keys to navigate
a to insert cell Above
b to insert cell Below
dd to delete cell

when in "Edit" mode:

Tab for tab completion of file-names
Shift + Tab in parenthesis of function to display docstring
Cmd + / toggle line comment

Running code, getting help
In the notebook, to run a cell of code, hit Shift + Enter. This executes the cell and puts the cursor in the next cell below, or makes a new one if you are at the end.  Alternately, you can use:

Alt + Enter to force the creation of a new cell unconditionally (useful when inserting new content in the middle of an existing notebook).
Control + Enter executes the cell and keeps the cursor in the same cell, useful for quick experimentation of snippets that you don't need to keep permanently.
Esc and Enter to toggle between "Command" and "Edit" mode --> see colour indicator of frame surrounding the cell
h for Help or checkout "Help" --> "Keyboard shortcuts" from the menu
Shift + Tab inside of the brackets of a function to get the Docstring
Cmd + / to toggle line comment

Shell commands
Can you write Python code for it?

!pwd

!ls -la

var = !ls
len(var), var[:3]

%%bash
echo "My shell is:" $SHELL
echo "My disk usage is:"
df -h


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Operations
Stats
Operations in general exclude missing data.
Performing a descriptive statistic
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Pivot
We can produce pivot tables from this data very easily:
Can you write Python code for it?

df = pd.DataFrame({'A' : ['one', 'one', 'two', 'three'] * 3,
                   'B' : ['a', 'b', 'c'] * 4,
                   'C' : ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,
                   'D' : np.random.randn(12),
                   'E' : np.random.randn(12)})
df

pd.pivot_table(df, values=['D', "E"], index=['A', 'B'], columns=['C'])


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Reshaping
Stack
“compresses” a level in the DataFrame’s columns.
Can you write Python code for it?

tuples = list(zip(*[['bar', 'bar', 'baz', 'baz',
                     'foo', 'foo', 'qux', 'qux'],
                    ['one', 'two', 'one', 'two',
                     'one', 'two', 'one', 'two']]))
index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])
df2 = df[:4]
df2

stacked = df2.stack()
stacked


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Row wise division
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Selecting via [], which slices the rows.
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Selection by Label
df.loc[rows, columns]
.loc is primarily label based, but may also be used with a boolean array. .loc will raise KeyError when the items are not found. Allowed inputs are:
- A single label, e.g. 5 or 'a', (note that 5 is interpreted as a label of the index. This use is not an integer position along the index)
- A list or array of labels ['a', 'b', 'c']
- A slice object with labels 'a':'f', (note that contrary to usual python slices, both the start and the stop are included!)
- A boolean array
- A callable function with one argument (the calling Series, DataFrame or Panel) and that returns valid output for indexing (one of the above)
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
df

# Let's transpose the DataFrame and change the order of the columns (to force us to the string labels of the novel row index).
# row-index: Strings, columns: Integers
dfx = df.T
import random
columns = dfx.columns.tolist()
random.shuffle(columns)
dfx.columns = columns
dfx

dfx.loc["A":"C", ]

dfx.loc[["C", "A", "D"], ]

dfx.loc[["A", "D"], 0:2]


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Selection by Position
df.iloc[rows, columns]
.iloc is primarily integer position based (from 0 to length-1 of the axis), but may also be used with a boolean array. .iloc will raise IndexError if a requested indexer is out-of-bounds, except slice indexers which allow out-of-bounds indexing. (this conforms with python/numpy slice semantics). Allowed inputs are:
- An integer e.g. 5
- A list or array of integers [4, 3, 0]
- A slice object with ints 1:7
- A boolean array
- A callable function with one argument (the calling Series, DataFrame or Panel) and that returns valid output for indexing (one of the above)
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
df

### transpose
# df = df.T
# df

df.iloc[0:3, :]


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Setting
Setting a new column automatically aligns the data by the indexes
Can you write Python code for it?

dates = pd.date_range('20130101', periods=6)
dfx = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
dfx

s1 = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130102', periods=6))
s1 = s1.sort_values(ascending=False)
s1

dfx["F"] = s1
dfx


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Setting values by label
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
df['E'] = ['one', 'one','two','three','four','three']
df

df.loc[5, "A"] = 0.815


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Setting values by position
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Sorting by values
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Splitting
Iterate over a groupby object, just to illustrate the split part of the groupby method.
Can you write Python code for it?

grouped = df.groupby(['A','B'])
for name, group in grouped:
    print "Name:", name
    print group
    print "#"*50


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Summing boolean arrays can come in handy (True=1, False=0)
Can you write Python code for it?

sum(cond), len(cond) # or cond.sum(), cond.shape[0]


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
The full range of basic statistics that are quickly calculable and built into the base Pandas package are:
|  Function | Description  |
|:---:|:---:|
|  count | Number of non-null observations  |
|  sum | Sum of values  |
|  mean | Mean of values  |
|  mad | Mean absolute deviation  |
|  median | Arithmetic median of values  |
|  min | Minimum  |
|  max | Maximum  |
|  mode | Mode  |
|  abs | Absolute Value  |
|  prod | Product of values  |
|  std | Bessel-corrected sample standard deviation  |
|  var | Unbiased variance  |
|  sem | Standard error of the mean  |
|  skew | Sample skewness (3rd moment)  |
|  kurt | Sample kurtosis (4th moment)  |
|  quantile | Sample quantile (value at %)  |
|  cumsum | Cumulative sum  |
|  cumprod | Cumulative product  |
|  cummax | Cumulative maximum  |
|  cummin | Cumulative minimum  |
The need for custom functions is minimal unless you have very specific requirements. c.f.
http://pandas.pydata.org/pandas-docs/stable/basics.html
e.g. compute pairwise covariance of columns, excluding NA/null values
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
This comparison yields a Pandas.Series of Booleans.
Can you write Python code for it?

cond = df["A"] > 0
print type(cond)
cond


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
To drop any rows that have missing data.
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Unstack
the inverse operation of stack() is unstack(), which by default unstacks the last level
Can you write Python code for it?

stacked.unstack() # in this particular case equivalent to stacked.unstack(2)

stacked.unstack(0)

stacked.unstack(1)


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
Viewing Data
See the top & bottom rows of the frame
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
a current list of I/O tools:
- read_csv
- read_excel
- read_hdf
- read_sql
- read_json
- read_msgpack (experimental)
- read_html
- read_gbq (experimental)
- read_stata
- read_sas
- read_clipboard
- read_pickle
checkout the documentation http://pandas.pydata.org/pandas-docs/stable/ for more infos
writing data to file
Can you write Python code for it?

fn_out = r"data/Tidy_data.txt"
df.to_csv(fn_out, sep='\t', header=True, index=False)


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
apply on a DataFrame perform a Series-wise operation
Can you write Python code for it?

df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])
df

def example_custom_function_on_series_1(series):
    return series / series.sum()

### notice how example_custom_function_on_series_1 return a single value per Series --> transformation
df.apply(example_custom_function_on_series_1)

def example_custom_function_on_series_2(series):
    return series.max() - series.min()

### notice how example_custom_function_on_series_2 return a single value per Series --> aggregation
df.apply(example_custom_function_on_series_2)


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
delete (drop) a column
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
delete (drop) some rows
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
from Excel
Can you write Python code for it?

xls = pd.ExcelFile('data/microbiome/MID1.xls')
print xls.sheet_names  # see all sheet names
df = xls.parse("Sheet 1", header=None)
df.columns = ["Taxon", "Count"]
df.head()


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
press Shift + Tab inside of the brackets to get the Docstring
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
rpy2 example
call an R function from Python
https://en.wikipedia.org/wiki/Quantile_normalization
https://bioconductor.org/packages/release/bioc/html/limma.html
Can you write Python code for it?

import rpy2.robjects as robjects
import rpy2.robjects.packages as rpackages
from rpy2.robjects.numpy2ri import numpy2ri
robjects.conversion.py2ri = numpy2ri

dict_ = {"Intensity First"  : [5.0, 2.0, 3.0, 4.0],
         "Intensity Second" : [4.0, 1.0, 4.0, 2.0],
         "Intensity Third"  : [3.0, 4.0, 6.0, 8.0]}
df = pd.DataFrame(dict_, index=list("ABCD"))
df

def R_function_normalizeQuantiles():
    rpackages.importr('limma')
    normalizeQuantiles = robjects.r['normalizeQuantiles']
    return normalizeQuantiles

def quantile_normalize(df, cols_2_norm):
    """
    :param df: DataFrame
    :param cols_2_norm: ListOfString (Columns to normalize)
    :return: DataFrame
    """
    normalizeQuantiles = R_function_normalizeQuantiles()

    # set Zero to NaN and transform to log-space
    df[cols_2_norm] = df[cols_2_norm].replace(to_replace=0.0, value=np.nan)
    df[cols_2_norm] = np.log10(df[cols_2_norm])

    # quantile normalize
    df[cols_2_norm] = np.array(normalizeQuantiles(df[cols_2_norm].values))

    # Transform back to non-log space and replace NaN with Zero
    df[cols_2_norm] = np.power(10, df[cols_2_norm])
    df[cols_2_norm] = df[cols_2_norm].replace(to_replace=np.nan, value=0.0)

    return df

df_norm = quantile_normalize(df, df.columns.tolist())
df_norm


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
see Installation_Customization_Resources.txt for useful infos and links
IPython: beyond plain Python
IPython provides a rich toolkit to help you make the most out of using Python, with:

*TAB COMPLETION*
Powerful Python shells (terminal and Qt-based).
A web-based notebook with the same core features but support for code, text, mathematical expressions, inline plots and other rich media.
Support for interactive data visualization and use of GUI toolkits.
Flexible, embeddable interpreters to load into your own projects.
Easy to use, high performance tools for parallel computing.

http://ipython.org/ipython-doc/stable/interactive/tutorial.html
http://ipython.readthedocs.io/en/stable/
Jupyter notebook
The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.
Checkout the "command palette" for useful commands and shortcuts.
Can you write Python code for it?

from IPython.core.display import HTML
HTML("<iframe src=https://jupyter.readthedocs.io/en/latest/_images/notebook_components.png width=800 height=400></iframe>")


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
see also duplicated method to return boolean Series denoting duplicate rows, optionally only considering certain columns
Can you write Python code for it?


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
string processing methods
for Series and Index, to make it easy to operate on each element of the array. These methods exclude missing/NA values automatically. These are accessed via the str attribute and generally have names matching the equivalent (scalar) built-in string methods.
Can you write Python code for it?

s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
s

s.str.lower()


I am working on the file "Pandas_Introduction.ipynb".
The first task is:
uncomment the transpose and execute the two cells above again and notice that this doesn't raise an Error compared to label based indexing.
Can you write Python code for it?


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Most of the column names are self-explanatory. One thing that initially confused me was that there is no column telling us the team of the player taking the shot. It turns out that that information is hidden in the MATCHUP column.
Can you write Python code for it?


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Part3: Questions comparing two teams in one game.
Q 3.1: How often did the winning team will have a lower make percentage than the losing team
Here it looks like we need to compare two rows in the shot_and_game_result_by_matchup dataframe. Don't do that! Remember that columns are for comparing not rows. Let me say that again
compare columns not rows!!!
ok so how do we turn two rows into two columns?
Can you write Python code for it?

shot_result_by_gameid_and_w = \
df.groupby(['GAME_ID','W','SHOT_RESULT']).size().unstack()

shot_result_by_matchup_gameid.head(4)

shot_result_by_gameid_and_w['make_percent'] = \
round((shot_result_by_gameid_and_w['made']/\
       shot_result_by_gameid_and_w.sum(axis=1))*100,1)

shot_result_by_gameid_and_w.head(4)

shot_result_by_gameid_and_w = \
shot_result_by_gameid_and_w.reset_index()

shot_result_by_gameid_and_w.head(4)

make_percent_by_gameid = \
shot_result_by_gameid_and_w.pivot(index='GAME_ID', columns='W')

make_percent_by_gameid.head()

make_percent_by_gameid[\
make_percent_by_gameid['make_percent']['W']<\
make_percent_by_gameid['make_percent']['L']].head()

len(make_percent_by_gameid[\
make_percent_by_gameid['make_percent']['W']<\
make_percent_by_gameid['make_percent']['L']])

len(make_percent_by_gameid)

(197./904)*100


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Personally I find the groupby operation to be more expressive.
Q1.1 which team made the most and least shots in a game
This is done by sorting the above on made
Can you write Python code for it?


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Q1.2: which team took the most and least shots in a game.
We'll make a new column called total and sort on that.
Can you write Python code for it?

shot_result_by_matchup['total'] = shot_result_by_matchup.sum(axis=1)

shot_result_by_matchup.sort_values(by='total').head()


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Q1.3 Which team made the most shots as a percentage of all shots taken in a game
we'll make a derived column called make_percent and sort on that.
Can you write Python code for it?

shot_result_by_matchup['make_percent'] = \
round((shot_result_by_matchup['made'] / shot_result_by_matchup['total'])*100,1)

shot_result_by_matchup.sort_values(by='make_percent').head()

shot_result_by_matchup.sort_values(by='make_percent', ascending=False).head()


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Q2.1 which team had the lowest make percentage (in a single game) but still won that game.
Again we'll make a derived column called make_percent
Can you write Python code for it?

shot_and_game_result_by_matchup['make_percent'] = \
round((shot_and_game_result_by_matchup['made']/\
       shot_and_game_result_by_matchup.sum(axis=1))*100,1)

shot_and_game_result_by_matchup[\
shot_and_game_result_by_matchup['W']=='W'].sort_values(by='make_percent').head()

shot_and_game_result_by_matchup[\
shot_and_game_result_by_matchup['W']=='L'].sort_values(by='make_percent', \
                                                       ascending=False).head()


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
Q2.2 Did winning teams have a higher make percentage on average than losing teams?
We'll explore this question with a group and a plot
Can you write Python code for it?


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
These are likely game that were canceled
Can you write Python code for it?


I am working on the file "Pandas_Worksheet_Solutions.ipynb".
The first task is:
We see that the name of the team of the player taking the shot is the first team listed after the date. It turns out that having things structured this way is actually very convenient.
Part1: Questions about SHOT_RESULT for one team in a one game
Here It makes sense restructure our data so that each row is referring to one team in one game and the columns give us the number of shots made or missed. This can be done with either DataFrame.groupby or pandas.pivot_table which I show below.
Can you write Python code for it?

shot_result_by_matchup = df.groupby(['MATCHUP','SHOT_RESULT']).size().unstack()

shot_result_by_matchup.head()

df.pivot_table(index='MATCHUP', columns='SHOT_RESULT', \
               values='W',aggfunc=lambda x: len(x)).head()


I am working on the file "Paper-figures.ipynb".
The first task is:
Comparing phase diagrams
Can you write Python code for it?

orb = create_holoviews('data/0_to_2T_4x4_angles_misaligned_with_electric_field.h5')
no_orb = create_holoviews('data/0_to_2T_4x4_angles_misaligned_no_orbital_with_electric_field.h5')

%%opts Path (color='g')
%%opts Image.d [colorbar=True cbar_ticks=np.linspace(0, 140, 5).tolist()]
%%opts Layout [vspace=0.20 hspace=0.15 sublabel_position=(-0.07, 0.79) sublabel_size=10 sublabel_format='({alpha})']
%%opts VLine (linewidth=0.5 color='k')

test = orb.Phase_diagram.Band_gap[0, 0.5]

comparing_phase_diagrams = (
    ticks((no_orb.Phase_diagram.Band_gap * hv.VLine(1)).relabel(r"$\bm{B} \parallel \hat{x}, \; \bm{A} = 0$")[0.5, 0], x=False)
    + ticks(no_orb.Phase_diagram.Band_gap.relabel(label=r"$\bm{B} \parallel \hat{z}, \; \bm{A} = 0$")[0, 0.5], x=False, y=False)
    + ticks(orb.Phase_diagram.Band_gap.relabel(r"$\bm{B} \parallel \hat{x}, \; \bm{A} \ne 0$")[0.5, 0])
    + ticks(orb.Phase_diagram.Band_gap.relabel(label=r"$\bm{B} \parallel \hat{z}, \; \bm{A} \ne 0$", group='d', depth=2)[0, 0.5], y=False)).cols(2)

comparing_phase_diagrams

# renderer.save(comparing_phase_diagrams, 'paper/figures/comparing_phase_diagrams', fmt='pdf')


I am working on the file "Paper-figures.ipynb".
The first task is:
Comparing phase diagrams, misaligned fields
Can you write Python code for it?

%%opts Path (color='g')
%%opts Image.d [colorbar=True cbar_ticks=np.linspace(0, 120, 5).tolist()]
%%opts Layout [vspace=0.20 hspace=0.15 sublabel_position=(-0.07, 0.79) sublabel_size=10 sublabel_format='({alpha})']

kys = no_orb.Phase_diagram.Band_gap.keys()

test = orb.Phase_diagram.Band_gap[nearest(kys, 0.05), 0.5]

misaligned = (
    ticks(no_orb.Phase_diagram.Band_gap.relabel(label=r"$\bm{B} \parallel (10, 1, 0)^T, \; \bm{A} = 0$")[0.5, nearest(kys, 0.05)], x=False)
    + ticks(no_orb.Phase_diagram.Band_gap.relabel(label=r"$\bm{B} \parallel (0, 1, 10)^T, \; \bm{A} = 0$")[nearest(kys, 0.05), 0.5], x=False, y=False)
    + ticks(orb.Phase_diagram.Band_gap.relabel(label=r"$\bm{B} \parallel (10, 1, 0)^T, \; \bm{A} \ne 0$")[0.5, nearest(kys, 0.05)])
    + ticks(orb.Phase_diagram.Band_gap.relabel(label=r"$\bm{B} \parallel (0, 1, 10)^T, \; \bm{A} \ne 0$", group='d', depth=2)[nearest(kys, 0.05), 0.5], y=False)).cols(2)

misaligned

# renderer.save(misaligned, 'paper/figures/misaligned', fmt='pdf')


I am working on the file "Paper-figures.ipynb".
The first task is:
Eigenvalue problem graphic
Uncomment the lower cells and start an ipcluster to calculate the spectrum.
Can you write Python code for it?

# import os
# from scripts.hpc05 import HPC05Client
# os.environ['SSH_AUTH_SOCK'] = os.path.join(os.path.expanduser('~'), 'ssh-agent.socket')
# cluster = HPC05Client()

# v = cluster[:]
# v.use_dill()
# lview = cluster.load_balanced_view()
# len(v)

# %%px
# import sys
# import os
# sys.path.append(os.path.join(os.path.expanduser('~'), 'orbitalfield'))
# import kwant
# import numpy as np
# from fun import *
# lead = make_3d_wire()
# p = make_params(orbital=False, B_x=1)

lead = make_3d_wire()
p = make_params(orbital=False, B_x=1)
mus = np.linspace(0, 35, 2000)

if os.path.exists('data/gaps_plot.npy'):
    gaps = np.load('data/gaps_plot.npy')
else:
    print('Start cluster with the cells above.')
    gaps = lview.map_async(lambda mu: find_gap(lead, p, ((1, 0, 0), mu, True), tol=1e-4), mus).result()
    np.save('data/gaps_plot', gaps)
if os.path.exists('data/spectrum_ev_plot.npy'):
    Es = np.load('data/spectrum_ev_plot.npy')
else:
    Es = np.array([kwant.physics.Bands(lead, args=[p])(k=0) for p.mu in mus])
    np.save('data/spectrum_ev_plot', Es)

%%output fig='svg'
%%opts VLine (lw=0.5) HLine (lw=0.5, color='g')
%%opts Layout [vspace=.35 aspect_weight=1 sublabel_position=(-0.3, 0.9) sublabel_format='({alpha})' sublabel_size=13]
%%opts Overlay [yticks=3 aspect=1.5 vspace=0.]

E_dim = hv.Dimension(('E_k0', r'$E(k=0)$'), unit='meV')
spectrum = hv.Path((mus, Es), kdims=[dimensions.mu, E_dim])
ind_E = 100
idx = np.argsort(np.min(np.abs(Es), axis=1))
VPoints = hv.Points([(mus[ind_E], E) for E in Es[ind_E]])
p.mu = 0
phase_bounds = np.sort(find_phase_bounds(lead, p, (1, 0, 0), num_bands=40).real)[::2]
HPoints = hv.Points([(x, 0) for x in phase_bounds if x > 0])(style={'color': 'g'})
ev_plot = (spectrum * hv.VLine(mus[ind_E]) * VPoints * HPoints * hv.HLine(0))[:35, -10:10]

bool_array = np.array(np.digitize(mus, phase_bounds)%2, dtype=bool)

gaps_plot = (spectrum
             * hv.Area((mus, np.array(gaps) * bool_array))(style={'facecolor': '#FF6700'})
             * hv.Area((mus, np.array(gaps) *...


I am working on the file "Paper-figures.ipynb".
The first task is:
Full phase diagram for superconductor on side of wire
Band gaps
Can you write Python code for it?

%%opts Layout [vspace=0] Image (cmap=mymap clims=(-197, 197))
%%opts Layout [sublabel_position=(-0.4, 0.9) sublabel_format='({alpha})' sublabel_size=13]
%%opts Path (color='g')

im1 = sc_on_side_no_orb.Phase_diagram.Band_gap[0.5, 0]
im2 = sc_on_side.Phase_diagram.Band_gap[0.5, 0]

im1 = im1.relabel(r"$\bm{B} \parallel x, \; \bm{A} = 0$", depth=1)
im2 = im2.relabel(r"$\bm{B} \parallel x, \; \bm{A} \ne 0$", depth=1)
max1 = np.nanmax(im1.Im.Band_gap.data)
max2 = np.nanmax(im2.Im.Band_gap.data)
max_gap = np.max((max1, max2))

sc_on_side_hist = (ticks(im1, x=False).hist(bin_range=(0, max_gap)) +
                   ticks(im2).hist(bin_range=(0, max_gap)))
sc_on_side_hist.cols(1)

# print the maximum band gaps
print("""The maximum band gap of the top plot is {:.4} meV.
The maximum band gap of the lower plot is {:.4} meV.""".format(max1, max2))

%%opts Layout [vspace=0] Image (cmap=mymap clims=(-197, 197))
%%opts Layout [sublabel_position=(-0.4, 0.9) sublabel_format='({alpha})' sublabel_size=13]
%%opts Path (color='g')

im1_alpha100 = sc_on_side_no_orb_alpha100.Phase_diagram.Band_gap[0.5, 0]
im2_alpha100 = sc_on_side_alpha100.Phase_diagram.Band_gap[0.5, 0]

im1_alpha100 = im1_alpha100.relabel(r"$\bm{B} \parallel x, \; \bm{A} = 0$", depth=1)
im2_alpha100 = im2_alpha100.relabel(r"$\bm{B} \parallel x, \; \bm{A} \ne 0$", depth=1)
max1_alpha100 = np.nanmax(im1_alpha100.Im.Band_gap.data)
max2_alpha100 = np.nanmax(im2_alpha100.Im.Band_gap.data)
max_gap_alpha100 = np.max((max1_alpha100, max2_alpha100))

sc_on_side_hist_alpha100 = (ticks(im1_alpha100, x=False).hist(bin_range=(0, max_gap_alpha100)) +
                   ticks(im2_alpha100).hist(bin_range=(0, max_gap_alpha100)))
(sc_on_side_hist_alpha100).cols(1)

# renderer.save(sc_on_side_hist, 'paper/figures/sc_on_side_hist', fmt='pdf')

# print the maximum band gaps
print("""The maximum band gap of the top plot is {:.4} meV.
The maximum band gap of the lower plot is {:.4} meV.""".format(max1_alpha100, max2_alpha100))


I am working on the file "Paper-figures.ipynb".
The first task is:
Load data and create a custom cmap
Can you write Python code for it?

import matplotlib.cm
import matplotlib.colors as mcolors
colors1 = matplotlib.cm.binary_r(np.linspace(0.5, 1, 128))
colors2 = matplotlib.cm.gist_heat_r(np.linspace(0, 0.8, 127))
colors = np.vstack((colors1, colors2))
mymap = mcolors.LinearSegmentedColormap.from_list('my_colormap', colors)
sc_on_side_alpha100 = create_holoviews('data/0_to_2T_1x1_angles_sc_on_side_mu_ranging_from_minus_2_to_plus_2_full_phase_diagram_with_correction_A_alpha100.h5')
sc_on_side_no_orb_alpha100 = create_holoviews('data/0_to_2T_1x1_angles_sc_on_side_mu_ranging_from_minus_2_to_plus_2_full_phase_diagram_with_correction_A_no_orbital_alpha100.h5')
sc_on_side = create_holoviews('data/0_to_2T_1x1_angles_sc_on_side_mu_ranging_from_minus_2_to_plus_2_full_phase_diagram_with_correction_A.h5')
sc_on_side_no_orb = create_holoviews('data/0_to_2T_1x1_angles_sc_on_side_mu_ranging_from_minus_2_to_plus_2_full_phase_diagram_with_correction_A_no_orbital.h5')


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
All operations that update the weight vector is done within the WeightVector object (relevant lines in sgd_fast.pyx here). In particular we look at the  add function, which scales sample x by constant c and add it to the weight vector. We copy the add function below
Can you write Python code for it?

cdef void add(self, double *x_data_ptr, int *x_ind_ptr, int xnnz,
                  double c) nogil:
        """Scales sample x by constant c and adds it to the weight vector.
        This operation updates ``sq_norm``.
        Parameters
        ----------
        x_data_ptr : double*
            The array which holds the feature values of ``x``.
        x_ind_ptr : np.intc*
            The array which holds the feature indices of ``x``.
        xnnz : int
            The number of non-zero features of ``x``.
        c : double
            The scaling constant for the example.
        """
        cdef int j
        cdef int idx
        cdef double val
        cdef double innerprod = 0.0
        cdef double xsqnorm = 0.0

        # the next two lines save a factor of 2!
        cdef double wscale = self.wscale
        cdef double* w_data_ptr = self.w_data_ptr

        for j in range(xnnz):
            idx = x_ind_ptr[j]
            val = x_data_ptr[j]
            innerprod += (w_data_ptr[idx] * val)
            xsqnorm += (val * val)
            w_data_ptr[idx] += val * (c / wscale)

        self.sq_norm += (xsqnorm * c * c) + (2.0 * innerprod * wscale * c)


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
Getting values from $X_i$ is done using x_data_ptr[j], modifying the weight vector is done similarly. The optimizations for this step include ther aforementioned static typing, memoryviews and pointer data access. A further optimization here comes from the "nogil" aregument after the function definition. This releases the global interpreter lock, allowing for higher efficiency through multi-tasking.
Optimizations in SGD algorithm
The raw code for the SGD algorithm in scikit-learn is here. Since we're assuming OPTIMAL learning rate, hinge loss and L2 regularization, we copy the relevant parts of the code below
Can you write Python code for it?

def _plain_sgd(np.ndarray[double, ndim=1, mode='c'] weights,
               double intercept,
               np.ndarray[double, ndim=1, mode='c'] average_weights,
               double average_intercept,
               LossFunction loss,
               int penalty_type,
               double alpha, double C,
               double l1_ratio,
               SequentialDataset dataset,
               int n_iter, int fit_intercept,
               int verbose, bint shuffle, np.uint32_t seed,
               double weight_pos, double weight_neg,
               int learning_rate, double eta0,
               double power_t,
               double t=1.0,
               double intercept_decay=1.0,
               int average=0):

    # get the data information into easy vars
    cdef Py_ssize_t n_samples = dataset.n_samples
    cdef Py_ssize_t n_features = weights.shape[0]

    cdef WeightVector w = WeightVector(weights, average_weights)
    cdef double* w_ptr = &weights[0]
    cdef double *x_data_ptr = NULL
    cdef int *x_ind_ptr = NULL
    cdef double* ps_ptr = NULL

    # helper variables
    cdef bint infinity = False
    cdef int xnnz
    cdef double eta = 0.0in
    cdef double p = 0.0
    cdef double update = 0.0
    cdef double sumloss = 0.0
    cdef double y = 0.0
    cdef double sample_weight
    cdef double class_weight = 1.0
    cdef unsigned int count = 0
    cdef unsigned int epoch = 0
    cdef unsigned int i = 0
    cdef int is_hinge = isinstance(loss, Hinge)
    cdef double optimal_init = 0.0
    cdef double dloss = 0.0
    cdef double MAX_DLOSS = 1e12

    # q vector is only used for L1 regularization
    cdef np.ndarray[double, ndim = 1, mode = "c"] q = None
    cdef double * q_data_ptr = NULL

    if penalty_type == L2:
        l1_ratio = 0.0

    if learning_rate == OPTIMAL:
        typw = np.sqrt(1.0 / np.sqrt(alpha))
        # computing eta0, the initial learning rate
        initial_eta0 = typw / max(1.0, loss.dloss(-typw, 1.0))
        # initialize t such that eta at first sample equals eta0a
      ...


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
Optimizations in Data Storage
Scikit-learn first convert the data into a ArrayDataset object, which is also implemented in Cython. The code that converts the raw input data to a ArrayDataset is linked here.
First, we look at how ArrayDataSet is initialized. At lines 195-196 we see that ArrayDataSet creates memoryviews to the numpy array X and Y and assign their pointers to X_data_ptr and Y_data_ptr.
Can you write Python code for it?

self.X_data_ptr = <double *>X.data
self.Y_data_ptr = <double *>Y.data


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
The WeightVector object is defined here. Its mechanism of data storage is similar to ArrayDataset, (relevant line copied below).
Can you write Python code for it?

cdef double *wdata = <double *>w.data

        if w.shape[0] > INT_MAX:
            raise ValueError("More than %d features not supported; got %d."
                             % (INT_MAX, w.shape[0]))
        self.w = w
        self.w_data_ptr = wdata


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
The purpose of SVM is to find a weight vector $w$ and intercept $b$ such that the line described by $w_0X_1+w_0X_2+b=0$ best separates the two classes of points. Stochastic gradient descent will obtain the optimal $w$ and $b$ from the data.
Implementation in Python
Can you write Python code for it?

class Lossfunction:
    def loss(self, p, y):
        return 0
    def dloss(self, p, y):
        return 0

class Hinge(Lossfunction):
    def __init__(self, threshold=1):
        self.threshold=threshold
    def loss(self, p, y):
        z=p*y

        if z<=self.threshold:
            return (self.threshold-z)
        return 0
    def _dloss(self, p ,y):
        z= p*y
        if z<=self.threshold:
            return -y
        return 0

import numpy as np
class sgddata:
    def __init__(self, X, Y, sample_weights=0,seed=None):
        if len(X)!=len(Y):
            raise IndexError('X, Y not same length')
        self.X = np.array(X)
        self.Y = np.array(Y)
        self.sample_weights = sample_weights
        self.n_samples = self.X.shape[0]
        self.n_features = self.X.shape[1]
        self.current_index = -1
    def __next__(self):
        self.current_index += 1
        return self.X[self.current_index], self.Y[self.current_index]
    def _reset(self):
        self.current_index = -1
    def shuffle(self ,seed=None):
        np.random.seed(seed)
        idx = np.random.permutation(self.n_samples)
        self.X = self.X[idx]
        self.Y = self.Y[idx]
    def __str__(self):
        return '{},{}'.format(str(self.X), str(self.Y))

from time import time
def sgd( weights,
               intercept,
               loss,
               penalty_type,
               alpha,
               dataset,
               n_iter, fit_intercept=True,
               verbose=False, shuffle=True, seed=None,
               weight_pos=1, weight_neg=1,
               eta0=0,
               t=1.0,
               intercept_decay=1.0):
    MAX_DLOSS = 1e12
    eta = eta0
    l1_ratio = 0.0
    sumlosslist = []
    n_samples=dataset.n_samples
    typw = np.sqrt(1.0 / np.sqrt(alpha))
    # computing eta0, the initial learning rate
    initial_eta0 = typw / max(1.0, loss.dloss(-typw, 1.0))
    # initialize t such that eta at first sample equals eta0
    optimal_init = 1.0 / (initial_eta0 * alpha)
    t_start =...


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
We see that the _sample function takes x_data_ptr and y and make it so they point to the pointer to the start of $X_i$. x_data_ptr and y are then used by the rest of the SGD algorithm to access the current sample. The main optimizations in this data storage step, aside from static typing, are the usage of memoryviews and the usage of pointer to access data. Memoryviews allows sharing memory between data structures without copying, while data access through pointers is much faster than a Python function such as the getitem command.
Optimizations in Weights
In the SGD algorithm the weight vector is stored in a WeightVector object (relevant line copied below).
Can you write Python code for it?

cdef WeightVector w = WeightVector(weights, average_weights)


I am working on the file "Paper/SGD algorithm paper.ipynb".
The first task is:
We take a closer look at the .next() function. The .next function is implemented at lines 20-46 of seq_dataset.pyx. The ArrayDataSet object keeps a data index, which .next() calls, advancing the index by one, and then calls the ._sample function,
Can you write Python code for it?

cdef void _sample(self, double **x_data_ptr, int **x_ind_ptr,
                      int *nnz, double *y, double *sample_weight,
                      int current_index) nogil:
        cdef long long sample_idx = self.index_data_ptr[current_index]
        cdef long long offset = sample_idx * self.X_stride

        y[0] = self.Y_data_ptr[sample_idx]
        x_data_ptr[0] = self.X_data_ptr + offset
        x_ind_ptr[0] = self.feature_indices_ptr
        nnz[0] = self.n_features
        sample_weight[0] = self.sample_weight_data[sample_idx]


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
An example is seen here:

Setting up the Phases and Physics
Now we are ready to set up our phases (water and air) and the physics corresponding to each of these phases. openpnm has built in air and water phases, so we can use those. However, Gostick specifies using a water pore contact angle of 100, so we will reset this value after regenerating our fluids.
Can you write Python code for it?

air = op.phases.Air(network = sgl, name = 'air')
water = op.phases.Water(network = sgl, name = 'water')
#reset pore contact angle
water['pore.contact_angle'] = 100


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
Before we move on to setting up our fluid and physics objects, we must constrict throats in the z and y direction by a factor (Gostick et al included this tightening of throats in only these two directions to create realistic anisotropy in the model).  For his SGL simulation, Gostick uses a constriction factor of .95.  Finally, because we have changed values for pore and throat diameters (first by accounting for pores and throats that are too big, and the finally constricting throats in the y and z directions), we must recalculate all pore and throat values relying on these diameters.
Can you write Python code for it?

throats = geo.throats()
connected_pores = sgl.find_connected_pores(throats)
x1 = [sgl['pore.coords'][pair[0]][0] for pair in connected_pores]
x2 = [sgl['pore.coords'][pair[1]][0] for pair in connected_pores]
same_x = [x - y == 0 for x, y in zip(x1,x2)]
factor = [s*.95 + (not s)*1 for s in same_x]
throat_diameters = sgl['throat.diameter'][throats]*factor
geo['throat.diameter']=throat_diameters
geo.regenerate_models(exclude=['throat.diameter'])


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
Now we can plot the results including those from the paper
Can you write Python code for it?


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
OpenPNM makes it very easy to visualize the network we have generated through the "Visualization" methods.  We can create vtk files to be viewed using ParaView (downloadable at http://www.paraview.org/download/ ). If we visualize our pore network model it would appear like this (the pores have been visualized using boxes- darker boxes are larger. Because the network is so big, visualization of the throats has been left out for clarity):
Can you write Python code for it?

import openpnm.io.VTK as iovtk
iovtk.save(network=sgl, filename='sgl')


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
Running Ordinary Percolation, Fickian Diffusion, and Stokes Flow
Gostick uses ordinary percolation to spread water through his GDL before calculating relative permeability and relative diffusivity.  This way, a graph showing the relationship between saturation and relative permeability and between saturation and relative diffusivity can be created.
To run our ordinary percolation, we will:

pick inlet pores
create an Ordinary Percolation algorithm object
setup our algorithm object
run our algorithm object
call results() so that occupancy of pores and throats for each fluid will can be set and multiphysics updated
Can you write Python code for it?

inlets = sgl.pores('bottom_boundary')
used_inlets = [inlets[x] for x in range(0, len(inlets), 2)]

OP_1 = op.algorithms.OrdinaryPercolation(project=proj)
OP_1.set_inlets(pores=used_inlets)
OP_1.setup(phase=water, pore_volume='pore.volume', throat_volume='throat.volume')
OP_1.run(points=100)

data = OP_1.get_intrusion_data()
# Filter for evenly spaced sat inc. first and last
filter_pc = [data.Pcap[0]]
sat = [data.Snwp[0]]
for i, pc in enumerate(data.Pcap):
    if  data.Snwp[i] - sat[-1] > 0.05:
        filter_pc.append(pc)
        sat.append(data.Snwp[i])
filter_pc.append(data.Pcap[-1])
sat.append(data.Snwp[-1])


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
The next step will be to calculate effective diffusivity and permeability at different saturations.  Note that we want to run Fickian diffusion and Stokes flow algorithms at different points within our ordinary percolation process.
The rest of our code will exist within a loop updating our network to different stages of percolation, so that we may view our relative diffusivity and permeability at different points of saturation.
Before we add in the loop aspect, we will walk through the code that will be inside the loop.
Note that we want the algorithms that are single phase (where only the specified fluid exists in the network) to help us make our permeability and diffusivity values relative.  Any algorithm that is single phase will use the hydraulic or diffusive conductances before we recalculated based on occupancy.  This calls for our conductance parameter to be 'hydraulic_conductance' or 'diffusive_conductance' instead of 'conduit_hydraulic_conductance' or 'conduit_diffusive_conductance'.
The need for all these different algorithms can be made clearer by the equation relating effective permeability to the absolute permeability and relative permeability:
$$K_{eff, p}(s_p) = K*K_{r, p}(s_p)$$
|Symbol|Description|
|-------------------|-------------------------------------------------------------|
|$$K_{eff, p}(s_p)$$|effective permeability of phase p as a function of saturation|
|$$K$$              |absoulte permeability (or single phase permeability)         |
|$$K_{r, p}(s_p)$$  |relative permeability of phase p as a function of saturation |
Therefore, relative permeability can be found by dividing the effective permeability by the absolute permeability.  Thus the need for a single phase algorithm (absolute permeability) for every multi phase algorithm (effective permeability).
The same goes for relative diffusivity, which has an very similar equation that looks like this:
$$D_{eff, p}(s_p) = D*D_{r, p}(s_p)$$
Can you write Python code for it?

perm_air = {'0': [], '1': [], '2': []}
diff_air = {'0': [], '1': [], '2': []}
perm_water = {'0': [], '1': [], '2': []}
diff_water = {'0': [], '1': [], '2': []}

max_Pc = max(OP_1['throat.invasion_pressure'])

num_seq = 20
pore_volumes = sgl['pore.volume']
throat_volumes = sgl['throat.volume']
totV = np.sum(pore_volumes) + np.sum(throat_volumes)

single_perms_air = [None, None, None]
single_diffs_air = [None, None, None]
single_perms_water = [None, None, None]
single_diffs_water = [None, None, None]
bounds = [['front', 'back'], ['left', 'right'], ['top', 'bottom']]

for bound_increment in range(len(bounds)):
    # Run Single phase algs effective properties
    BC1_pores = sgl.pores(labels=bounds[bound_increment][0]+'_boundary')
    BC2_pores = sgl.pores(labels=bounds[bound_increment][1]+'_boundary')
    # Ka
    Stokes_alg_single_phase_air = op.algorithms.StokesFlow(network=sgl, phase=air)
    Stokes_alg_single_phase_air.setup(conductance='throat.hydraulic_conductance')
    Stokes_alg_single_phase_air.set_value_BC(values=0.6, pores=BC1_pores)
    Stokes_alg_single_phase_air.set_value_BC(values=0.2, pores=BC2_pores)
    Stokes_alg_single_phase_air.run()
    single_perms_air[bound_increment] = Stokes_alg_single_phase_air.calc_eff_permeability()
    proj.purge_object(obj=Stokes_alg_single_phase_air)
    # Da
    Fickian_alg_single_phase_air = op.algorithms.FickianDiffusion(network=sgl,phase=air)
    Fickian_alg_single_phase_air.setup(conductance='throat.diffusive_conductance')
    Fickian_alg_single_phase_air.set_value_BC(values=0.6, pores=BC1_pores)
    Fickian_alg_single_phase_air.set_value_BC(values=0.2, pores=BC2_pores)
    Fickian_alg_single_phase_air.run()
    single_diffs_air[bound_increment] = Fickian_alg_single_phase_air.calc_eff_diffusivity()
    proj.purge_object(obj=Fickian_alg_single_phase_air)
    # Kw
    Stokes_alg_single_phase_water = op.algorithms.StokesFlow(network=sgl, phase=water)
    Stokes_alg_single_phase_water.setup(conductance='throat.hydraulic_conductance')
   ...


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
We are now ready to establish physical properties for our fluid objects. To do this, we will: 1) create physics objects associated with our fluids (by using StandardPhyics we don't have to add methods for calculating each property because they are already included) 2) use our regenerate_physics() method to calculate these properties. One physics object is required for each combination of phase and geometry.
Can you write Python code for it?

phys_water = op.physics.Standard(network=sgl, phase=water, geometry=geo)
phys_air = op.physics.Standard(network=sgl, phase=air, geometry=geo)
phys_water_b = op.physics.Standard(network=sgl, phase=water, geometry=boun)
phys_air_b = op.physics.Standard(network=sgl, phase=air, geometry=boun)


I am working on the file "PaperRecreations/Gostick2007.ipynb".
The first task is:
We now define a helper function to update the phases and properties with the results of the OP algorithm. The multiphase conduit conductance model looks at the phase occupancy in the conduits made by the 1/2 pore - throat - 1/2 pore neighbor elements. When the mode is 'strict' the phase must occupy all three elements for the conduit to be considered open to flow for that phase. If the phase is not present in at least one of the elements in the conduit then the throat conductance is divided by 6 orders of magnitude. In this way the conductivity is severely reduced by the presence of the other phase and flow must go around, thus decreasing the permeability/diffusivity of the network.
Can you write Python code for it?

def update_phase_and_phys(results):
    water['pore.occupancy'] = results['pore.occupancy']
    air['pore.occupancy'] = 1-results['pore.occupancy']
    water['throat.occupancy'] = results['throat.occupancy']
    air['throat.occupancy'] = 1-results['throat.occupancy']
    #adding multiphase conductances
    mode='strict'
    phys_air.add_model(model=mods.physics.multiphase.conduit_conductance,
                       propname='throat.conduit_diffusive_conductance',
                       throat_conductance='throat.diffusive_conductance',
                       mode=mode)
    phys_water.add_model(model=mods.physics.multiphase.conduit_conductance,
                         propname='throat.conduit_diffusive_conductance',
                         throat_conductance='throat.diffusive_conductance',
                         mode=mode)
    phys_air.add_model(model=mods.physics.multiphase.conduit_conductance,
                       propname='throat.conduit_hydraulic_conductance',
                       throat_conductance='throat.hydraulic_conductance',
                       mode=mode)
    phys_water.add_model(model=mods.physics.multiphase.conduit_conductance,
                         propname='throat.conduit_hydraulic_conductance',
                         throat_conductance='throat.hydraulic_conductance',
                         mode=mode)
    phys_air_b.add_model(model=mods.physics.multiphase.conduit_conductance,
                         propname='throat.conduit_diffusive_conductance',
                         throat_conductance='throat.diffusive_conductance',
                         mode=mode)
    phys_water_b.add_model(model=mods.physics.multiphase.conduit_conductance,
                           propname='throat.conduit_diffusive_conductance',
                           throat_conductance='throat.diffusive_conductance',
                           mode=mode)
    phys_air_b.add_model(model=mods.physics.multiphase.conduit_conductance,
                         propname='throat.conduit_hydraulic_conductance',
        ...


I am working on the file "PaperRecreations/Wu2010_part_a.ipynb".
The first task is:
As the paper requires some lengthy calculation we have split it into parts and put the function in a separate notebook to be re-used in each part. The following code runs and loads the shared functions into this kernel
Can you write Python code for it?


I am working on the file "PaperRecreations/Wu2010_part_a.ipynb".
The first task is:
The main function runs the simulation for a given network size 'n' and number of points for the relative diffusivity curve. Setting 'npts' to 1 will return the single phase diffusivity. the network size is doubled in the z direction for percolation but the diffusion calculation is effectively only calculated on the middle square section of length 'n'. This is achieved by copying the saturation distribution from the larger network to a smaller one.
We can inspect the source in this notebook by running a code cell with the following: simulation??
Run the simulation once for a network of size 8 x 8 x 8
Can you write Python code for it?

x_values, y_values = simulation(n=8)

plt.figure()
plt.plot(x_values, y_values, 'ro')
plt.title('normalized diffusivity versus saturation')
plt.xlabel('saturation')
plt.ylabel('normalized diffusivity')
plt.show()


I am working on the file "PaperRecreations/Wu2010_part_b.ipynb".
The first task is:
And finally extract the g(S) function for relative diffusivity.
Can you write Python code for it?


I am working on the file "PaperRecreations/Wu2010_part_b.ipynb".
The first task is:
Set the workspace loglevel to not print anything
Can you write Python code for it?

wrk = op.Workspace()
wrk.loglevel=50

%run shared_funcs.ipynb


I am working on the file "PaperRecreations/Wu2010_part_b.ipynb".
The first task is:
We can run multiple times as the network sizes are randomly generated between a given range we can obtain an average
Can you write Python code for it?

x_values = []
y_values = []

for ensemble in range(10):
    x_ensemble, y_ensemble = simulation(n=8)
    x_values.append(x_ensemble)
    y_values.append(y_ensemble)

x_values = np.asarray(x_values).flatten()
y_values = np.asarray(y_values).flatten()
plt.figure()
from matplotlib.font_manager import FontProperties
fontP = FontProperties()
fontP.set_size('small')

wu_average_x_values = [0.004, 0.021, 0.052, 0.081, 0.129, 0.162, 0.186, 0.219, 0.261,
                       0.286, 0.324, 0.363, 0.42, 0.478, 0.531, 0.586, 0.64, 0.698, 0.747, 0.802]
wu_average_y_values = [0.118, 0.113, 0.105, 0.096, 0.085, 0.078, 0.07, 0.062, 0.054, 0.049, 0.04,
                       0.033, 0.027, 0.02, 0.012, 0.006, 0.003, 0.002, 0.002, 0.002]

p1, = plt.plot(x_values, y_values, 'ko')
p2, = plt.plot(wu_average_x_values, wu_average_y_values, 'ro')
plt.title('normalized diffusivity versus saturation')
plt.xlabel('saturation')
plt.ylabel(r'$\frac{D_e}{D_b}$')
#plt.ylim([0, .15])
plt.xlim([0, 1])
plt.legend([p1, p2],
                   [r'$\frac{D_e}{D_b} = f(\epsilon, \phi)g(s, \phi)$' + '\n' + r'$X = 1.8$' +
                   '\n' + r'$Z_t = 2.0$' + '\n' + r'$Z_i = 4.0$' + '\n' + r'$\beta = 1.0$' + '\n' + r'$n = 14$', "Wu's results"])

plt.show()


I am working on the file "PaperRecreations/Wu2010_part_c.ipynb".
The first task is:
Set the workspace loglevel to not print anything
Can you write Python code for it?


I am working on the file "Paper_Implementation.ipynb".
The first task is:
4. Sampling Algorithm
We create a random test set of 5 numbers for our initial walk.
Can you write Python code for it?

size_ts = 5
ts = np.random.randint(m, size=(size_ts,))
ts


I am working on the file "Paper_Implementation.ipynb".
The first task is:
For each test user, we hold out 90% of their rated items.
Can you write Python code for it?

def generate_held_outs(test_sets, R = Rnan, p = 0.9):
    l = []
    for i, test_set in enumerate(test_sets):
        ho = np.zeros((m,n), dtype = np.bool_)
        for test_user in test_set:
            items = O_u(test_user, R = R)[0]
            np.random.shuffle(items)
            items = items[:int(p*items.shape[0])]
            ho[test_user,items] = 1
        l.append(ho)
        np.save('HO'+str(i)+'.npy', ho)
    return l

np.sum(generate_held_outs(test_sets)[0])

def load_held_outs(n_folds = 4):
    return [np.load('HO'+str(i)+'.npy') for i in range(n_folds)]

held_outs = load_held_outs(4)
held_outs

def get_held_outs(test_sets, R = Rnan, p = 0.9):
    held_outs = None
    try:
        held_outs = load_held_outs(len(test_sets))
    except FileNotFoundError:
        held_outs = generate_held_outs(test_sets, R = R, p = p)
    return held_outs

def generate_maes(n_folds = 4, R = Rnan, held_ratio = 0.9, alpha = 0.9):
    test_sets = get_test_sets(n_folds)
    held_outs = get_held_outs(test_sets, R = R, p = held_ratio)
    maes = []
    for i, (test_set, held_out) in enumerate(zip(test_sets, held_outs)):
        R_test = np.copy(R)
        R_test[held_out] = np.nan
        P = get_P(R = R_test, ind = i)

        size_ts = test_set.shape[0]
        P_new = front_swap(P, test_set, dim = 2)
        P_star = P_new[size_ts:, size_ts:]
        pi = P_new[:size_ts, size_ts:]
        W = get_W(m, m - size_ts, P_star, alpha = alpha, ind = i)

        C = get_C(W, pi, test_set, alpha, ind = i)
        MAE = get_MAE(C, test_set, held_out, R = R, ind = i)

        maes.append(MAE)

    return maes
with np.errstate(divide='raise', invalid='raise'):
    maes = generate_maes()

C0 = np.load('C0.npy')
print(C0)


I am working on the file "Paper_Implementation.ipynb".
The first task is:
Using the rating similarity matrix, we can quickly compute the total similarity score for each item $o_j$ over all users, given $u_i$. By not including NaNs, we are calculating the denominators of the Type 2 probabilities.
Can you write Python code for it?

def sum_sim(u_i, R = Rnan): # Sum of similarities for any o_j, give u_i
    return np.nansum(sim(u_i, R = R), axis = 0)

sum_sim(0)


I am working on the file "Paper_Implementation.ipynb".
The first task is:
We can now calculate $P^*$ and $\pi_{\tau.}$.
Can you write Python code for it?

P_star = P_new[size_ts:, size_ts:]
P_star.shape

pi = P_new[:size_ts, size_ts:]
pi.shape


I am working on the file "Paper_Implementation.ipynb".
The first task is:
We can now define the probability function from item (movie) $o_j$ to user $u_k$, given the previous transition was from $u_i$ to $o_j$. Again, we calculate a transition probability matrix to lessen the number of computations. Note that we actually return the transpose of the transition probability matrix, to ease further calculations.
Can you write Python code for it?

def P_ou(u_i, R = Rnan):
    """
    Transition probability matrix from movie to user, given
    a base user u_i. Note that axis 0 is still the user and
    the axis 1 is the movie.
    """
    with np.errstate(divide='ignore', invalid='ignore'):
        s = sim(u_i, R = R)
        return s / np.nansum(s, axis = 0)

P_ou(0)