I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Parse reply graph
Can you write Python code for it?

start = time.time()
# REPLY = nx.read_edgelist(os.path.join(DATA_DIR, 'HiggsDiscovery_RE.edges.gz'),
#                          create_using=nx.DiGraph(), nodetype=int, data=(('weight', int),))
# REPLY.name = 'Higgs RE'
# nx.write_gpickle(REPLY, os.path.join(DATA_DIR, 'reply.gpickle'))

REPLY = nx.read_gpickle(os.path.join(DATA_DIR, 'reply.gpickle'))
print 'Reply graph loaded in:', time.time() - start
print nx.info(REPLY)


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Parse social network (follower network)
Can you write Python code for it?

start = time.time()
# SOCIAL = nx.read_edgelist(os.path.join(DATA_DIR, 'HiggsDiscovery_social.edges.gz'),
#                           create_using=nx.DiGraph(), nodetype=int, )
# SOCIAL.name = 'Higgs SOCIAL'
# nx.write_gpickle(SOCIAL, os.path.join(DATA_DIR, 'social.gpickle'))

SOCIAL = nx.read_gpickle(os.path.join(DATA_DIR, 'social.gpickle'))
print 'Social graph loaded in:', time.time() - start
print nx.info(SOCIAL)

start = time.time()
# SOCIAL_RE = nx.reverse(SOCIAL)
# SOCIAL_RE.name = 'Higgs SOCIAL reversed'
# nx.write_gpickle(SOCIAL_RE, os.path.join(DATA_DIR, 'social_re.gpickle'))
SOCIAL_RE = nx.read_gpickle(os.path.join(DATA_DIR, 'social_re.gpickle'))
print 'Social reversed graph loaded in:', time.time() - start
print nx.info(SOCIAL_RE)


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Plot spatial spread of biggest component trough time
Can you write Python code for it?

big_comp = gl_comps_10min_rt_tgt[0]

def plot_component_spatial_spread(comp, layer_to_ts, size=(10, 10), savepath=None):
    hist = Counter(sorted(big_comp['layers']))
    x, y = zip(*sorted(hist.items(), key=operator.itemgetter(0)))

    comp_time = map(lambda t: (t, layer_to_ts[t]), x)
    # Get xticks by hour
    xticks, labels = zip(*[(k, v.strftime('%H-%M')) for (k, v) in comp_time if v.minute == 0])

    fig = plt.figure(0)
    fig.set_size_inches(size)
    ax = plt.plot(x, y)
    plt.xticks(xticks, labels, rotation=45)

    plt.xlabel('Time')
    plt.ylabel('Number of retweets')
    plt.title('Spatial spread of component {}'.format(comp['component_id']))
    plt.show()

# plot_component_spatial_spread(big_comp, layer_to_ts)


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
Ah, there is Shakespeare's Macbeth, Hamlet, as well as Julius Cesar. There is also the King James version of the bible as well as Jane Austen's Emma. Let's get on to the business of making our JayZ corpus.
Based on my perusal of the nltk book, I know that there is a plaintext corpus reading function named PlaintextCorpusReader that I can use to make my corpus. So from nltk corpus function, which I access through the '.' (dot) operator, I import PlaintextCorpusReader
python
from nltk.corpus import PlaintextCorpusReader
I create a variable that I name corpus_root and assign the full system path location to where I unzipped the JayZ file I downloaded.
python
corpus_root = 'JayZ'
I then call the plain text corpus reader function with the root location and the token '.' that means grab every file in that folder.
python
wordlist = PlaintextCorpusReader(corpus_root, '.*')
I have adapted a function from nltk to reading in my corpus. This function is below. In order to make this function work, we will need to get some additional libraries of functions. In this case, we will need the regular expressions library named "re". So the first thing we need to do is import that package.
Can you write Python code for it?

import re

"""
This function takes in an object of the type PlaintextCorpusReader, and system path.
It returns an nltk corpus

It requires the regular expression package re to work
"""

#In here is where I should get rid of the rap stopwords.

def create_corpus(wordlist, some_corpus): #process the files so I know what was read in
    for fileid in wordlist.fileids():
        raw = wordlist.raw(fileid)
        raw = re.split(r'\W+', raw) ## split the raw text into appropriate words
        some_corpus.extend(raw)
        print fileid

    return some_corpus


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
Building a concordance, gets us to the area of elementary information retrieval (IR)1, think,  basic search engine. So why do we even need to “normalize” terms? We want to match U.S.A. and USA. Also when we enter roll, we would like to match Roll, and rolling. One way to do this is to stem the word. That is, reduce it down to its base/stem/root form. As such automate(s), automatic, automation all reduced to automat. Most stemmers are pretty basic and just chop off standard affixes indicating things like tense (e.g., "-ed") and possessive forms (e.g., "-'s"). Here, we'll use the most popular english language stemmer, the Potter stemmer, which comes with NLTK.
Once our tokens are stemmed, we can rest easy knowing that roll, Rolling, Rolls will all stem to roll.
1. Some of this content has been adapted from Dan Jurafsky's Stanford CS124 class
Can you write Python code for it?

porter = nltk.PorterStemmer()

stemmer = nltk.PorterStemmer()
stemmed_tokens = [stemmer.stem(t) for t in AmericanGangster_corpus]

for token in sorted(set(stemmed_tokens))[860:870]:
    print token + ' [' + str(stemmed_tokens.count(token)) + ']'


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
From the plot we see that the basketball term "roll" seems to be used extensively in the song Party Life. Let's take a closer look at this phenomenon, and determine if "roll" was used in the "basketball" sense of the term. To do this, we need to see the context in which it was used. What we really need is a concordance. Let's build one.
The first thing I want to do is to create a corpus that only contain words from the American Gangster album.
Can you write Python code for it?

AmericanGangster_wordlist = PlaintextCorpusReader(corpus_root, 'JayZ_American Gangster_.*')
AmericanGangster_corpus = create_corpus(AmericanGangster_wordlist, [])


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
If we call our function on the Jay Z sliced corpus, it should give us a score.
Can you write Python code for it?

emma = nltk.corpus.gutenberg.words('austen-emma.txt')

print "The lexical diversity score for the first 35,000 words in the Jay Z corpus is ",
        lexical_diversity(the_corpus[:35000])
print "The lexical diversity score for the first 35,000 words in the Emma corpus is ",
        lexical_diversity(emma[:35000])


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
If you read this article, you will see that Matt used the first 35,000 lyrics of each artist. For the sake of If you read the article, you will see that Matt used the first 35,000 lyrics of each artist. For the sake of simplicity, I am going to use the artist Jay Z as the subject of our analysis. So let's go and collect the first 35,000 words of the Jay Z lyrical catalog.
How are we going to do this you might ask? Well first off, you can go to your favorite search engine and search for Jay Z lyrics. On the other hand, you can actually use the rap annotation site genius(http://rap.genius.com/) to get all that information and then some. According to Genius, Jay Z has a lot of songs. On average, most rap songs are usually set with 3 verses containing sixteen bars, or sixteen sentences each.
python
35000/(16 * 3)
729
So if 16 by 3 = 48, and 48 goes into 35000 gives approximately 729 songs. That can't be right, even though Jay Z is prolific, he hasn't written 700+ songs. So I must have gotten my understanding wrong.
Can you write Python code for it?


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
Now we can go ahead and create a concordance to test if "roll" is used in the basketball (pick and roll) sense or not.
Can you write Python code for it?

AmericanGangster_lyrics = IndexedText(porter, AmericanGangster_corpus)
AmericanGangster_lyrics.concordance('roll')

print AmericanGangster_wordlist.raw(fileids='JayZ_American Gangster_Party Life.txt')


I am working on the file "HipHopDataExploration/HipHopDataExploration.ipynb".
The first task is:
We can now go ahead and figure out the number of unique words used in Jay Z's first 35,000 lyrics. An astute observer will notice that we have not done any data cleaning. For example, take a look inside a slice of the corpus, the last 10 words the_corpus[34990:35000], ['die', 'And', 'even', 'if', 'Jehovah', 'witness', 'bet', 'he', 'll', 'never'], you will see it has treated the contraction "I'm" as two separate words. The create_corpus function that we used, works by separating each contiguous chunk of alphabets separated by punctuations or space as a word. As a result contractions like "I'm" gets treated as two words. We can use the function lexical_diversity to determine the number of unique words in our Jay Z corpus.
Can you write Python code for it?

def lexical_diversity(my_text_data):
    word_count = len(my_text_data)
    vocab_size = len(set(my_text_data))
    diversity_score = word_count / vocab_size
    return diversity_score


I am working on the file "Histogram.ipynb".
The first task is:
Lets create a data frame of a column made up of 1's and 0's and another categorical column.
Can you write Python code for it?

# Class label would be categorical variable derived from binning the continuous column
x = ['Class1']*300 + ['Class2']*400 + ['Class3']*300

# Column of random 0s and 1s
y = np.random.choice([0,1], 1000)

# Dataframe from the above variables
df = pd.DataFrame({'Class':x, 'N':y})


I am working on the file "Histogram.ipynb".
The first task is:
OK, but what about using a continuous variable? We can use pandas.cut() to bin the continuous variable:
Can you write Python code for it?

# Random x data: values from 0 - 9
x = np.random.rand(1000) * 9

# Here we bin the continuous x variable into bins (I set the end points to be from)
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.cut.html
bins = pd.cut(x, [0, 3, 6, 9])
bins


I am working on the file "Histogram.ipynb".
The first task is:
OK, lets weigh the creation of the binary column, using p in random.choice():
Can you write Python code for it?

x = ['Class1']*300 + ['Class2']*400 + ['Class3']*300
y = np.random.choice([0,1], 1000, p=[0.25, 0.75])
df = pd.DataFrame({'Class':x, 'N':y})
# grouped = df.groupby('Class')
plts = df['N'].hist(by=df['Class'])


I am working on the file "HistogramExample.ipynb".
The first task is:
But note that even those don't yield quite the same results!  But we can do better!
An obvious thing to do is to simply show all of the data.
Can you write Python code for it?

# execute this cell
plt.hist(x,histtype="step")
plt.plot(x, 0*x, '|', color='k', markersize=25) #Note markersize is (annoyingly) in *points*


I am working on the file "HistogramExample.ipynb".
The first task is:
Compare this to what you get using stats.sigmaG() from the astroML package.  You'll have to import stats from astroML and give it a different name since stats right now refers to scipy.stats.
Now set the bin size accordingly, using np.arange() and plot.  Make sure that you don't throw away the last object in data set!  How many bins do you get?  How does that compare to what we were using above?
Can you write Python code for it?

binsize = # complete
bins = np.append(np.arange( # complete

plt.hist( # complete


I am working on the file "HistogramExample.ipynb".
The first task is:
How would you characterize this distribution?  Could we reasonably think of it as a normal distribution that we could characterize by some mean and standard deviation?  Maybe, but even just by looking at this plot we see that it wouldn't be a particularly good description of the data.
Now do what we would typically do in astronomy: try re-plotting the histogram a couple of different ways to see if it reveals anything more about the data.  Try using only 5 bins bins=5, 20 bins, and with left-centered bins align = "left" instead of "mid" centered bins (which is the default)
Can you write Python code for it?

plt.hist( # complete
plt.hist( # complete
plt.hist( # complete


I am working on the file "HistogramExample.ipynb".
The first task is:
If you are familiar with slicing, then you know that [::2] and [::3] say to count by 2 and count by 3.  But that isn't what they really do.  They say to take every other index of the array or every 3rd index of the array.  So, if your array is sorted (like a is), then you could use this to instead define the number of values in a bin.  That is for any given value of M
bins = np.append(np.sort(x)[::M], np.max(np.sort(x)[:-1]))

would give bins with M objects in each bin.  Note that you need to add the maximum value to set the right edge of the last bin.  Try it for M=100, 50, and 25.
Can you write Python code for it?


I am working on the file "HistogramExample.ipynb".
The first task is:
Note that the heights of these PDFs are all different!  That's because the y-axis only refers to the first histogram.  Try using normed = True so that the histograms are all normalized to 1.
Can you write Python code for it?


I am working on the file "HistogramExample.ipynb".
The first task is:
Now let's see what we get with the Gaussian and Epanechnikov kernels.  Play with the bandwidths until you get something that looks reasonable (and roughly matches) for the two kernels.  They need not be the same.
Can you write Python code for it?

PDFgaussian = kde_sklearn( # complete
PDFepanechnikov = kde_sklearn( # complete
plt.plot( # complete
plt.plot( # complete
plt.legend( # complete


I am working on the file "HistogramExample.ipynb".
The first task is:
Now let's take a first look at the data.  Use
python
plt.hist()
where that function plots a histogram of any univariate data array given as an argument.  It takes lots of other arguments too.  See (http://matplotlib.org/api/pyplot_api.html?highlight=hist#matplotlib.pyplot.hist).  You might start with histtype = "step" so that we can later add information to the plot and still see the original histogram.  See what happens if you don't do this.
Can you write Python code for it?


I am working on the file "HistogramExample.ipynb".
The first task is:
The defaults give a result that is essentially what you would get if you made a histogram with a really large number of bins.
Now let's compare what happens when we adjust the bandwidth (which is just the width of the kernel function).  Try 0.1 and 0.5.
Can you write Python code for it?

PDFtophat1 = kde_sklearn( # complete
plt.plot( # complete

PDFtophat5 = kde_sklearn( # complete
plt.plot( # complete
plt.legend( # complete


I am working on the file "HistogramExample.ipynb".
The first task is:
This is called a rug plot and now we have a better idea of where most of the data and where the gaps really are (as opposed to where the binning makes them appear to be).  However, the markers are all piled up, so we have lost all sense of the relative numbers of objects.  Are there ~10 at x=2.5 or could there be 100?
This is where Kernel Density Estimation (KDE) comes in.  (As a side note, KDE is the core of the quasar classification work that I do, which is how I got into this machine learning business in the first place.  Way before it was popular I might add!)  In short the idea here is to represent each data point not as a delta function, but rather as a distribution (e.g., a Gaussian).  Then those distributions ("kernels") are summed up to produce the PDF.  One of the advantages of this is that it combines the best of 1) the histgram and 2) the rug plot: where 1) tells us the relative height of the distribution and 2) centers the data points at the actual location of the data instead of within some arbitrary bin.
Just about any distribution can be used as the kernel, but the most common are a Gaussian kernal and an Epanechnikov kernel.  One downside of the Gaussian kernel is that the tails are technically infinite in extent.  So each point has some finite probability of being everywhere.  The Epanechnikov kernel has truncated wings.
One still has the problem of deciding the width of the kernel (e.g., for the Gaussian the "mean" is fixed at the value of the point, but how wide should you make the Gaussian?).  For my work, we do this with a self-test of the data.  Specifically, what is the optimal width such that objects with a known classification are indeed given that classification by our machine learning algorithm.  But that is the topic for another day.  For now, we'll just play with the widths by hand to see what might work best.  N.B. the widths of the kernel distribution are referred to as "bandwidth".
Can you write Python code for it?

# execute this cell to load the KDE module
# No need to try to understand what is going on here now, we'll come back to this later.
# But see the note below
from sklearn.neighbors import KernelDensity
xplot = np.linspace(x.min(),x.max(),1000)  # Use this instead of 'x' for plotting
def kde_sklearn(data, bandwidth = 1.0, kernel="linear"):
    kde_skl = KernelDensity(bandwidth = bandwidth, kernel=kernel)
    kde_skl.fit(data[:, np.newaxis])
    log_pdf = kde_skl.score_samples(xplot[:, np.newaxis]) # sklearn returns log(density)

    return np.exp(log_pdf)


I am working on the file "HistogramExample.ipynb".
The first task is:
This is pretty different from the histogram that we started out with, isn't it?
Lastly, we have used 1000 points, so you aren't seeing the kernel shape for the individual points.  Try remaking $x$ with only 15 points and see what this looks like.  Adjust the figure size (using figsize), the bandwidth and the axis limits until you can see the differences between the two kernels.  Play with the bandwidths to see what affect they have now that you can see the individual kernels.
Can you write Python code for it?

plt # Complete following the above solution


I am working on the file "HistogramExample.ipynb".
The first task is:
We find that small changes in parameters to the histogram function significantly change the PDF.  That's bad, because the underlying data clearly have not changed.
One of the problems with histograms is that some bins end up with little (or no) data.  We can fix this by making variable-width bin sizes that have the same number of objects in each bin.  How can we do this?
Can you write Python code for it?

#Execute this cell
a = np.linspace(1,42,num=42)
print a
print a[::2]
print a[::3]


I am working on the file "HistogramExample.ipynb".
The first task is:
You have some one-dimensional ("univariate") data that you would like to try to understand.  Where by "understand" we mean "know the distribution in the measured space", i.e., you want to know the probability distribution function (PDF).  The easiest way to do that is to make a histogram.  Simple, right?
Let's work through some examples to see what problems we encounter and how we might overcome them.
Can you write Python code for it?

# Execute this cell to generate a univariate data array, x
# this is the same data used in Ivezic, Figure 6.5
np.random.seed(0)
N = 1000
mu_gamma_f = [(5, 1.0, 0.1),
              (7, 0.5, 0.5),
              (9, 0.1, 0.1),
              (12, 0.5, 0.2),
              (14, 1.0, 0.1)]
true_pdf = lambda x: sum([f * stats.cauchy(mu, gamma).pdf(x)
                          for (mu, gamma, f) in mu_gamma_f])
x = np.concatenate([stats.cauchy(mu, gamma).rvs(int(f * N))
                    for (mu, gamma, f) in mu_gamma_f])
np.random.shuffle(x)
x = x[x > -5]
x = x[x < 25]


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Color of your histogram
You can use the presets or array_like of colors.
Can you write Python code for it?

plt.hist(data, color='red');

plt.hist(data, color=[0.2, 0.3, 0.8, 0.3]); # RGBA


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Cumulative histogram
This is to create the cumulative histogram. Use cimulative=True, so that now each bin has its counts and also all the counts of the previous bins.
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Different draw types
Use the histtype= option for other draw options of your histogram. Basics are:

bar  -> Traditional bar type histogram
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Fit using Kernel Density Estimation
Instead of specifying a distribution we can fit the best probability density function. This can be achieved thanks to the non-parametric techique of kernel density estimation.
KDE is a non parametric way to estimate the probability density function of a random variable.
How it works?
Suppose $(x_1, x_2, ..., x_n)$ are i.i.d. with unknown density $f$. We want to estimate the shape of this function $f$. Its kernel density estimator is
$ \hat{f}{h}(x) = \frac{1}{n} \sum{i=1}^{n}(x-x_i) = \frac{1}{nh}\sum_{i=1}^{n}K\left(\frac{x-x_i}{h}\right)$
where the $K()$ is the kernel. Kernel is a non-negative function that intergrates to one and has mean zero, also h>0 is a smoothing parameter called bandwidth.
A kernel with a subscript h is called a scaled kernel and is defined as $K_h(x)=\frac{1}{h}K(\frac{x}{h})$.
Usually one wants to use small $h$, but is always a trade of between the bias of the estimator and its variance.
Kernel functions commonly used:
    - uniform
    - triangular
    - biweight
    - triweight
    - Epanechinikov
    - normal
More under https://en.wikipedia.org/wiki/Kernel_(statistics)#Kernel_functions_in_common_use
In python this is done using the  scipy.stats.kde  submodule.
For gaussian kernel density estimation we use the gaussian kde
Can you write Python code for it?

from scipy.stats import gaussian_kde
pdf_gaus = gaussian_kde(fit_data);
pdf_gaus

pdf_gaus = pdf_gaus.evaluate(x);  # get the "y" values from the pdf for the "x" axis, this is an array

pdf_gaus

plt.hist(fit_data, normed=1);
plt.plot(x, pdf_gaus, 'k', lw=3)

plt.plot(x,fit_gaus_func, lw=4, label="fit");
plt.plot(x, pdf_gaus, 'k', lw=3, label="KDE");
plt.legend();


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
In this case:

n : is an array or a list of arrays that hold the values of the histogram bins. (Careful in case of the weights and/or normed options are used.
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Label your histogram
Use the label=string option. This takes a string or a sequence of strings.
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
N.B.:  Notice the difference in the two fit curves! This comes from the fact that the Gaussian kernel is a mixture of normal distrubutions; a Gaussian mixture may be skew or heavy-, light-tailed or multimodal. Thus it does not assume the original distrubution of any particular form.
Fit using Scipy's Optimize submodule
Scipy comes with an optimize submodule that provides several commonly used optimization algorithms.
One of the easiest is curve_fit, which uses non-linear least squares to fit a function $f$ to the data. It assumes that :
$ y_{data} = f(x_{data}, *params) + eps$
The declaration of the function is :
scipy.optimize.curve_fit(f, xdata, ydata, p0=None, sigma=None, absolute_sigma=False, check_finite=True, bounds=(-inf, inf), method=None, jac=None, **kwargs)
where
- f : model function (callable) f(x, ... ) the independent variable must be its first argument
- xdata : sequence or array (array (k,M) when we have multiple funcitons with k predictors)
- ydata : sequence of dependent data
- p0 : initial guess for the parameters
- sigma : if it is not set to None this provides the uncertainties in the ydata array. These are used as weights in the LSquere problem, i.e. minimizing : $\sum{\left(\frac{f(xdata, popt)-ydata}{sigma}\right)^2}$. If set to None, uncertainties are assumed to be 1.
-absolute_sigma : (bool) When False, sigma denotes relative weights of datapoints. The returned covariance matrix is based on estimated errors in the data and is not affected by the overall magnitude of the values in sigma. Only the relative magnitudes of the sigma values matter. If true, then sigma describes one standard deviation errors of the input data points. The estimated covariance in pcov is based on these values.
- method : 'lm', 'trf', 'dogbox' (N.B.: lm does not work when the number of observations is less than the number of variables)
The function returns
- popt : array of optimal values for the parameters so that the sum of the squared error of $f(xdata, popt)-ydata$ is minimized

pcov : the covariance...
Can you write Python code for it?

from scipy.optimize import curve_fit

## define the model function:
def func(x, a,b,c):
    return a*np.exp(-b*x)+c

## the x-points
xdata = np.linspace(0,4,50);

## get some data from the model function...
y = func(xdata, 2.5, 1.3, 0.5)

##and then add some gaussian errors to generate the "data"
ydata = y + 0.2*np.random.normal(size=len(xdata))

## now run the curve_fit()
popt, pcov = curve_fit(func, xdata, ydata)

popt

pcov

### To constrain the optimization to the region of 0<a<3.0 , 0<b<2 and 0<c<1
popt, pcov = curve_fit(func, xdata, ydata, bounds=(0, [3., 2., 1.]))

popt

pcov


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Normalizing your histogram
To normalize a histogram use the normed=True option.
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Now, looking at this function norm we see it has the loc option and the scale option.
 loc is the mean value and scale the standard deviation
To fit a gaussian on top of the histogram, we need the normed histogram and also to get the mean and std of the gaussian that fits the data. Therefore we have
Can you write Python code for it?

plt.hist(fit_data, normed=True);

mean, std = norm.fit(fit_data);

mean

std


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Or using a textbox...
Can you write Python code for it?

plt.hist(data);
plt.ylim(0,100);
#plt.text(800,80,textstr);
plt.annotate(textstr, xy=(0.7, 0.8), xycoords='axes fraction') # annotate for specifying the
                                                                # fraction of the canvas


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Orientation of the bins
You can orient the histogram vertical or horizontal using the orientation option.
Can you write Python code for it?

plt.hist(data, orientation="horizontal");

plt.hist(data, orientation="vertical");


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Raise your histogram using bottom
You can raise your histogram by adding either a scalar (fixed) amount on your y-axis, or even an array-like raise.
To do this use the bottom=(array,scalar,None) option
Can you write Python code for it?

plt.hist(data, weights=weights, bottom=5);

nbins = 10
bot = 5*np.random.rand(nbins)

plt.hist(data, bins=nbins, bottom=bot);


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Range of histogram
Use the range=(tuple) option
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Relative width of the bars
The option rwidth=(scalar,None) defines the relative width of the bars as a fraction of the bin width. If None (default) automatically computes the width.
Can you write Python code for it?

plt.hist(data);

plt.hist(data, rwidth=0.2);

plt.hist(data, rwidth=0.8);


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Stack multiple histograms
To stack more than one histogram use the stacked=True option.
If True multiple data are stacked on top of each other, otherwise, if False multiple data are aranged side by side (or on-top of each other)
Can you write Python code for it?

data2 = np.random.rand(500)*1300;

plt.hist(data, stacked=True);
plt.hist(data2, stacked=True);


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Then create the string and add these values
Can you write Python code for it?

textstr = 'Entries=$%i$\nMean=$%.2f$\n$\sigma$=$%.2f$'%(entries, mean, stdev)

plt.hist(data, label=textstr);
plt.ylim(0,100);
plt.legend(loc='best',markerscale=0.01);


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
Then we create the curve for that using the norm.pdf in the range of fit_data.min() and fit_data.max()
Can you write Python code for it?

x = np.linspace(fit_data.min(), fit_data.max(), 1000);

fit_gaus_func = norm.pdf(x, mean, std);

plt.hist(fit_data, normed=True);
plt.plot(x,fit_gaus_func, lw=4);


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
This assures that the integral of the distribution is equal to unity.
If stacked is also True, the sum of the histograms is normalized to 1.

Special Normalize
However, sometimes it is useful to visualize the height of the bins to sum up to unity.
For this we generate weights for the histogram. Each bin has the weight:  1/(number of data points)
N.B. : Using this technique you MUST NOT USE the normed=True option.
This way adding up the bars will give you 1.
Can you write Python code for it?

weights = np.ones_like(data)/len(data);

plt.hist(data, weights=weights);  ## We have NOT used the normed = True option


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
What about the fit errors?
To get the standard deviation of the parameters simply get the square root of the sum of the diagonal elements of the covariance matrix.
Can you write Python code for it?

errors = []
for i in range(len(popt)):
    try:
        errors.append(np.absolute(pcov[i][i])**0.5)
    except:
        errors.append(0.00)

for i in range(len(popt)):
    print popt[i],"+/-",errors[i]


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
barstacked -> bar-type where multiple data are stacked on-top of the other
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
my_bins : Is an array. This holds the edges of the bins. The length of the my_bins is nbins+1 (that is nbins left edges and the right edge of the last bin). This is always a single array, even if more than one datasets are passed in.
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
my_patches : This is a silent list of the individual patches that are used to create the histogram or list of such list if multiple datasets are plotted.
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
step -> To create the line plot only
Can you write Python code for it?


I am working on the file "Histogram_Manipulation_with_Python.ipynb".
The first task is:
stepfilled  ->  to create the step but also fill it (similar to bar but without the vertical lines)
Can you write Python code for it?


I am working on the file "Histograme.ipynb".
The first task is:
Fisierul Vizite.txt contine numarul de vizite ale site-urilor monitorizate de sati in luna martie.
Histograma ilustreaza ca o proportie mare din numarul de site-uri monitorizate sunt vizitate de un  numar redus de ori si doar cateva sunt vizitate foarte mult.
Can you write Python code for it?

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Histograme.ipynb".
The first task is:
Functia np.random.exponential(theta, size=None) returneaza valori de observatie asupra unei variabile aleatoare exponential distribuita, de parametru $\theta=theta$.
Functia care evalueaza  densitatea distributiei $Exp(\theta)$:
Can you write Python code for it?

def fexpo(theta, x):
    return  np.exp(-x/theta)/theta


I am working on the file "Histograme.ipynb".
The first task is:
Seed-ul  se seteaza in perioada de debugging a codului, pentru ca avandu-l setat in orice rulare se genereaza acelasi sir de numere.
Daca nu este setat explicit, atunci Numpy alege un seed aleator bazat pe momentul de timp al lansarii in executie a codului.
Pentru a ilustra ca  o functie genereaza valori de observatie asupra unei variabile aleatoare de distributie prescrisa, se asociaza sirului de date histograma si se traseaza peste histograma graficul densitatii
distributiei respective, pentru comparatie.
Sa generam de exemplu 2000 de valori de observatie asupra unei v.a. $U\sim Unif[0,1)$ si apoi sa desenem
                                                                            histograma datelor cu un numar prescris de bare.
Functia care genereaza histograma este:
matplotlib.pyplot.hist() pe care o apelam simplu, plt.hist.
Argumentele si cuvintele cheie uzuale ale functiei sunt:
- x este array-ul ce contine datele
- bins=10  numarul de bare; implicit 10
- range=None, range reprezinta limitele valorilor din x intre care generam histograma. Daca
nu este setat, atunci se genereaza histograma datelor cuprinse in intervalul [xmin, xmax].
- normed=False (implicit) sau True;
Daca  normed=True, atunci bara corespunzatoare fiecarui interval
de divizare a datelor are aria egala cu (vezi algoritmul prezentat in Cursul 10):
$\displaystyle\frac{nr.\: valori\: din\: interval}{nr\: total\: de\: valori}$
Daca normed=False, atunci bara are aria egala cu lungimea bazei ori numarul de valori ce apartin intervalului corespunzator barei.

color=None; acest cuvant cheie seteaza culoarea barelor in histograma.
Can you write Python code for it?

x=np.random.random(2000)
histo=plt.hist(x, bins=15, normed=True, color='g')
plt.plot([0,1], [1,1], 'r')# graficul densitatii uniforme pe [0,1)


I am working on the file "Hit_Run_Sampler/main.ipynb".
The first task is:
2. Exponential Distributions on the Unit Simplex
The exponential distribution on the $\textit{n}$-unit simplex $\Delta^n$ is defined as:
$$p((x_1,x_2,...,x_{n+1})) = \frac{\text{exp}(a_1 x_1 + a_2 x_2 + ... + a_{n+1}x_{n+1})}{Z}$$
$\text{where    }(x_1,x_2,...,x_{n+1}) \in \Delta^n;$
$Z = \sum_{i=1}^{n+1}{\frac{\text{exp}(a_i)}{\prod_{j\neq i}{(a_i - a_j)}}},$ and $a_1,a_2,...,a_{n+1}$ are constants
This distribution arise from one of my research projects. In the project, sampling from the distribution is required. I originally thought that the distribution seems pretty standard and there must be some standard method for sampling directly from it, like the method for sampling from the uniform distribution on the unit simplex. However, after some searching, I realize it is not the case. I still do not find existing methods for sampling directly from it.
To work around it, I turn to Markov Chain Monte Carlo methods.
2.1 Gibbs Sampler
After a simple manipulation, we can see that sampling from the above distribution is equivalent to sampling from the following distribution:
$$p((x_1,x_2,...,x_n)) = \text{exp}(a_{n+1}) \cdot \frac{\text{exp}((a_1 - a_{n+1}) x_1 + (a_2 - a_{n+1}) x_2 + ... + (a_{n} - a_{n+1})x_{n})}{Z}$$
where $0 \leq \sum_{i=1}^{n}{x_i} \leq 1$ and $x_i \geq 0$ for $i = 1,2,...,n.$
One method for sampling from this distribution is the Gibbs sampler, which samples each individual $x_i$ successively from its conditional distribution given all other $x_j, j \neq i$.
Can you write Python code for it?

def restricted_exp_sample(c = 1, x_low = 0, x_up = 1):
    """
    Sample from the exponential distribution $p(x) \propto \text{exp}(c*x)$ restricted in the interval [a,b]
    """
    u = np.random.uniform()
    return 1.0/c * np.log(np.exp(c*x_low) + u * (np.exp(c*x_up) - np.exp(c*x_low)))

def Gibbs_Sampler(a_list):
    n = len(a_list)
    a_p = np.array(a_list[0:-1]) - a_list[-1]
    x = np.zeros(n-1)
    K = np.random.choice(range(50,100))
    for k in range(K):
        idx = np.random.choice(range(n-1))
        x_low = 0
        x_up = 1 - np.sum([x[i] for i in range(n-1) if i != idx])
        x[idx] = restricted_exp_sample(a_p[idx], x_low, x_up)
    return list(x) + [1-np.sum(x)]

a_list = [1,3,6]
num_samples = 5000
samples = []
for i in range(num_samples):
    samples.append(Gibbs_Sampler(a_list))

fontsize = 20
figure, tax = ternary.figure(scale = 1)
tax.boundary(linewidth=2.0)
tax.gridlines(multiple = 0.2, color="blue")
tax.clear_matplotlib_ticks()
tax.ticks(locations = np.linspace(0,1,5))
tax.left_axis_label("$x_1$", fontsize = fontsize)
tax.right_axis_label("$x_3$", fontsize = fontsize)
tax.bottom_axis_label("$x_2$", fontsize = fontsize)
tax.scatter(samples, marker = ".", color = "blue")

x_1 = [s[0] for s in samples]
x_2 = [s[1] for s in samples]
plt.plot(x_1, x_2, '.')
plt.xlabel("$x_1$", fontsize = fontsize)
plt.ylabel("$x_2$", fontsize = fontsize)
plt.plot(np.linspace(0,1,20), 1-np.linspace(0,1,20), color = "r")


I am working on the file "Hofstadter-buterfly.ipynb".
The first task is:
An alternative is to send the figure to Plotly cloud:
Can you write Python code for it?

import plotly.plotly as py

import warnings
warnings.filterwarnings("ignore")

py.sign_in('empet', '')
py.iplot(fw, filename='Hofstadter1')


I am working on the file "Hofstadter-buterfly.ipynb".
The first task is:
At the first sight the effective computation of the spectrum for many Harper matrices seems to be cumbersome. But with Anaconda Python packages it is very fast, because Anaconda packaged MKL-powered binary versions of some of the most popular numerical/scientific Python libraries into MKL Optimizations that improve performance. (MKL stands for Intel™ Math Kernel Library, a set of vectorized math routines that accelerate math functions).
Let us test get_butterfly_points_even():
Can you write Python code for it?

%time phi1, E1, text1=get_butterfly_points_even(qmax=101)

len(E1)#points are plotted


I am working on the file "Hofstadter-buterfly.ipynb".
The first task is:
Define data for plotting both above Hofstadter butterflies on the same figure.
Can you write Python code for it?

data_global=[get_butterfly_trace(phi1, E1, text1, color='rgb(192,192,192)'),
             get_butterfly_trace(phi_m1, E_m1, text_m1, color='rgb(255,215,0)')]
fw_global=go.FigureWidget(data=data_global, layout=layout)
fw_global.layout.title='Hofstadter Butterfly'

fw_global


I am working on the file "Hofstadter-buterfly.ipynb".
The first task is:
Generating the butterfly data,  with the function get_butterfly_points_odd() and then via get_butterfly_points_even(), corresponding both to the same s, we get two indistinguishable plots.
The user can experiment plotting the Hofstadter butterfly associated to both data, succesively, or to their union.
Can you write Python code for it?

data=[get_butterfly_trace(phi1, E1, text1)]

axis_style=dict(showline=True,
                mirror=True,
                zeroline=False,
                showgrid=False,
                ticklen=4)

layout=dict(title='Hofstadter butterfly<br> K=2, s=1',
            font=dict(family='Balto'),
            width=600, height=675,
            autosize=False,
            showlegend=False,
            xaxis=dict(axis_style, **dict( title='Phi (magnetic flux)', dtick=0.25)),
            yaxis=dict(axis_style, **dict( title='E (Energy)')),
            hovermode='closest',
           plot_bgcolor='rgb(10,10,10)')

fw=go.FigureWidget(data=data, layout=layout)

fw # running this cell  the FigureWidget is plotted in the next one


I am working on the file "Hofstadter-buterfly.ipynb".
The first task is:
It's naturally to ask yourself  how behaves the spectrum of the associated Harper-type matrix, $H(p,q, s=-1)$.
Taking into account the interlacing property mentioned above we expect to get also a butterfly.
Let us plot it:
Can you write Python code for it?

phi_m1, E_m1, text_m1=get_butterfly_points_even(qmax=101, s=-1)
data1=[get_butterfly_trace(phi_m1, E_m1, text_m1, color='rgb(192,192,192)')]
fw1=go.FigureWidget(data=data1, layout=layout)
fw1.layout.title='Hofstadter butterfly<br> s = -1, K=2'

fw1

py.iplot(fw1, filename='Hofstadterm1')


I am working on the file "Hofstadter-buterfly.ipynb".
The first task is:
The two butterflies  look very similar. Even if we plot them in the same figure, we cannot distinguish the gold points from the silver ones,
because the distance between two consecutive eigenvalues, one in H(p,q, s=1) and another in H(p,q, s=-1) is very small.
For example:
Can you write Python code for it?


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
... to the novel collection assignment:
Can you write Python code for it?

# Plot
tsne_novel = TSNEVisualizer()
labels = ["c{}".format(c) for c in pipe.named_steps.kmeans.labels_]
tsne_novel.fit(X_bow, labels)
tsne_novel.poof()


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
Decided for 3 clusters, because of highest avg Silhoutte score compared to other cluster sizes.
Can you write Python code for it?

from yellowbrick.cluster import SilhouetteVisualizer

n_clusters = 3
model = KMeans(n_clusters=n_clusters, n_jobs=-1, random_state=23)

viz = SilhouetteVisualizer(model)

viz.fit(X_bow)
viz.poof()


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
Evaluation
Cluster density
Silhoutte coefficient: [-1,1], where 1 is most dense and negative vals correspond to ill seperation.
Can you write Python code for it?

from sklearn.metrics import silhouette_score

print("Avg Silhoutte score:", silhouette_score(X_bow, pred), "(novel collections)")


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
Input
Can you write Python code for it?

from sklearn.datasets import load_files

corpus = load_files("../data/")

doc_count = len(corpus.data)
print("Doc count:", doc_count)
assert doc_count is 56, "Wrong number of documents loaded, should be 56 (56 stories)"


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
Let's see how the the books are differently assigned to collections by Sir Arthur Conan Doyle (Original Collection), respectively by the clustering algo (Novel Collection).
Can you write Python code for it?

orig_assignment = [collections_map[c] for c in corpus.target]
novel_assignment = [novel_collections_map[p] for p in pred]

titles = [" ".join(f_name.split("/")[-1].split(".")[0].split("_"))
          for f_name in corpus.filenames]

# Final df, compares original with new assignment
df_documents = pd.DataFrame([orig_assignment, novel_assignment],
                            columns=titles, index=["Original Collection", "Novel Collection"]).T
df_documents.to_csv("collections.csv")
df_documents

df_documents["Novel Collection"].value_counts()


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
Nonetheless, the assignment isn't perfect. Cluster #1 looks good, but the many negative vals in cluster #0 & #1 suggest that there exist a cluster with more similar docs than in the actual assigned cluster. As a cluster size of 2 also leads to an inhomogen cluster and has a lower avg Silhoutte score, we go with the size of 3. Nevertheless, in general those findings suggest that the Sherlock Holmes stories should be represented in a single collection only.
Training
Can you write Python code for it?

from sklearn.pipeline import Pipeline

pipe = Pipeline([("bow", bow),
                 ("kmeans", model)])
pipe.fit(corpus.data)

pred = pipe.predict(corpus.data)


I am working on the file "HolmesClustering/holmes_clustering/notebook/2_Modeling.ipynb".
The first task is:
Vectorizer
Can you write Python code for it?

from helpers.tokenizer import TextWrangler
from sklearn.feature_extraction.text import CountVectorizer

bow = CountVectorizer(strip_accents="ascii", tokenizer=TextWrangler(kind="lemma"))
X_bow = bow.fit_transform(corpus.data)


I am working on the file "HolmesTopicModels/holmes_topic_models/notebook/2_Modeling.ipynb".
The first task is:
Evaluation
Can you write Python code for it?

print("LDA")
print("Log Likelihood:", lda_pipe.score(corpus.data))


I am working on the file "HolmesTopicModels/holmes_topic_models/notebook/2_Modeling.ipynb".
The first task is:
Models
Can you write Python code for it?

from sklearn.decomposition import LatentDirichletAllocation, TruncatedSVD, NMF

n_topics = 5

lda = LatentDirichletAllocation(n_components=n_topics,
                                learning_decay=0.5, learning_offset=1.,
                                random_state=23)
lsa = TruncatedSVD(n_components=n_topics, random_state=23)
nmf = NMF(n_components=n_topics, solver="mu", beta_loss="kullback-leibler", alpha=0.1, random_state=23)

lda_params = {"lda__learning_decay": [0.5, 0.7, 0.9],
              "lda__learning_offset": [1., 5., 10.]}


I am working on the file "HolmesTopicModels/holmes_topic_models/notebook/2_Modeling.ipynb".
The first task is:
Pipelines
Can you write Python code for it?

from sklearn.pipeline import Pipeline

lda_pipe = Pipeline([
    ("bow", bow_stem),
    ("lda", lda)
])

lsa_pipe = Pipeline([
    ("tfidf", tfidf_stem),
    ("lsa", lsa)
])

nmf_pipe = Pipeline([
    ("tfidf", tfidf_stem),
    ("nmf", nmf)
])


I am working on the file "HolmesTopicModels/holmes_topic_models/notebook/2_Modeling.ipynb".
The first task is:
Training
Can you write Python code for it?


I am working on the file "HolmesTopicModels/holmes_topic_models/notebook/2_Modeling.ipynb".
The first task is:
Vectorizer
Can you write Python code for it?

from helpers.tokenizer import TextWrangler
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

bow_stem = CountVectorizer(strip_accents="ascii", tokenizer=TextWrangler(kind="stem"))
X_bow_stem = bow_stem.fit_transform(corpus.data)

tfidf_stem = TfidfVectorizer(strip_accents="ascii", tokenizer=TextWrangler(kind="stem"))
X_tfidf_stem = tfidf_stem.fit_transform(corpus.data)


I am working on the file "HolmesTopicModels/holmes_topic_models/notebook/2_Modeling.ipynb".
The first task is:
Visual Inspection
Can you write Python code for it?

def df_topic_model(vectorizer, model, n_words=20):
    keywords = np.array(vectorizer.get_feature_names())
    topic_keywords = []
    for topic_weights in model.components_:
        top_keyword_locs = (-topic_weights).argsort()[:n_words]
        topic_keywords.append(keywords.take(top_keyword_locs))

    df_topic_keywords = pd.DataFrame(topic_keywords)
    df_topic_keywords.columns = ['Word '+str(i) for i in range(df_topic_keywords.shape[1])]
    df_topic_keywords.index = ['Topic '+str(i) for i in range(df_topic_keywords.shape[0])]

    return df_topic_keywords

print("LDA")
df_topic_model(vectorizer=bow_stem, model=lda_pipe.named_steps.lda, n_words=15)

print("LSA")
df_topic_model(vectorizer=tfidf_stem, model=lsa_pipe.named_steps.lsa, n_words=15)

print("NMF")
df_topic_model(vectorizer=tfidf_stem, model=nmf_pipe.named_steps.nmf, n_words=15)

import pyLDAvis
from pyLDAvis.sklearn import prepare
pyLDAvis.enable_notebook()

prepare(lda_pipe.named_steps.lda, X_bow_stem, bow_stem, mds="tsne")

prepare(nmf_pipe.named_steps.nmf, X_tfidf_stem, tfidf_stem, mds="tsne")


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
1. I want to make sure my Plate ID is a string. Can't lose the leading zeroes!
Can you write Python code for it?

plate_info = {'Plate ID': 'str'}
df = pd.read_csv("small-violations.csv", dtype=plate_info)

df

df.head()

df.head(10)

df.tail()


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
11. Make a chart of the top few.
Can you write Python code for it?

%matplotlib inline

out_of_staters_other.sort_values(ascending=False).plot(kind='bar', x='Violation code')

out_of_staters_manhattan.sort_values(ascending=False).plot(kind='bar', x='Violation code')


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
12. What time of day do people usually get their tickets? You can break the day up into several blocks - for example 12am-6am, 6am-12pm, 12pm-6pm, 6pm-12am.
13. What's the average ticket cost in NYC?
Can you write Python code for it?

average_tix_price = total_out_of_staters_violations / diff_violations_df['Violation code'].value_counts().sum()
average_tix_price


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
15. Make a graph of the amount of revenue collected per day.
Can you write Python code for it?

daily_revenue = total_out_of_staters_violations / new_violations_df['New Issue Date'].value_counts()
daily_revenue.sort_values(ascending=False).head(20).plot(kind='bar')


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
16. Manually construct a dataframe out of https://dmv.ny.gov/statistic/2015licinforce-web.pdf (only NYC boroughts - bronx, queens, manhattan, staten island, brooklyn), having columns for borough name, abbreviation, and number of licensed drivers.
Can you write Python code for it?

nyc_licenses = pd.read_excel("NYC.xlsx")
nyc_licenses


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
17. What's the parking-ticket-$-per-licensed-driver in each borough of NYC? Do this with pandas and the dataframe you just made, not with your head!
Can you write Python code for it?

diff_violations_df.columns

diff_violations_df['Violation County'].value_counts()

bronx_violations = diff_violations_df[diff_violations_df['Violation County'] == 'BX']
bronx_licenses = nyc_licenses['Total'][nyc_licenses['Abbreviation'] == 'BX']
bronx_tix = bronx_violations.groupby('Violation code')['All Other Areas'].sum()
driver_bronx_tix = bronx_licenses / bronx_tix.sum()
driver_bronx_tix

queens_violations = diff_violations_df[diff_violations_df['Violation County'] == 'Q']
queens_licenses = nyc_licenses['Total'][nyc_licenses['Abbreviation'] == 'Q']
queens_tix = queens_violations.groupby('Violation code')['All Other Areas'].sum()
driver_queens_tix = queens_licenses / queens_tix.sum()
driver_queens_tix

ny_violations = diff_violations_df[diff_violations_df['Violation County'] == 'NY']
ny_licenses = nyc_licenses['Total'][nyc_licenses['Abbreviation'] == 'NY']
ny_tix = ny_violations.groupby('Violation code')['All Other Areas'].sum()
driver_ny_tix = ny_licenses / ny_tix.sum()
driver_ny_tix

brooklyn_violations = diff_violations_df[diff_violations_df['Violation County'] == 'R']
brooklyn_licenses = nyc_licenses['Total'][nyc_licenses['Abbreviation'] == 'R']
brooklyn_tix = brooklyn_violations.groupby('Violation code')['All Other Areas'].sum()
driver_brooklyn_tix = brooklyn_licenses / brooklyn_tix.sum()
driver_brooklyn_tix

staten_is_violations = diff_violations_df[diff_violations_df['Violation County'] == 'K']
staten_is_licenses = nyc_licenses['Total'][nyc_licenses['Abbreviation'] == 'K']
staten_is_tix = violations_kings.groupby('Violation code')['All Other Areas'].sum()
driver_staten_is_tix = staten_is_licenses / staten_is_tix.sum()
driver_staten_is_tix


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
4. "Date first observed" is a pretty weird column, but it seems like it has a date hiding inside. Using a function with .apply, transform the string (e.g. "20140324") into a Python date. Make the 0's show up as NaN.
Can you write Python code for it?

other_df.columns

other_df['Date First Observed'].dtypes

other_df['Date First Observed'].tail()

import dateutil

other_df['Date First Observed']

other_df['Violation Time'].head()

other_df['Violation Time'].tail()

def int_to_date(integer):
    if not pd.isnull(integer):
        date = str(int(integer))
        parsed_date = dateutil.parser.parse(date)
        return parsed_date.strftime("%Y-%-m-%d")

other_df['Date First Observed'].apply(int_to_date)


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
5. "Violation time" is... not a time. Make it a time.
Can you write Python code for it?

def violation_time_to_time(time):
    try:
        hour = time[0:2]
        minutes = time[2:4]
        am_pm= time[4]
        regular_time= hour + ":" + minutes + " " + am_pm + 'm'
        violation_time_fixed = dateutil.parser.parse(regular_time)
        return violation_time_fixed.strftime("%H:%M%p")
    except:
        return None

other_df['Violation Time'].apply(violation_time_to_time)


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
6.  There sure are a lot of colors of cars, too bad so many of them are the same. Make "BLK" and "BLACK", "WT" and "WHITE", and any other combinations that you notice.
Can you write Python code for it?

other_df['Vehicle Color'].value_counts()

def color_rename(color):
    if (color == 'BLACK') or (color == 'BLK') or (color == 'BK'):
        return 'BLACK'
    elif (color == 'WHITE') or (color == 'WHT') or (color == 'WH') or (color == 'W'):
        return 'WHITE'

other_df['Vehicle Color'].apply(color_rename)


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
7. Join the data with the Parking Violations Code dataset from the NYC Open Data site.
Can you write Python code for it?

parking_violations_df = pd.read_csv("DOF_Parking_Violation_Codes.csv", encoding="mac_roman", error_bad_lines=False)
parking_violations_df.head()

parking_violations_df['CODE'].describe()

other_df['Violation Code'].describe()

def convert_to_str(n):
    return str(n)

parking_violations_df['Code'] = parking_violations_df['CODE'].apply(convert_to_str)

other_df['Violation code'] = other_df['Violation Code'].apply(convert_to_str)

parking_violations_df.head()

updated_parking_violations_df = parking_violations_df.rename(columns={'Manhattan¬† 96th St. & below': 'Manhattan 96th & below', 'All Other Areas': 'All other areas'})
updated_parking_violations_df.head()

other_df.head()

diff_violations_df = pd.merge(other_df, updated_parking_violations_df, left_on='Violation code', right_on='Code')
diff_violations_df.head()


I am working on the file "Homework 11 Soma.ipynb".
The first task is:
9. What's the most lucrative kind of parking violation? The most frequent?
Can you write Python code for it?

manhattan_violations.sort_values(ascending=False)

violations_not_man.sort_values(ascending=False)

new_violations_df['Violation code'].value_counts()


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
$$corr(x,y) = \frac{cov(x,y)}{(n-1)s_x s_y} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_i (x_i - \bar{x})^2 \sum_i (y_i - \bar{y})^2}}$$
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
A useful summarization that gives a quick snapshot of multiple statistics for a Series or DataFrame is describe:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Above, Pandas recognized NA and an empty field as missing data.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Additionally, the order of the set of indices in a hierarchical MultiIndex can be changed by swapping them pairwise:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Alternatively, you can break ties via one of several methods, such as by the order in which they occur in the dataset:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
But note, we cannot use the attribute indexing method to add a new column:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
By default, dropna drops entire rows in which one or more values are missing.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Clearly, sum is more meaningful for some columns than others. For methods like mean for which application to string variables is not just meaningless, but impossible, these columns are automatically exculded:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Critically, the labels are used to align data when used in operations with other Series objects:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Exercise 1
Try out these commands to see what they return:

data.head()
data.tail(3)
data.shape
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Exercise 2
From the data table above, create an index to return all rows for which the phylum name ends in "bacteria" and the value is greater than 1000.
Can you write Python code for it?

# get all of the values from the data where the phylum name ends with bacteria
data[[phylum.endswith('bacteria') for phylum in data.phylum]]
# take all of the rows where the value which are above 1000
data[data.value>1000]

data
data.phylum


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Exercise 4
Calculate on base percentage for each player, and return the ordered series of estimates.
$$OBP = \frac{H + BB + HBP}{AB + BB + HBP + SF}$$
Can you write Python code for it?

def onbasepercent(x):
    onbases = x['h'] + x['bb'] + x['hbp']
    basetotal = x['ab'] + x['bb'] + x['hbp'] + x['sf']+1e-6

    return onbases/basetotal

baseball.apply(onbasepercent, axis=1).round(3)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
For a more concise (and readable) syntax, we can use the new query method to perform selection on a DataFrame. Instead of having to type the fully-specified column, we can simply pass a string that describes what to select. The query above is then simply:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Hierarchical indexing
In the baseball example, I was forced to combine 3 fields to obtain a unique index that was not simply an integer value. A more elegant way to have done this would be to create a hierarchical index from the three fields.
Can you write Python code for it?

baseball_h = baseball.set_index(['year', 'team', 'player'])
baseball_h.head(10)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Hierarchical indices can be created on either or both axes. Here is a trivial example:
Can you write Python code for it?

frame = pd.DataFrame(np.arange(12).reshape(( 4, 3)),
                  index =[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
                  columns =[['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']])

frame


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
If an index is not specified, a default sequence of integers is assigned as the index. A NumPy array comprises the values of the Series, while the index is a pandas Index object.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
If we have a DataFrame with a hierarchical index (or indices), summary statistics can be applied with respect to any of the index levels:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
If we only want to import a small number of rows from, say, a very large data file we can use nrows:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
If we plan on modifying an extracted Series, its a good idea to make a copy.
Can you write Python code for it?

vals = data.value.copy()
vals[5] = 1000
vals


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
If we wish to access columns, we can do so either by dict-like indexing or by attribute:
Can you write Python code for it?

data['patient']

data.patient

type(data.value)

data[['value']]


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
If you want to get fancy, both the row and column indices themselves can be given names:
Can you write Python code for it?

frame.index.names = ['key1', 'key2']
frame.columns.names = ['state', 'color']
frame


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Importing data
A key, but often under-appreciated, step in data analysis is importing the data that we wish to analyze. Though it is easy to load basic data structures into Python using built-in tools or those provided by packages like NumPy, it is non-trivial to import structured data well, and to easily convert this input into a robust data structure:
genes = np.loadtxt("genes.csv", delimiter=",", dtype=[('gene', '|S10'), ('value', '<f4')])

Pandas provides a convenient set of functions for importing tabular data in a number of formats directly into a DataFrame object. These functions include a slew of options to perform type inference, indexing, parsing, iterating and cleaning automatically as data are imported.
Let's start with some more bacteria data, stored in csv format.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
In a DataFrame we can slice along either or both axes:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
In addition to using loc to select rows and columns by label, pandas also allows indexing by position using the iloc attribute.
So, we can query rows and columns by absolute position, rather than by name:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Indexing and Selection
Indexing works analogously to indexing in NumPy arrays, except we can use the labels in the Index object to extract values in addition to arrays of integers.
Can you write Python code for it?

# Sample Series object
hits = baseball_newind.h
hits

# Numpy-style indexing
hits[:3]

# Indexing by label
hits[['womacto01CHN2006','schilcu01BOS2006']]


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Its important to note that the Series returned when a DataFrame is indexted is merely a view on the DataFrame, and not a copy of the data itself. So you must be cautious when manipulating this data:
Can you write Python code for it?

vals = data.value
vals

vals[5] = 0
vals


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Keep in mind that reindex does not work if we pass a non-unique index series.
We can remove rows or columns via the drop method:
Can you write Python code for it?

baseball.shape

baseball.drop([89525, 89526])

baseball.drop(['ibb','hbp'], axis=1)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Lets use apply to calculate a meaningful baseball statistics, slugging percentage:
$$SLG = \frac{1B + (2 \times 2B) + (3 \times 3B) + (4 \times HR)}{AB}$$
And just for fun, we will format the resulting estimate.
Can you write Python code for it?

def slugging(x):
    bases = x['h']-x['X2b']-x['X3b']-x['hr'] + 2*x['X2b'] + 3*x['X3b'] + 4*x['hr']
    ab = x['ab']+1e-6

    return bases/ab

baseball.apply(slugging, axis=1).round(3)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Missing data
The occurence of missing data is so prevalent that it pays to use tools like Pandas, which seamlessly integrates missing data handling so that it can be dealt with easily, and in the manner required by the analysis at hand.
Missing data are represented in Series and DataFrame objects by the NaN floating point value. However, None is also treated as missing, since it is commonly used as such in other contexts (e.g. NumPy).
Can you write Python code for it?

foo = pd.Series([np.nan, -3, None, 'foobar'])
foo

foo.isnull()


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Missing values can also be interpolated, using any one of a variety of methods:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Missing values may be dropped or indexed out:
Can you write Python code for it?

bacteria2

bacteria2.dropna()

bacteria2.isnull()

bacteria2[bacteria2.notnull()]


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Most real-world data is incomplete, with values missing due to incomplete observation, data entry or transcription error, or other reasons. Pandas will automatically recognize and parse common missing data indicators, including NA and NULL.
Can you write Python code for it?

!cat Data/microbiome_missing.csv

pd.read_csv("Data/microbiome_missing.csv").head(20)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Notice that fillna by default returns a new object with the desired filling behavior, rather than changing the Series or  DataFrame in place (in general, we like to do this, by the way!).
We can alter values in-place using inplace=True.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Notice that the id index is not sequential. Say we wanted to populate the table with every id value. We could specify and index that is a sequence from the first to the last id numbers in the database, and Pandas would fill in the missing data with NaN values:
Can you write Python code for it?

id_range = range(baseball.index.values.min(), baseball.index.values.max())
baseball.reindex(id_range).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Notice that the indexing operation preserved the association between the values and the corresponding indices.
We can still use positional indexing if we wish.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Notice that we specified the id column as the index, since it appears to be a unique identifier. We could try to create a unique index ourselves by combining player and year:
Can you write Python code for it?

player_id = baseball.player + baseball.year.astype(str)
baseball_newind = baseball.copy()
baseball_newind.index = player_id
baseball_newind.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Notice the DataFrame is sorted by column name. We can change the order by indexing them in the order we desire:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Notice this is different than with Series, where dict-like indexing retrieved a particular element (row).
If we want access to a row in a DataFrame, we index its loc attribute.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Now, let's add them together, in hopes of getting 2-year home run totals:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
NumPy's math functions and other operations can be applied to Series without losing the data structure.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Operations
DataFrame and Series objects allow for several operations to take place either on a single object, or between two or more objects.
For example, we can perform arithmetic on the elements of two objects, such as combining baseball statistics across years. First, let's (artificially) construct two Series, consisting of home runs hit in years 2006 and 2007, respectively:
Can you write Python code for it?

hr2006 = baseball.loc[baseball.year==2006, 'hr']
hr2006.index = baseball.player[baseball.year==2006]

hr2007 = baseball.loc[baseball.year==2007, 'hr']
hr2007.index = baseball.player[baseball.year==2007]

hr2007


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Pandas Data Structures
Series
A Series is a single vector of data (like a NumPy array) with an index that labels each element in the vector.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Pandas uses a custom data structure to represent the indices of Series and DataFrames.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Pandas' data alignment places NaN values for labels that do not overlap in the two Series. In fact, there are only 6 players that occur in both years.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Ranking does not re-arrange data, but instead returns an index that ranks each value relative to others in the Series.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Rather than omitting missing data from an analysis, in some cases it may be suitable to fill the missing value in, either with a default value (such as zero) or a value that is either imputed or carried forward/backward from similar data points. We can do this programmatically in Pandas with the fillna argument.
Can you write Python code for it?

bacteria2.fillna(0)

data.fillna({'year': 2013, 'treatment':2})


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
References
Python for Data Analysis Wes McKinney
Can you write Python code for it?

#do isin to get only New cases of confirmed and New deaths registered for each country
liberia_frame_column = liberia_frame[liberia_frame['Variable'].isin(['New case/s (confirmed)', 'Newly reported deaths'])]
liberia_frame_column.head()

#select only the relevant columns for us - date, variable, national
liberia_frame_new= liberia_frame_column[['Date', 'Variable', 'National']]
liberia_frame_new.head()

#do isin to get only New cases of confirmed and New deaths registered for each country
sl_frame_column = sl_frame[sl_frame['variable'].isin(['new_confirmed', 'death_confirmed'])]
sl_frame_column.head()

#select only the relevant columns for us - date, variable, national
sl_frame_new= sl_frame_column[['date', 'variable', 'National']]
sl_frame_new.head()

#for each country
#made the dates the index and drop the date column
guinea_frame_new.index = guinea_frame_new.Date
guinea_drop = guinea_frame_new.drop('Date', axis=1)
guinea_drop.head()

liberia_frame_new.index = liberia_frame_new.Date
liberia_drop = liberia_frame_new.drop('Date', axis=1)
liberia_drop.head()

sl_frame_new.index = sl_frame_new.date
sl_drop = sl_frame_new.drop('date', axis=1)
sl_drop.head()

dataframe = pd.concat([guinea_drop, liberia_drop, sl_drop], axis=0)
dataframe.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
So, indices need not be unique. Our choice is not unique because some players change teams within years.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Sometimes we may not want to ignore missing values, and allow the nan to propagate.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Sorting and Ranking
Pandas objects include methods for re-ordering data.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
The complement to to_pickle is the read_pickle function, which restores the pickle to a DataFrame or Series:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
The dtypes attribute reveals the data type for each column in our DataFrame.

int64 is numeric integer values
object strings (letters and numbers)
float64 floating-point values
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
The most important consequence of a non-unique index is that indexing by label will return multiple values for some labels:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
The sep argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats:
sep='\s+'

For a more useful index, we can specify the first two columns, which together provide a unique index to the data.
Can you write Python code for it?

mb = pd.read_csv("Data/microbiome.csv", index_col=['Patient','Taxon'])
mb.head(5)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
There are several other data formats that can be imported into Python and converted into DataFrames, with the help of buitl-in or third-party libraries. These include JSON, XML, HDF5, relational and non-relational databases, and various web APIs. These are beyond the scope of this tutorial, but are covered in Python for Data Analysis.
Pandas Fundamentals
This section introduces the new user to the key functionality of Pandas that is required to use the software effectively.
For some variety, we will leave our digestive tract bacteria behind and employ some baseball data.
Can you write Python code for it?

baseball = pd.read_csv("Data/baseball.csv", index_col='id')
baseball.head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
This can be overridden by passing the how='all' argument, which only drops a row when every field is a missing value.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
This is called a hierarchical index, which we will revisit later in the section.
If we have sections of data that we do not wish to import (for example, known bad data), we can populate the skiprows argument:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
This is so that Index objects can be shared between data structures without fear that they will be changed.
Can you write Python code for it?

bacteria2.index = bacteria.index

bacteria2


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Ties are assigned the mean value of the tied ranks, which may result in decimal values.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Try running corr on the entire baseball DataFrame to see what is returned:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Try sorting the columns instead of the rows, in ascending order:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
Unfortunately, there will sometimes be inconsistency with the conventions for missing data. In this example, there is a question mark "?" and a large negative number where there should have been a positive integer. We can specify additional symbols with the na_values argument:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can also apply functions to each column or row of a DataFrame
Can you write Python code for it?

stats.apply(np.median)

def range_calc(x):
    return x.max() - x.min()

stat_range = lambda x: x.max() - x.min()
stats.apply(stat_range)


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can also calculate summary statistics across multiple columns, for example, correlation and covariance.
$$cov(x,y) = \sum_i (x_i - \bar{x})(y_i - \bar{y})$$
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can also filter according to the values in the Series:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can also slice with data labels, since they have an intrinsic order within the Index:
Can you write Python code for it?

hits['womacto01CHN2006':'gonzalu01ARI2006']

hits['womacto01CHN2006':'gonzalu01ARI2006'] = 5
hits


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can also use sort_values to sort a Series by value, rather than by label.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can assign meaningful labels to the index, if they are available:
Can you write Python code for it?

bacteria = pd.Series([632, 1638, 569, 115],
    index=['Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes'])

bacteria


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can extract the underlying data as a simple ndarray by accessing the values attribute:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We can use the drop method to remove rows or columns, which by default drops rows. We can be explicit by using the axis argument:
Can you write Python code for it?

data_nomonth = data.drop('month', axis=1)
data_nomonth


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
We will learn more about indexing below.
We can create a truly unique index by combining player, team and year:
Can you write Python code for it?

player_unique = baseball.player + baseball.team + baseball.year.astype(str)
baseball_newind = baseball.copy()
baseball_newind.index = player_unique
baseball_newind.head()

baseball_newind.index.is_unique


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
While we do want the operation to honor the data labels in this way, we probably do not want the missing values to be filled with NaN. We can use the add method to calculate player home run totals by using the fill_value argument to insert a zero for home runs where labels do not overlap:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
With a hierachical index, we can select subsets of the data based on a partial index:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
data.head shows all of the data up to a certain number, if it is indicated in the parantheses
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
describe can detect non-numeric data and sometimes yield useful information about it.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
read_csv is just a convenience function for read_table, since csv is such a common format:
Can you write Python code for it?

mb = pd.read_table("Data/microbiome.csv", sep=',')


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
this gives you the number of rows of data (8) and columns (3)
An alternative way of initializing a DataFrame is with a list of dicts:
Can you write Python code for it?

data = pd.DataFrame([{'patient': 1, 'phylum': 'Firmicutes', 'value': 632},
                    {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},
                    {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},
                    {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},
                    {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},
                    {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},
                    {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},
                    {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}])

data


I am working on the file "Homework/01 - Pandas and Data Wrangling/.ipynb_checkpoints/Intro+to+Pandas (Katia)-checkpoint.ipynb".
The first task is:
this shows the last 3 rows of data of the data row
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
A logical way to specify these numerically is to change them to integer values, perhaps using "Placebo" as a baseline value. If we create a dict with the original values as keys and the replacements as values, we can pass it to the map method to implement the changes.
Can you write Python code for it?

treatment_map = {'Placebo': 0, '5000U': 1, '10000U': 2}

cdystonia['treatment'] = cdystonia.treat.map(treatment_map)
cdystonia.treatment


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
A related function qcut uses empirical quantiles to divide the data. If, for example, we want the quartiles -- (0-25%], (25-50%], (50-70%], (75-100%] -- we can just specify 4 intervals, which will be equally-spaced by default:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
A slightly cleaner way of doing this is to set the patient-level information as an index before unstacking:
Can you write Python code for it?

(cdystonia.set_index(['patient','site','id','treat','age','sex','week'])['twstrs']
     .unstack('week').head())


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Alternately, if we simply want to replace particular values in a Series or DataFrame, we can use the replace method.
An example where replacement is useful is dealing with zeros in certain transformations. For example, if we try to take the log of a set of values:
Can you write Python code for it?

vals = pd.Series([float(i)**10 for i in range(10)])
vals

np.log(vals)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Alternately, we can transform the data, using a function of our choice with the transform method:
Can you write Python code for it?

normalize = lambda x: (x - x.mean())/x.std()

cdystonia_grouped.transform(normalize).head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Alternatively, one can specify custom quantiles to act as cut points:
Can you write Python code for it?

quantiles = pd.qcut(segments.seg_length, [0, 0.01, 0.05, 0.95, 0.99, 1])
quantiles[:30]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Alternatively, you can pass keys to the concatenation by supplying the DataFrames (or Series) as a dict, resulting in a "wide" format table.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Apply
We can generalize the split-apply-combine methodology by using apply function. This allows us to invoke any function we wish on a grouped dataset and recombine them into a DataFrame.
The function below takes a DataFrame and a column name, sorts by the column, and takes the n largest values of that column. We can use this with apply to return the largest values from every group in a DataFrame in a single call.
Can you write Python code for it?

def top(df, column, n=5):
    return df.sort_values(by=column, ascending=False)[:n]


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
As a convenience, Pandas has a to_datetime method that will parse and convert an entire Series of formatted strings into datetime objects.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
By default the Categorical type represents an unordered categorical.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
By default, groupby groups by row, but we can specify the axis argument to change this. For example, we can group our columns by dtype this way:
Can you write Python code for it?

grouped_by_type = cdystonia.groupby(cdystonia.dtypes, axis=1)
{g:grouped_by_type.get_group(g) for g in grouped_by_type.groups}


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Categorical Data
Pandas provides a convenient dtype for reprsenting categorical (factor) data, called category.
For example, the treat column in the cervical dystonia dataset represents three treatment levels in a clinical trial, and is imported by default as an object type, since it is a mixture of string characters.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Columns of the datetime type have an accessor to easily extract properties of the data type. This will return a Series, with the same row index as the DataFrame. For example:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Data transformation
There are a slew of additional operations for DataFrames that we would collectively refer to as "transformations" which include tasks such as removing duplicate values, replacing values, and grouping values.
Dealing with duplicates
We can easily identify and remove duplicate values from DataFrame objects. For example, say we want to removed ships from our vessels dataset that have the same name:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Exercise 2
Load the dataset in titanic.xls. It contains data on all the passengers that travelled on the Titanic.
Can you write Python code for it?

from IPython.core.display import HTML
HTML(filename='Data/titanic.html')

#import titanic data file
titanic = pd.read_excel("Data/titanic.xls", index_col=None)
titanic.head()

# turn "sex" attribute into numerical attribute
# 0 = male ; 1= female

sex_map = {'male': 0, 'female': 1}
titanic['sex'] = titanic.sex.map(sex_map)
titanic.head()

# clean duplicate values
titanic_2 = titanic.drop_duplicates(['name'])

# convert attributes to categorical data
pd.Categorical(titanic_2.pclass)
pd.Categorical(titanic_2.survived)
pd.Categorical(titanic_2.sex)
pd.Categorical(titanic_2.age)
pd.Categorical(titanic_2.sibsp)
pd.Categorical(titanic_2.parch)
pd.Categorical(titanic_2.ticket)
pd.Categorical(titanic_2.fare)
pd.Categorical(titanic_2.cabin)
pd.Categorical(titanic_2.embarked)
pd.Categorical(titanic_2.boat)
pd.Categorical(titanic_2.body)

titanic_2

# describe passenger class
pclasses = titanic_2.pclass.value_counts()
class1 = (pclasses[1]/1307)*100
class2 = (pclasses[2]/1307)*100
class3 = (pclasses[3]/1307)*100
d = {'1st Class' : class1, '2nd Class' : class2, '3rdclass' : class3}
pd.Series(d)
#24% of passengers travelled 1st class, 21% travelled in 2nd class and 54% travelled in 3rd class

# plot classes 1 = 1st 2 = 2nd and 3 = 3rd
pclasses.plot.pie()

# describe passenger survival
survivals = titanic_2.survived.value_counts()
survived = (survivals[1]/1307)*100
survived
# 38.25% of passengers survived

# plot survivals 0 = death & 1 = survival
survivals.plot.pie()

# describe passenger sex
sex = titanic_2.sex.value_counts()
sex
male_ratio = (sex[1]/1307)*100
male_ratio
# results show that 35% of passengers are male and 65% are female

# plot gender distribution 0 = male & 1 = female
sex.plot.pie()

# calculate proportions of port of embarcation S = Southtampton & C = Cherbourg & Q = Queenstown
port = titanic_2.embarked.value_counts()
S = (port[0]/1307)*100
C = (port[1]/1307)*100
Q = (port[2]/1307)*100
d = {'S' : S, 'C' : C, 'Q' : Q}
pd.Series(d)
# 20.6% of passengers boarded in C, 9.4% boarded in Q and 69.7%...


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
For a simple cross-tabulation of group frequencies, the crosstab function (not a method) aggregates counts of data according to factors in rows and columns. The factors may be hierarchical if desired.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
For random sampling, DataFrame and Series objects have a sample method that can be used to draw samples, with or without replacement:
Can you write Python code for it?

vessels.sample(n=10)

vessels.sample(n=10, replace=True)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Having a custom data type for dates and times is convenient because we can perform operations on them easily. For example, we may want to calculate the difference between two times:
Can you write Python code for it?

my_age = now - datetime(1970, 1, 1)
my_age

print(type(my_age))
my_age.days/365


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
Hence, we get a considerable speedup simply by using the appropriate dtype for our data.
Discretization
Pandas' cut function can be used to group continuous or countable data in to bins. Discretization is generally a very bad idea for statistical analysis, so use this function responsibly!
Lets say we want to bin the ages of the cervical dystonia patients into a smaller number of groups:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
However, since there are multiple taxonomic units with the same class, our index is no longer unique:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
However, the grouping is only an intermediate step; for example, we may want to iterate over each of the patient groups:
Can you write Python code for it?

for patient, group in cdystonia_grouped:
    print('patient', patient)
    print('group', group)


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If we are only interested in taxa that are included in both DataFrames, we can specify a join=inner argument.
Can you write Python code for it?

pd.concat([mb1, mb2], axis=1, join='inner').head()


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If we concatenate along axis=0 (the default), we will obtain another data frame with the the rows concatenated:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If we omit the values argument, we get a DataFrame with hierarchical columns, just as when we applied unstack to the hierarchically-indexed table:
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If we want to transform this data so that repeated measurements are in columns, we can unstack the twstrs measurements according to obs.
Can you write Python code for it?

twstrs_wide = cdystonia2['twstrs'].unstack('obs')
twstrs_wide.head()

cdystonia_wide = (cdystonia[['patient','site','id','treat','age','sex']]
                  .drop_duplicates()
                  .merge(twstrs_wide, right_index=True, left_on='patient', how='inner')
                  .head())
cdystonia_wide


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If we wanted to use the second table to fill values absent from the first table, we could use combine_first.
Can you write Python code for it?


I am working on the file "Homework/01 - Pandas and Data Wrangling/Data Wrangling with Pandas.ipynb".
The first task is:
If we wish, we can easily aggregate according to multiple keys:
Can you write Python code for it?