I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
Exercise 4
a) Generate random function samples using the optmized model and plot alongside the input data
Can you write Python code for it?

# get desired input points where model needs to be evaluated
Xp = np.linspace(0.0,1.0,100)[:,None]

# obtain posterior mean and variances
mu, C = m.predict(Xp, full_cov=True)

# generate 10 random paths of the distribution
nPaths = 10
paths = np.random.multivariate_normal(mu[:,0], C, nPaths)

# plot the dataset and the generated paths
plt.figure()
plt.plot(X,Y,'kx',mew=5)

for i in range(nPaths):
    plt.plot(Xp[:],paths[i,:])


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
Exercise 5
a) Build a Gaussian process model for the olympic data set using a combination of an RBF and a bias covariance function. Fit the covariance function parameters and the noise to the data. Plot the fit and error bars from 1870 to 2030. Do you think the predictions are reasonable? If not why not?
Can you write Python code for it?

# create the covariance function
kern = GPy.kern.RBF(1) + GPy.kern.Bias(1)

# create the GP Regression model
model = GPy.models.GPRegression(X, Y, kern)

# optimize the model
model.optimize()
model.plot()


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
It is also possible to multiply two kernel functions:
Can you write Python code for it?

kern = kern1*kern2
print kern
kern.plot(plot_limits=[-6,8])


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
It is also possible to plot the kernel as a function of one of its inputs (whilst fixing the other) with k.plot(). Use "?" to view the properties of the plot function for kern instance.
Can you write Python code for it?


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
Setting Covariance Function Parameters
The value of the covariance function parameters can be accessed and modified using k.* where the * can refer to the parameter name as it appears in print k. We'll now set the lengthscale of the covariance to different values, and then plot the resulting covariance using the k.plot() method.
Can you write Python code for it?

# by default, all the parameters are set to 1. for the RBF kernel
k = GPy.kern.RBF(d)

# we experiment with different length scale parameter values here
theta = np.asarray([0.2,0.5,1.,2.,4.])

# create an instance of a figure
fig = plt.figure()
ax = plt.subplot(111)

# iterate over the lengthscales
for t in theta:
    k.lengthscale=t
    # plot in the same figure with a different color
    k.plot(ax=ax, color=np.random.rand(3,), plot_limits=[-10.0,10.0])
plt.legend(theta)


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
The actual predictions of the model for a set of points Xstar can be computed using m.predict(Xstar):
Can you write Python code for it?

# obtain 5 test points
Xstar = np.linspace(0.01,0.99,5)[:,None]

# predict the output for the test points
Ystar, Vstar = m.predict(Xstar)

# print results
print Ystar


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
We can optimize the hyperparameters of the model using the m.optimize() method.
Can you write Python code for it?


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
b) Can you tell the covariance structures that have been used for generating the
sample paths shown in the figure below?

3 Gaussian Process Regression Model
We will combine the Gaussian process prior with data to form a GP regression model with GPy. We will generate data from the function $f ( x ) = − \cos(\pi x ) + \sin(4\pi x )$ over $[0, 1]$, adding some noise to give $y(x) = f(x) + \epsilon$, with the noise being Gaussian distributed, $\epsilon \sim \mathcal{N}(0, 0.01)$.
Can you write Python code for it?

# input data points
X = np.linspace(0.05,0.95,10)[:,None]
# generate observations through function f
Y = -np.cos(np.pi*X) + np.sin(4*np.pi*X) + np.random.normal(loc=0.0, scale=0.1, size=(10,1))

# plot the generated data
plt.figure()
plt.plot(X,Y,'kx',mew=1.5)


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
b) Modify the kernel used for building the model to investigate its influence on the model
Can you write Python code for it?

# create instance of kernel and GP Regression model
k = GPy.kern.Matern32(input_dim=1, variance=1., lengthscale=1.)
m = GPy.models.GPRegression(X,Y,k)

# optimize the model
m.constrain_positive()
m.optimize()

# view model parameters
print m
m.plot()

# get desired input points where model needs to be evaluated
Xp = np.linspace(0.0,1.0,100)[:,None]

# obtain posterior mean and variances
mu, C = m.predict(Xp, full_cov=True)

# generate 10 random paths of the distribution
nPaths = 10
paths = np.random.multivariate_normal(mu[:,0], C, nPaths)

# plot the dataset and the generated paths
plt.figure()
plt.plot(X,Y,'kx',mew=5)

for i in range(nPaths):
    plt.plot(Xp[:],paths[i,:])


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
c) Instead of rbf, try constructing and plotting the following  covariance functions: exponential, Matern32, Matern52, Brownian, linear, bias, rbfcos, periodic_Matern32, etc. Use the tab key to look for the functions GPy.kern submodule or search in the GPy documentation.
Can you write Python code for it?

# look for the kernel documentation
GPy.kern.Matern32?

# input dim
d = 1

# create the Matern32 kernel
k = GPy.kern.Matern32(d)

# view the kernel
print k
k.plot()


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
c) Modify your model by including two covariance functions. Intitialize a covariance function with an exponentiated quadratic part, a Matern 3/2 part and a bias covariance. Set the initial lengthscale of the exponentiated quadratic to 80 years, set the initial length scale of the Matern 3/2 to 10 years. Optimize the new model and plot the fit again. How does it compare with the previous model?
Can you write Python code for it?

# create the covariance function and set the desired parameter values
kern3 = GPy.kern.RBF(1, lengthscale=80.0) + GPy.kern.Matern32(1, lengthscale=10.0) + GPy.kern.Bias(1)
model3 = GPy.models.GPRegression(X, Y, kern3)

# optimize and plot the model
model3.optimize()
model3.plot()

# compare the log likelihoods
print model.log_likelihood(), model3.log_likelihood()


I am working on the file "Experiments/Notebook1.ipynb".
The first task is:
e) Now model the data with a product of an exponentiated quadratic covariance function and a linear covariance function. Fit the covariance function parameters. Why are the variance parameters of the linear part so small? How could this be fixed?
Can you write Python code for it?

# create the covariance function and set the desired parameter values
kern5 = GPy.kern.RBF(1, lengthscale=5.0, variance=5.0)*GPy.kern.Linear(1)
model5 = GPy.models.GPRegression(X, Y, kern5)

# optimize and plot the model
model5.optimize()
model5.plot()

# get the model parameters
print model5

# compare the log likelihoods
print model.log_likelihood(), model5.log_likelihood()


I am working on the file "Experiments/Notebook2.ipynb".
The first task is:
2 Principal Component Analysis
Principal component analysis (PCA) finds a rotation of the observed outputs, such that the rotated principal component (PC) space maximizes the variance of the data observed, sorted from most to least important (most to least variable in the corresponding PC).
Can you write Python code for it?

# it is important to normalize the data
Yn = Y-Y.mean()


I am working on the file "Experiments/Notebook2.ipynb".
The first task is:
3 Gaussian Process Latent Variable Model
Gaussian Process Latent Variable Model (GP-LVM) embeds PCA into Gaussian process framework. In the Gaussian Process, any covariance function can be used. One example is the automatic relevance determination (ARD) linear kernel:
Can you write Python code for it?

# set latent dimensions to use
input_dim = 4

# define covariance function to be used
kernel = GPy.kern.Linear(input_dim, ARD=True)

# setup the GPLVM model
m = GPy.models.GPLVM(Yn, input_dim=input_dim, kernel=kernel)

# optimize for 1000 iterations
m.optimize(messages=1, max_iters=1000)

# visualize the ARD kernel parameters
m.kern.plot_ARD()

# visualize the two most significant dimensions
plot_model(m.X, m.linear.variances.argsort()[-2:], labels.flatten())
plt.legend()


I am working on the file "Experiments/Notebook2.ipynb".
The first task is:
For the linear covariance, these latent points can be optimized with an eigenvalue problem, but generally, for non-linear covariance functions, we have to use gradient based optimization.
Exercise 1
a) How do your linear solutions differ between PCA and GPLVM with a linear kernel? Look at the plots and also try and consider how the linear ARD parameters compare to the eigenvalues of the principal components.
b) The next step is to use a non-linear mapping between inputs $\mathbf{X}$ and ouputs $\mathbf{Y}$ by selecting the RBF (GPy.kern.rbf) covariance function.
Can you write Python code for it?

# set latent dimensions to use
input_dim = 4

# define covariance function to be used
kernel = GPy.kern.RBF(input_dim, ARD=True)

# setup the GPLVM model
m = GPy.models.GPLVM(Yn, input_dim=input_dim, kernel=kernel)

# optimize for 1000 iterations
m.optimize(messages=1, max_iters=1000)

# visualize the ARD kernel parameters
m.kern.plot_ARD()

# visualize the two most significant dimensions
plot_model(m.X, m.rbf.lengthscale.argsort()[:2], labels.flatten())
plt.legend()


I am working on the file "Experiments/Notebook2.ipynb".
The first task is:
The following code is utility functions for plotting and to prepare the bigger models for later usage. If you are interested, you can have a look, but this is not essential.
Can you write Python code for it?

colors = ["#3FCC94", "#DD4F23", "#C6D63B", "#D44271",
          "#E4A42C", "#4F9139", "#6DDA4C", "#85831F",
          "#B36A29", "#CF4E4A"]
def plot_model(X, which_dims, labels):
    fig = plt.figure()
    ax = fig.add_subplot(111)

    X = X[:,which_dims]

    ulabs = []
    for lab in labels:
        if not lab in ulabs:
            ulabs.append(lab)
            pass
        pass
    for i, lab in enumerate(ulabs):
        ax.scatter(*X[labels==lab].T,marker='o',color=colors[i],label=lab)
        pass
    pass


I am working on the file "Experiments/Notebook2.ipynb".
The first task is:
The resulting plota will show the lower dimensional representation of the digits in 2 dimensions,
Can you write Python code for it?

# plot first 20 eigenvalue fractions
p.plot_fracs(20)

# plot the latent space by PCA
p.plot_2d(Y,labels=labels.flatten(), colors=colors)
plt.legend()


I am working on the file "Experiments/Notebook2.ipynb".
The first task is:
You can try to plot some of the digits using plt.matshow (the digit images have size 16x16).
Can you write Python code for it?


I am working on the file "Experiments/Printing Colour.ipynb".
The first task is:
Identifying quotation marks
https://regex101.com/r/cB0kB8/1
https://stackoverflow.com/questions/39713487/extracting-quotations-citations-with-nltk-not-regex
Can you write Python code for it?

import regex as re

print(re.findall(r'"(.*?)"', comments[0]))


I am working on the file "Experiments/Printing Colour.ipynb".
The first task is:
Insults package
Can you write Python code for it?

from insults import Insults

comment = "You are a disgusting maggot of a person."
Insults.load_model()
print(Insults.rate_comment(comment))

comments = ["You called me a \"dickhead\", so I'll say you're a cunt.", "These shitakes taste like shit."]
print(Insults.foul_language(comments, context=False))


I am working on the file "Explainer_notebook.ipynb".
The first task is:
... and the Tags with most songs.
Can you write Python code for it?

query = """
MATCH (t:Tag)<-[r:TAGGED]-()
RETURN
    DISTINCT t.name as tag,
    COUNT(r) as count
ORDER BY count DESC
LIMIT 10
"""
r = %cypher {query}
print r.get_dataframe()


I am working on the file "Explainer_notebook.ipynb".
The first task is:
As we can see for this particular community (and for others, see webpage) we obtain groups of tags that refer to similar kinds of music. Although certain tags within a group definitely share songs with other genres, the high modularity within this group makes them end up in one community.
Natural Language Processing
Characteristic words for tags
In order to find out what words are characteristic for the lyrics of songs from certain tags we will use the TF-IDF score to rank words. While the TF (term frequency) is individual to each document, the IDF (inverse document frequency) is a global score for each unique word. While the TF score is simply the number of times a term occurs in a document, we can compute the IDF score by
$$
IDF_i = log\left(\frac{N}{n_i}\right)
$$
with

$N$: number of songs with lyrics
$n_i$: number of songs that contain word $i$

The TF-IDF score defined by
$$
TFIDF = TF\cdot IDF
$$
now provides a balanced measure of how often a term occurs in a document and how unique the term is with respect to all documents.
We will let the database do the IDF calculation by
Can you write Python code for it?

query = """
MATCH (s:Song)-[:CONTAINS]->(:Word)
WITH COUNT(DISTINCT s) as songCount
MATCH (w:Word)<-[:CONTAINS]-(s:Song)
RETURN DISTINCT coalesce(w.unstemmed, w.word) as word, log10(songCount/toFloat(COUNT(DISTINCT s))) as IDF
"""
r = %cypher {query}
idf_df = r.get_dataframe()
idf_df.set_index("word",inplace=True)

print "Words with lowest IDF:"
print idf_df.sort_values("IDF",ascending = True).head(5)


I am working on the file "Explainer_notebook.ipynb".
The first task is:
As we can see, out of the roughly 100000 nodes, there are a lot which do not have any similar songs. If we are only considering the rest almost 95% of those belong to the giant connected component.
Song-Tag graph
Another graph is the bipartite graph containing songs and tags. As there are about one million song-tag relationships in our database, we avoided having to construct the complete graph in-memory and extracted just the data we need for the individual tasks.
Degree distributions
In order to compare the two graphs, we will first look at the degree distributions of songs and tags. We let the database do the work of computing the degrees to avoid constructing the whole graph in-memory. We obtain the degree of similarity for songs (both incoming and outgoing similarity relationships to other songs) as well as the in-degree for tags, i.e. how many songs are tagged by a certain tag.
Can you write Python code for it?

query = """
MATCH (s:Song)-[r:SIMILAR]-()
WHERE s.year IS NOT NULL
RETURN distinct s.id, count(r)
"""

r = %cypher {query}
sim_degrees = r.get_dataframe().values[:,1]

query = """
MATCH (s:Song)-[r:TAGGED]->(t:Tag)
WHERE s.year IS NOT NULL
RETURN distinct t.name, count(r)
"""
r = %cypher {query}
tag_degrees = r.get_dataframe().values[:,1]

_,ax = plt.subplots(1,2,figsize=(10,4),sharey=True)
for i,degrees in enumerate([sim_degrees, tag_degrees]):
    edges = np.subtract(range(np.min(degrees),np.max(degrees) + 1),0.5)
    hist, _ = np.histogram(degrees,bins=edges)
    plotedges = np.add(edges[:-1],edges[1:])/2

    ax[i].loglog(plotedges,hist,".k")
    ax[i].set_xlabel("degree")
    ax[i].spines['right'].set_visible(False)
    ax[i].spines['top'].set_visible(False)
    ax[i].xaxis.set_ticks_position('bottom')
    ax[i].yaxis.set_ticks_position('left')

ax[0].set_ylabel("count")
ax[0].set_title("Song similarity degree distribution")
ax[1].set_title("Tag degree distribution")
ax[1].set_xlim(1,3000)
plt.savefig('tag_degree_distribution.png', bbox_inches='tight')
plt.show()


I am working on the file "Explainer_notebook.ipynb".
The first task is:
Finally we add year information for all songs where it is available
Can you write Python code for it?

with io.open("./songdata/tracks_per_year.txt","r",encoding="UTF-8") as year_file:
    year_data = {}
    for line in year_file:
        parts = line.split("<SEP>")
        year_data[parts[1]] = int(parts[0])

limit = 1000
skip = 0
while True:
    # get a batch of nodes that have no year
    query = """
        MATCH (s:Song)
        WHERE (s)-[:CONTAINS]->()
        AND s.year IS NULL
        RETURN s.id AS track_id
        SKIP {skip}
        LIMIT {limit}
    """
    result = %cypher {query}
    ids = [rec.track_id for rec in result.get_dataframe().itertuples()]

    if not ids:
        break
    else:
        years = [{"id" : track, "year": year_data[track]} for track in ids if track in year_data]
        skip += limit-len(years)
        query = """
            UNWIND {years} as year
            MATCH (s:Song {id:year.id})
            SET s.year = year.year
        """
        %cypher {query}


I am working on the file "Explainer_notebook.ipynb".
The first task is:
Here we can see, very common words get a low IDF score, which says they carry little information if we observe them. Indeed, these words tell us nothing about the particular topic of a text.
We will now compute the TF-IDF scores for the 100 most frequent tags using results from all songs tagged by a one of those tags.
Can you write Python code for it?

%%capture
from nltk.corpus import stopwords
query = """
MATCH (t:Tag)<-[r:TAGGED]-()
RETURN distinct t.name as tag,COUNT(r) as c
ORDER BY c DESC
LIMIT 100
"""
r = %cypher {query}
df = r.get_dataframe()
tags = df["tag"].values.tolist()

tags.append("Gangsta Rap")
tags.append("death metal")
tags.append("political")
tags.append("country")
tags = list(set(tags))

exclude = set(stopwords.words('english'))
tf_idf = {}
tf = {}
for current_tag in tags:
    query = """
        MATCH (t:Tag {name:{current_tag}})<-[:TAGGED]-(:Song)-[r:CONTAINS]->(w:Word)
        RETURN DISTINCT coalesce(w.unstemmed, w.word) as word, COUNT(r) as count
        ORDER BY count DESC
    """
    r = %cypher {query}
    df = r.get_dataframe()
    tf[current_tag] = {word[0] : word[1] for word in df.values if word[0] not in exclude}
    tf_idf[current_tag] = {word[0] : idf_df.get_value(word[0],"IDF")*word[1] for word in df.values if word[0] not in exclude}

import string
select_tags = ["Gangsta Rap","spanish","black metal","rock n roll","romantic","Dreamy","christian"]
for tag in select_tags:
    keywords = [v[0] for v in sorted(tf_idf[tag].items(),key=lambda v: -v[1])[:10]]
    print "%-20s : %s" % (tag, string.join(keywords,sep=", "))


I am working on the file "Explainer_notebook.ipynb".
The first task is:
Here we obtained some basic count statistics. We can see that only around a third of all nodes has full information, i.e. at least on similar song and tag and release year information.
Release year distribution
Can you write Python code for it?

import matplotlib.pyplot as plt
# Aggregate by year to find distribution
r = %cypher MATCH (s:Song) WHERE s.year IS NOT NULL RETURN DISTINCT s.year as year, COUNT(s) as count ORDER BY year
df = r.get_dataframe()
years = df["year"].values.tolist()
counts = df["count"].values.tolist()
years.append(years[-1]+1)
counts.append(0)

years_sorted = sorted({year:counts[i] for i,year in enumerate(years)}.items(),key=lambda v: -v[1])
print "Year range:         %d-%d" % (np.min(years),np.max(years)-1)
print "Most frequent year: %d (%d songs)" % years_sorted[0]

_,ax = plt.subplots(1,1,figsize=(10,4))
plt.plot(years,counts,"k",linewidth=3)
plt.xlabel("Year")
plt.ylabel("Number of Songs")
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('none')

plt.savefig('year_distribution.png', bbox_inches='tight')
plt.show()


I am working on the file "Explainer_notebook.ipynb".
The first task is:
Here we see the top 10 words by TF-IDF score for some selected tags. It is obvious that the used words hint at possible tags. This could be used to find the most probable tags given the lyrics of a song. Naturally, it was a lot less clear for popular tags like "rock" or "pop" which did not show any significant keywords.
Finally, we can arrange the tags in WordClouds to get a nice visualization of certain tags, where the result was very expressive. We use the tags "death metal", "Gangsta Rap", "political", and "country"
Can you write Python code for it?

from scipy import misc
import matplotlib.pyplot as plt
%matplotlib inline
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator

def draw_wordcloud(tag, mask, ax):
    img_mask = misc.imread(mask,mode="RGB")
    keywords = " ".join([ word for k,v in tf_idf[tag].iteritems() for word in ["%s " % k] * int(10*v) if v > 0.5])
    wordcloud = WordCloud(stopwords=STOPWORDS,background_color='rgb(255,255,255)', random_state=42,
                          mask=img_mask).generate(keywords).recolor(color_func=ImageColorGenerator(img_mask))
    ax.imshow(wordcloud)
    ax.axis("off")

_,ax = plt.subplots(2,2,figsize=(11,11))
draw_wordcloud("death metal","./masks/death_colored.png",ax[0,0])
draw_wordcloud("Gangsta Rap","./masks/gangsta_colored.png",ax[0,1])
draw_wordcloud("political","./masks/political_colored.png",ax[1,0])
draw_wordcloud("country","./masks/rock_colored.png",ax[1,1])


I am working on the file "Explainer_notebook.ipynb".
The first task is:
Now we iterate through the lyrics file and for each song obtain meta data from the LastFM data and add the song to the database including its similar songs (i.e. they are created as "empty" nodes if they don't exist yet). This process can be stopped and run again to pupulate the database in several runs (it takes a long time). As the database queries are the main bottleneck, we obtain all track_ids of songs that already have full data from the database first and only run queries for new songs.
Can you write Python code for it?

# find all songs in the database that have lyric information
from IPython.display import clear_output
print "Getting database songs..."
limit = 10000
skip = 0
added = 0
all_saved_ids = set()
while True:
    query = """
        MATCH (s:Song)
        WHERE EXISTS((s)-[:CONTAINS]->(:Word))
        RETURN s.id AS track_id
        SKIP {skip}
        LIMIT {limit}
    """
    result = %cypher {query}
    saved_ids = {row.track_id for row in result.get_dataframe().itertuples()}
    skip += len(saved_ids)
    if len(saved_ids) == 0:
        break
    all_saved_ids.update(saved_ids)

try:
    # iterate through songs and insert new ones
    with io.open("./songdata/lyrics.txt","r",encoding="UTF-8") as lyrics_file:
        for line in lyrics_file:
            # jump over comments
            if line.startswith("#") or line.startswith("%"):
                continue

            # jump over empty lines
            parts = line.strip().split(",")
            if len(parts) == 1:
                continue

            # get id and word counts
            track_id = parts[0]
            if track_id in all_saved_ids:
                continue

            wordcounts = []
            for part in parts[2:]:
                entry = part.split(":")
                wordcounts.append({"index":int(entry[0]),"count":int(entry[1])})

            # load song metadata
            path = "%s/%s/%s/%s.json" % (track_id[2],track_id[3],track_id[4],track_id)
            try:
                with io.open("./songdata/meta/" + path,"r") as meta_file:
                    data = json.loads(meta_file.read())
            except:
                continue

            # prepare properties
            title = data["title"]
            artist = data["artist"]
            tags = [ {"name":tag[0],"score":tag[1]} for tag in data["tags"] if int(tag[1]) > 0 ]
            similars = [ {"id":similar[0],"score":similar[1]}
                        for similar in data["similars"]
                        if similar[1] > 0.05 ]
            print...


I am working on the file "Explainer_notebook.ipynb".
The first task is:
Sentiment analysis
Sentiment analysis is another approach of characterizing the data. During the research of [Dodds], the team obtained a set of 10222 (labMT 1.0) word and happiness score pairs indicating how much happiness the individual word conveys. We will use this list to define an average sentiment score for a set of words as
$$
S = \frac{1}{N} \sum_{i=1}^N S_i
$$
where:
* $N$: the number of words for which sentiment data was available
* $S_i$: the sentiment value for word i
Having obtained the word counts for the top 100 tags we can use these to calculate an average sentiment score for each tag. What we yield is a ranking of how happy certain tags are on average.
Can you write Python code for it?

import io
import json
labmt_df = pd.read_csv("./songdata/labmt.txt",sep="\t", usecols=[0,2])
sentiment_map = {v[0]:v[1] for v in labmt_df.values}

def calculate_sentiment(wordcounts):
    count = 0
    sentiment = 0
    for k,v in wordcounts.iteritems():
        if k in sentiment_map:
            sentiment += v*sentiment_map[k]
            count += v
    return sentiment/count

sentiments = {}
for tag in tags:
    sentiments[tag] = calculate_sentiment(tf[tag])

with io.open("tag_sentiment.txt","w",encoding="UTF-8") as sentiment_file:
    sentiment_file.write(unicode(json.dumps(sentiments,ensure_ascii=False)))

exclude = {"spanish", "german", "latin"} # exclude foreign language tags

tags_ordered = [t[0] for t in sorted(sentiments.items(),key=lambda v: -v[1]) if t[0] not in exclude]

plt.rcParams["figure.figsize"] = (30,4)
plt.hlines(1,0,len(tags_ordered))  # Draw a horizontal line
plt.eventplot(range(len(tags_ordered)), orientation='horizontal', colors='k',linelengths=0.1)
plt.ylim(-1,2)
plt.axis("off")
ax = plt.gca()

for i,tag in enumerate(tags_ordered):
    ax.text(i,1.1,tag,ha='center',va='bottom',rotation=90,fontsize=6)

plt.savefig("sentiment_ordered.png",transparent=True,dpi=300,bbox_inches='tight')
plt.show()

print "%-20s\t%s" % ("TOP 5","BOTTOM 5")
for i in range(5):
    print "%-20s\t%s" % (tags_ordered[i],tags_ordered[-i-1])


I am working on the file "Explainer_notebook.ipynb".
The first task is:
The results are not particularly surprising. Obviously rock and pop are the most popular tags indicating two very popular genres. While there are a lot of tags not carrying genre information ("favourites", "best song ever", "female vocalists") in the dataset, the most common tags contain a higher proportion of genre tags.
Graph statistics
Now that we have the graph representation with full information in our database, we can extract partial information to construct a graph which is processable in-memory. We will focus our analysis on the giant connected component which was almost the full graph in all cases.
Similarity graph
The first graph we will use is the similarity graph containing Songs as nodes and similarity relationships as edges.
Can you write Python code for it?

# get all similarity relationships
query ="""
MATCH (s:Song)-[r:SIMILAR]->(s2:Song)
RETURN s.id AS from, s2.id AS to, r.score AS score
"""
r = %cypher {query}
df = r.get_dataframe()

# build similarity graph
G = nx.DiGraph()
for row in df.iterrows():
    r = row[1]
    G.add_edge(r["from"],r["to"],{"score":r["score"]})

# extract largest component
H = max(nx.weakly_connected_component_subgraphs(G), key=len)
print "Largest weakly connected component contains %d of %d nodes." % (H.number_of_nodes(),G.number_of_nodes())


I am working on the file "Explainer_notebook.ipynb".
The first task is:
We notice that the amount of similar songs, that a song can have seems to have an upper bound. If we look at tags on the other hand, their degree seems to follow a clear power law. We can thus expect the maximum tag degree to grow unbounded as the network size increases. It is a scale-free network. The song similarity on the other hand will always be restricted to a small number of songs independent of the network size.
The growth of the tag network could be modeled by a Barabási-Albert model [Barbarasi, 5.2]. The applied mechanism is called preferential attachment and causes that new nodes are more likely to connect to high degree nodes than to other nodes.
Community detection
This section aims at finding groups of nodes in the graph that build communities. In order to do this, we will use the louvain module to obtain a good partition of the network. The louvain algorithm [Louvain] and other community detection algorithms aim at finding a partition of the network (i.e. grouping of nodes) that maximizes the sum of the modularity of the individual communities (as defined in [Barabasi, 9.4])
$$
M = \sum_c \frac{L_c}{L} - \left(\frac{k_c}{2L}\right)^2
$$
where:
* $L$: Total number of links
* $L_c$: Number of links in community $c$
* $k_c$: Total degree of all nodes within the community $c$
The result will be a partition where nodes are mostly connected to nodes within their own community and less to other communities.
Detecting tag communities
Now, we would like to find tags that are similar to each other by using the amount of songs they share as a similarity measure. Subsequently, we will run the louvain community detection algorithm on the obtained tag-similarity graph to find groups of tags that hopefully define a cluster of similar genres. As a preparation, we first get all the tagging relationships from the database and prepare a Tag: [song_ids] dictonary to store all songs, that are tagged by a specific tag.
Can you write Python code for it?

query ="""
MATCH (s:Song)-[r:TAGGED]->(t:Tag)
RETURN s.id AS song, t.name as tag
"""
r = %cypher {query}
df = r.get_dataframe()

tags = {}
for row in df.iterrows():
    r = row[1]
    if r["tag"] not in tags:
        tags[r["tag"]] = {r["song"]}
    else:
        tags[r["tag"]].add(r["song"])


I am working on the file "Exploit Payload Sizes.ipynb".
The first task is:
The humble histogram finishes out our exploration today - note that it's on a log scale. This is significantly less useful than the above charts, since payload size is a cumulative number (i.e. smaller payloads still work in exploits with more than enough space for them), but this view is interesting in that it shows us where there are large clusters of exploits accepting a certain payload size.
But, what about platforms? Are our results being influenced by the category of the exploit? (Windows and Linux vs. Android and iOS, etc.)
Can you write Python code for it?

sploitsByCategory = {}
for folder, subs, files in os.walk(rootdir):
    for filename in files:
        with open(os.path.join(folder, filename), 'r') as sploit:
            #print("parsing " + filename + "...")
            text = sploit.read()
            # remove all whitespace
            text = ''.join(text.split())
            space = re.search("\'Space\'=>(\d+)\,", text)
            # Note that if no payload size limit is specified, we simply ignore that module
            if space:
                # get the first folder in the exploits directory by
                # 1. Stripping off the rootdir using [len(rootdir):]
                # 2. Splitting on the folder separator (not platform-independent)
                # 3. Taking the first one (the zeroth one is '' since there's always a leading /)
                sploitType = folder[len(rootdir):].split('/')[1]
                try:
                    sploitsByCategory[sploitType].append(int(space.group(1)))
                except KeyError:
                    sploitsByCategory[sploitType] = []
                    sploitsByCategory[sploitType].append(int(space.group(1)))

# Yeah, that's right, a nested list comprehension. To print.
[str(sploits) + ": " + ",".join([str(num) for num in sploitsByCategory[sploits]]) for sploits in sploitsByCategory]

sortedSploits = {}
platforms = ['linux', 'windows', 'unix', 'multi']
for platform in platforms:
    sortedSploits[platform] = np.sort(sploitsByCategory[platform])

colors = ['b', 'g', 'r', 'c', 'm', 'y', 'k']
plt.figure(figsize = (10,5))
for i, platform in enumerate(platforms):
    plt.plot(sortedSploits[platform], np.linspace(1,0,len(sortedSploits[platform])), linewidth=3, color=colors[i])

plt.xlim((0,10000))

plt.xlabel("Payload size (bytes)", size=16)
plt.ylabel("Fraction of exploits which\n accept that payload size", size=16)

plt.legend([platform for platform in platforms])

plt.show()


I am working on the file "Exploit Payload Sizes.ipynb".
The first task is:
We'll start with displaying payload size against the fraction of exploits which will work (or not work) for that size. It looks like any payload over 2048 bytes will only work with about 20% of exploits - a little less, in fact! If any of your stagers are just barely above 1024 bytes, it's well worth the effort to trim those last few bytes. Almost a quarter of exploits available in Metasploit have a payload size cutoff at 1024 bytes.
This chart can also be read as a probability: if I want to send a 1000 byte payload, and I pick an exploit at random (or, I find a vulnerable host at random), I have about a 60% chance that the exploit I end up with will be able to accomodate that payload. If my payload is 500 bytes, that probability becomes more than 90%.
Can you write Python code for it?

plt.figure(figsize = (10,5))
plt.hist(sorted_sizes, 500)

plt.xlim((0,35000))
plt.yscale('log')

plt.xlabel("Payload Size (bytes)", size=16)
plt.ylabel("Number of Exploits, Log Scale", size=16)
plt.show()


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
Caramba, ¡mucho más alto que antes! Intentamos pintarlo todo:
Can you write Python code for it?

op = OrbitPlotter(num_points=10000)

op.plot(earth)
op.plot(interm)
op.plot(perih)
plt.gca().set_autoscale_on(False)
#op.plot(planet9)
op.plot(State.from_vectors(Sun, perih.r, v_i))


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
El frenazo que tenemos que pegar es considerable. Viajemos ahora al perihelio y efectuemos la transferencia.
Can you write Python code for it?

interm = earth.apply_maneuver(hoh)
perih = interm.propagate(interm.period / 2)
norm(perih.r)

norm(perih.v)


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
En este gráfico se aprecian dos cosas:

Como hemos dicho antes, el Planeta Nueve pasa la mitad de su período en la parte más alejada del Sol (y por tanto, de nosotros), el afelio (sector gris claro). En cambio, la otra mitad del tiempo se la pasa acelerando hacia el perihelio. Es evidente entonces que lo más probable es que el planeta esté más lejos que cerca.
Y hablando de lejos: está muy lejos. Extremadamente lejos. Por supuesto está cerca comparado con Alfa Centauri, pero precisamente lo novedoso de este caso es que nadie esperaba encontrar un planeta con un tamaño tan considerable a una distancia tan grande.

Otra forma de comprobar la enormidad de estas distancias es tratando de averiguar cuánto nos costaría llegar hasta allí. ¡Sigue leyendo!
¿Cómo llegamos hasta allí?

**Importante**: Estos resultados son provisionales y están pendientes de revisión por problemas con el algoritmo, que no converge bien. De hecho se observa que he tenido que especificar una tolerancia relativa desmesurada. Después del tiempo dedicado he preferido publicar el código, que al menos muestra cómo utilizar la biblioteca, a pesar de que haya hecho aflorar sus limitaciones. [Ya he abierto una incidencia en poliastro](https://github.com/poliastro/poliastro/issues/112) para ver dónde está el problema, aunque posiblemente se deba a la fuerte inestabilidad numérica que supone acertar una diana que está a 168 000 000 000 km.

Ahora es donde empieza de verdad la ciencia ficción. Vamos a estudiar algunas trayectorias posibles que podríamos seguir para llegar a este hipotético Planeta Nueve, aunque lamentablemente obtengamos tiempos de cientos de años o velocidades inalcanzables con nuestra tecnología :)
Primera opción: Viaje directo
Supongamos que lanzamos la nave la semana que viene (no hay tiempo que perder) y que nos planteásemos llegar en un tiempo optimista: 200 años. Supongamos también que en el momento del lanzamiento tenemos al Planeta Nueve en su afelio, que de hecho es el caso peor. El...
Can you write Python code for it?

from poliastro import iod
from poliastro.util import norm

date_launch = time.Time('2016-02-01 12:00', scale='utc').tdb
time_of_flight = 200 * u.year
date_arrival = date_launch + time_of_flight

r_0, v_earth = ephem.planet_ephem(ephem.EARTH, date_launch)

r_f = planet9.propagate(time_of_flight).r

v_0, v_f = iod.lambert(Sun.k, r_0, r_f, time_of_flight, rtol=5)


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
La diferencia de radios se debe a que no partíamos de una órbita circular. Nos hemos saltado el paso de circularizar la órbita para simplificar.
Por último, rezamos para que el algoritmo del problema de Lambert converja y nos lleve hasta el Planeta Nueve:
Can you write Python code for it?

v_i, _ = iod.lambert(Sun.k, perih.r.to(u.km), planet9.r.to(u.km), 100 * u.year, rtol=12)  # De nuevo, tolerancia demasiado grande
norm(v_i)


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
Para el resto tendremos que jugar un poco con matplotlib y las funciones de plotting que proporciona poliastro.
Can you write Python code for it?

# Creamos la figura

fig, ax = plt.subplots(figsize=(8, 8))

op = OrbitPlotter(ax)
planet9_point, planet9_orbit = op.plot(planet9)

planet9_point.set_color("#6600ff")
planet9_orbit.set_color("#6600ff")

# Enmascaramos los sectores circulares con la órbita

mask = PathPatch(planet9_orbit.get_path(), fc='none', lw=0)

ax.add_patch(mask)
ax.add_patch(fifty_far)
ax.add_patch(fifty_close)

fifty_far.set_clip_path(mask)
fifty_close.set_clip_path(mask)

# Zoom en el sistema Solar

ax_zoom = zoomed_inset_axes(ax, 8, loc=3, axes_kwargs={'axisbg': '#fafafa'})

# Repetimos algunos plots

op_zoom = OrbitPlotter(ax_zoom)
op_zoom.set_frame(*planet9.pqw())
earth_point, earth_orbit = op_zoom.plot(earth)
nepune_point, _ = op_zoom.plot(neptune)

earth_orbit.set_linestyle("solid")  # ¡Para que se vea algo!

# Propiedades de la sección aumentada

ax_zoom.set_xlim(-7e9, 5e9)
ax_zoom.set_ylim(-4e9, 5e9)
ax_zoom.set_xticks([])
ax_zoom.set_yticks([])
ax_zoom.set_xlabel("")
ax_zoom.set_ylabel("")
ax_zoom.grid(False)
ax_zoom.set_title("8x zoom")
mark_inset(ax, ax_zoom, loc1=1, loc2=4, fc="none", ec='0.3')

# Leyenda de la gráfica

leg = ax.legend(
    [planet9_point, earth_point, nepune_point, fifty_close, fifty_far],
    ["Planeta 9", "Tierra", "Neptuno", "Perihelio", "Afelio"],
    numpoints=1
)
leg.get_frame().set_facecolor('#fafafa')


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
Para tener algo con lo que comparar vamos a pintar también las órbitas de la Tierra y Neptuno. Para ello poliastro utilizará unos ficheros llamados SPK que contienen información precisa sobre las órbitas de los planetas del sistema solar.
Can you write Python code for it?

# Recuperamos la órbita de la Tierra para comparar

r_earth, v_earth = ephem.planet_ephem(ephem.EARTH, epoch)
earth = State.from_vectors(Sun, r_earth.to(u.km), v_earth.to(u.km / u.s), epoch)

# Y ya que nos ponemos, órbita de Neptuno

r_nep, v_nep = ephem.planet_ephem(ephem.PLUTO, epoch)
neptune = State.from_vectors(Sun, r_nep.to(u.km), v_nep.to(u.km / u.s), epoch)


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
Prácticamente en línea recta, ¡como si fuese esto una autopista! Lamentablemente 200 años es demasiado tiempo, y si intentamos reducirlo los requisitos de velocidad solo empeorarían. Veamos otras opción un poco más loca.
Efecto Oberth con el Sol
Al leer esta posibilidad no he podido evitar acordarme de la entrañable viñeta de XKCD:

El efecto Oberth es el hecho de que las maniobras impulsivas son más eficientes cuanto más rápido va el vehículo. Gracias a esto se podría diseñar una trayectoria que pasase muy cerca del Sol y que efectuase un gran impulso en el perihelio. La secuencia de la viñeta es demasiado complicada así que vamos a dividir este recorrido en dos partes:

Una transferencia de Hohmann a 38 millones de kilómetros del Sol, y
un impulso para alcanzar la órbita del Planeta Nueve en 100 años.

Veremos qué números obtenemos ahora:
Can you write Python code for it?

from poliastro.maneuver import Maneuver

hoh = Maneuver.hohmann(earth, 38e6 * u.km)
hoh.get_total_cost()


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
Vamos a crear un objeto State para representar Planeta Nueve, añadiendo a los parámetros estimados del artículo un valor de la anomalía verdadera de $180^{\circ}$, es decir: en el punto más alejado (caso peor). El período de la órbita será:
Can you write Python code for it?

a = 700 * u.AU
ecc=0.6 * u.one
inc=30 * u.deg
raan=100 * u.deg
argp=150 * u.deg
nu=180 * u.deg  # ¡Solo para probar!

planet9 = State.from_classical(Sun, a, ecc, inc, raan, argp, nu,  # Solo para probar
                               epoch)

period = planet9.period.to(u.year)
period


I am working on the file "Explorando el Planeta Nueve con Python usando poliastro.ipynb".
The first task is:
Veamos la magnitud de la velocidad de partida:
Can you write Python code for it?


I am working on the file "Exploration.ipynb".
The first task is:
Word score
To score the words we'll use the z-score of their tf-idf score where the train corpus are the transcripts and the term frequencyies come from the script
Can you write Python code for it?

corpus =[]
for key,val in transcripts.items():
    corpus+=val.splitlines()
tfidf = TfidfVectorizer(tokenizer=Tokenizer(),stop_words='english',ngram_range=(1,4),smooth_idf=True)
tf_model = tfidf.fit(corpus)
script = tf_model.transform([d["script.txt"]])
tf_feature_names = tf_model.get_feature_names()

def display_word_ranks(vector, feature_names, no_top_words):

        ranking_dict = {feature_names[i]: -np.log(vector[i]) #dictionary comprehension, key is word val is score
                   for i in vector.argsort()  # iterate over indices sorted by the value
                   if vector[i] >0 # and only take values that are non zero e.g. appear in the document
                   }

        return ranking_dict
z = script.toarray()

z[0]

ranking_dict =(display_word_ranks(z[0],tf_feature_names,40))

S = pd.Series(ranking_dict)
S = S.sort_values(ascending=False)
S[S!=S.value_counts().values[0]][:20].plot.bar(title="Our top n words scored")

T =S.sample(30).index
Y = S[T]
Y.sort_values(ascending=False).plot.bar(figsize=(15,5),title="Our top n words scored")

S.rank(ascending=False,method='first')[:30]

S[T].rank(ascending=False,method='first').sort_values()


I am working on the file "Exploration_of_data_in_MillionMusicSubset.ipynb".
The first task is:
Artist_hotttnesss vs artist familiarity
Can you write Python code for it?

fig, ax = plt.subplots(nrows=1, ncols=2, sharex=True, sharey=True,
                       figsize=(15,8))

ax[0].scatter(songs.artist_familiarity, songs.artist_hotttnesss, marker='.')
ax[1].hexbin(songs.artist_familiarity, songs.artist_hotttnesss, cmap='viridis', gridsize=51, mincnt=1.0)


I am working on the file "Exploration_of_data_in_MillionMusicSubset.ipynb".
The first task is:
Exploring the tables
Can you write Python code for it?

songs = pd.read_sql_query('SELECT * FROM songs WHERE year!=0',con_meta)

songs.head(5)


I am working on the file "Exploration_of_data_in_MillionMusicSubset.ipynb".
The first task is:
First we need to find out the table names in each of our files:
Can you write Python code for it?

# subset_artist_similarity.db
res = con_simi.execute("SELECT name FROM sqlite_master WHERE type='table';")
for name in res:
    print(name[0])

# subset_artist_term
res = con_term.execute("SELECT name FROM sqlite_master WHERE type='table';")
for name in res:
    print(name[0])

# subset_track_metadata
res = con_meta.execute("SELECT name FROM sqlite_master WHERE type='table';")
for name in res:
    print(name[0])


I am working on the file "Exploration_of_data_in_MillionMusicSubset.ipynb".
The first task is:
Reading HDF5 files
Can you write Python code for it?

with pd.HDFStore(path_to_data+'AdditionalFiles/subset_msd_summary_file.h5') as store:
    print(store)
    analysis_summary = store.select('analysis/songs')
    metadata_summary = store.select('metadata/songs')
    musicbrainz_summary = store.select('musicbrainz/songs')

analysis_summary.head()

metadata_summary.head()

musicbrainz_summary.head()


I am working on the file "Exploration_of_data_in_MillionMusicSubset.ipynb".
The first task is:
Scatter plots of artist_hotttnesss vs year
Can you write Python code for it?

fig, ax = plt.subplots(nrows=1, ncols=2, sharex=True, sharey=True,
                       figsize=(15,8))

ax[0].scatter(songs.year, songs.artist_hotttnesss, marker='.')

ax[1].hexbin(songs.year, songs.artist_hotttnesss, cmap='viridis', gridsize=41, mincnt=1.0)

plt.subplots_adjust(wspace=0.02);


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Aby sme zamedzili oznaceniu nevyznamnych rozdielov ako vyznamnych kvoli tomu, ze sme opakovali vela vyhodnoteni podmnozin, tak musime pouzit korekciu. Najcastejsie sa pouziva znizenie p-hodnoty. Ak hladame signifikanciu na urovni 5%, tak pri jednom teste musi byt p-hodnota < 0.05. Pri viacerych opakovaniach by sme mali tuto hodnotu vydelit poctom opakovani experientu. Cize nova hranica signifikancie je 0.05 / 10 = 0.005. Tato korekcia sa vola Bonferroniho korekcia.
Tato korekcia ale moze byt prilis konzervativna. Namiesto nej sa pouziva Tukeyho test.
Can you write Python code for it?

from statsmodels.stats.multicomp import pairwise_tukeyhsd

tukey = pairwise_tukeyhsd(endog=voter_age,
                          groups=voter_race,
                          alpha=0.05)

tukey.plot_simultaneous()
tukey.summary()


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Aj ked pridame trochu sumu, tak je to velmi podobne
Can you write Python code for it?

x = np.arange(100)
y = x + stats.norm(0,30).rvs(100)

synth_data = pd.DataFrame({
    'x': x,
    'y': y
})

regr = lm.LinearRegression()
regr.fit(synth_data.x.values.reshape(100,1), synth_data.y)

seaborn.regplot(x="x", y="y", data=synth_data)
print("Pearson correlation: %.3f, Regresion coefficient: %.3f" % (synth_data.x.corr(synth_data.y), regr.coef_[0]))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Ak chcem vizualizovat vztah vsetkych kombinacii atributov, tak mozem spravit pairplot
Pozor, pri velkom pocte atributov je to dost necitatelne a velmi dlho sa to vytvara
Can you write Python code for it?


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Ak chceme overit, ci rozne podmnoziny maju rovnake/rozdielne vlastnosti, tak potrebujeme nejake statisticke testy.
Najcastejsie testy su tu t-test a Anova na overenie, ci jednotlive podmnoziny maju rozne priemery. Anova sa pouziva ak porovnavame viac ako dve podmnoziny.
T-test
Ak chceme porovnavat mnozinu s celou populaciou, tak potrebujeme jednovyberovy t-test
Ak chceme porovnavat dve nezavisle mnoziny, tak potrebujeme dvojvyberovy t-test.
Ak by sme chceli testovat rozdiely medzi dvoma vzorkami tej istej mnoziny, medzi ktorymi je napriklad casova zavislost (zmena hodnoty po nejakom case/ukone (vysledky testu pred a po pouziti vyucbovej metody, vaha pred a po diete ...) ), tak musime pouzit parovy t-test
Teraz chceme porovnavat dve mnzoiny, tak si dame ukazku Dvojvyberoveho a Paroveho t-testu
Dvojvyberovy t-test (Two sample t-test)
Can you write Python code for it?

np.random.seed(12)
ages1 = stats.poisson.rvs(loc=18, mu=33, size=30)
ages2 = stats.poisson.rvs(loc=18, mu=13, size=20)

print(ages1.mean(), ages2.mean())

plt.hist(ages1)

stats.ttest_ind(a= ages1,
                b= ages2,
                equal_var=False)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Ak ide o oznackovane data, tak ich vieme ofarbit pomocou znacky a pozriet sa na to, ci sa daju na zaklade tychto atributov rozdelit do skupin.
Can you write Python code for it?

for name, group in iris.groupby("species"):
    plt.scatter(group.sepal_length, group.sepal_width, label=name)

plt.legend()


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Analyza po jednom (Univariate analysis)
Zobrazenie vlastnosti jedneho atributu
Spojite atributy
Chceme zobrazit aky je tvar rozdelenia dat, ci sa zoskupuju okolo nejakeho centra, aka je rozptylenost hodnot
Zobrazenie centralnosti:

mean (priemer)
median (median, stredna hodnota, prostredna hodnota, centralna hodnota): hodnota, ktora rozdeluje vyssie a nizsie hodnoty
mode (Modus, modálna hodnota, najpravdepodobnejšia hodnota): najcastejsia hodnota (hodnota s najvacsou pravdepodobnostou vyskytu)

Rozptylenost

range (rozsah): max - min
quartile (kvartil): hodnota, od ktorej je 25% resp 75% hodnot vacsich
percentile (percentil): hodnota, od ktorej je XX% hodnot vacsich

inter quartile range (medzikvartilove rozpätie): rozdiel medzi 25% a 75% kvartilom, menej nachylne na outlierov ako rozsah

variance (variancia): priemerna kvadraticka odchylka od priemeru
$$ E[(X-E[X])^2] $$

standard deviation (standardna odchylka): druha odmocnina variancie, je v jednotkach meranej premennej
skewness (vychylenost?): metrika symetrickosti rozdelenia, ci je rozdelenie navazene na jednu stranu
kurtosis (zplostenost?): ake mnozstvo dat je vo chvoste rozdelenia

(zdroj obrazku: https://taps-graph-review.wikispaces.com/Box+and+Whisker+Plots)
Skewness a Kurtosis
Skewness
Skewness je metrika toho, ako je rozdelenie symetricke. Uplne symetricke rozdelenie ma hodnotu skewness rovnu 0. V podstate to porovnava relativnu velkost dvoch chvostov rozdelenia.
Rozdelenie naklonene do lava bude mat skewness vacsiu ako 0, naklonene doprava bude mat menej ako 1.
Can you write Python code for it?

sample_size = 10000

norm = stats.norm(0, 1)
x = np.linspace(-5, 5, 100)
sample = norm.rvs(sample_size)

plt.plot(x, norm.pdf(x))
plt.hist(sample, normed=True, bins=20)
plt.title("Normalne rozdelenie: ""Skewness %.5f" % (stats.skew(sample), ))

sample_size = 1000

chi2 = stats.chi2(5)
x = np.linspace(0, 30, 100)
sample = chi2.rvs(sample_size)

plt.plot(x, chi2.pdf(x))
plt.hist(sample, normed=True, bins=20)
plt.title("Chi-kvadrat(5) rozdelenie: ""Skewness %.5f" % (stats.skew(sample), ))

sample_size = 1000

chi2 = stats.chi2(5)
x = np.linspace(0, 30, 100)
sample = 30 - chi2.rvs(sample_size)

plt.plot(x, chi2.pdf(30 - x))
plt.hist(sample, normed=True, bins=20)
plt.title("30 - Chi-kvadrat(5) rozdelenie: ""Skewness %.5f" % (stats.skew(sample), ))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Analyza po jednom - spojite atributy - vizualizacia
Ako ste uz urcite pochopili, tak primarne sposoby vizualizacie su histogram a box plot
Can you write Python code for it?

sample_size = 100000

norm = stats.norm(0, 1)

x = np.linspace(-5, 5, 100)
sample = np.concatenate([
    stats.norm(0, 1).rvs(sample_size),
    stats.norm(2, 0.5).rvs(sample_size),
    ])

_ = plt.hist(sample, normed=True, bins=50)

plt.rc("lines", markeredgewidth=0.5)
_ = plt.boxplot(sample)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Anova nam povedala, ze je tam vyznamny rozdiel medzi priemernym vekom medzi niektorymi skupinami. Nepovedala nam ale medzi ktorymi.
Jeden mozny sposob ako zistit medzi ktorymi je spustit t-test nad kazdou dvojicou.
Tu ale hrozi, ze oznacime nevyznamne rozdiely ako vyznamne (vid XKCD).
Can you write Python code for it?

race_pairs = []

for race1 in range(len(races)):
    for race2  in range(race1+1,5):
        race_pairs.append((races[race1], races[race2]))

for race1, race2 in race_pairs:
    print(race1, race2)
    print(stats.ttest_ind(voter_age[groups[race1]],
                          voter_age[groups[race2]]))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Daju sa tiez spravit aj tabulky s vyssimi dimenziami. Tam sa ale uz velmi rychlo straca prehladnost
Can you write Python code for it?

pd.crosstab(index=titanic["Survived"],
            columns=[titanic["Pclass"], titanic["Sex"]],
            margins=True)

pd.crosstab(index=titanic["Survived"],
            columns=[titanic["Pclass"], titanic["Sex"], titanic["Embarked"]],
            margins=True)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Ked sa potrebujete rozhodnut aky test pouzit, tak moze pomoct takyto rozhodovaci strom

Tu je iny priklad tabulky na vyber statistickej metody
Je to z kurzu CHS 627: Multivariate Methods in Health Statistics (z Univerzity v Alabame)
pdf
Can you write Python code for it?

class PDF(object):
    def __init__(self, pdf, size=(200,200)):
        self.pdf = pdf
        self.size = size

    def _repr_html_(self):
        return '<iframe src={0} width={1[0]} height={1[1]}></iframe>'.format(self.pdf, self.size)

    def _repr_latex_(self):
        return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

PDF('choosing-the-correct-statistical-test-chs-627-university-of-alabama.pdf',size=(1000,1000))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Kriticka hodnota je mensia ako namerana hodnota statistiky na nasej vzorke a zaroven p-hodnota je mensia ako 0.01 na zvolenej hranici istoty, takze mozeme povedat ze pozorovania su z rovnakeho rozdelenia
Pozor!
ak by p-hodnota nebola mensia ako 0.01, tak by sme nemohli povedat, ze mame statisticky dokaz toho, ze data su z rozneho rozdelenia. Mozeme povedat len to, ze nemame dostatok dokazov na to aby sme zamietli nulovu hypotezu a teda ze nevieme zamietnut hypotezu, ze su z rozneho rozdelenia.
Can you write Python code for it?

# pre pohodlnost existuje pripravena funkcia, ktora to spocita za nas
# pozor na to, ze expected niesu namerane data pre celu populaciu ale ocakavane pocetnosti pri rovnakom pocte pozorovani ako ma
# sledovana datova sada
stats.chisquare(f_obs= observed, f_exp=expected)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Kurtosis
Kurtosis hovori aka je kombinovana velkost chvostov. Meria mnozstvo dat sustredene v chvostoch.
Velmi casto sa porovnava k hodnote kurtosis normalneho rozdelenia, ktora je 3.
Ak je to viac ako 3, tak viac dat je sustredenych na okrajoch. Ak menej ako 3, tak je menej dat v okrajoch.
Casto sa pouziva aj excess kurtosis, co je rozdiel oproti normalnemu rozdeleniu, cize kurtosis - 3.
Can you write Python code for it?

sample_size = 100000

norm = stats.norm(0, 1)
x = np.linspace(-5, 5, 100)
sample = norm.rvs(sample_size)

plt.plot(x, norm.pdf(x))
plt.hist(sample, normed=True, bins=20)
plt.title("Normalne rozdelenie: ""Kurtosis %.5f" % (stats.kurtosis(sample), ))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Nameranu statistiku musime porovnat s kritickou hodnotou
Can you write Python code for it?

crit = stats.chi2.ppf(q = 0.95, # 95% confidence
                      df = 4)   # pocet stupnov volnosti testu = pocet kategorii - 1

print("Critical value:", crit) # kriticka hodnota by mala byt mensia ako hodnota chi-kvadrat statistiky, ktoru sme ziskali z dat

p_value = 1 - stats.chi2.cdf(x=chi_squared_stat,  # aka je p-hodnota nasho testu
                             df=4)
print("P value:", p_value)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Nastastie vieme zmenit teoreticke rozdelenie a mozeme sa porovnan s nim
Can you write Python code for it?

import statsmodels.api as sm
x = stats.norm(8.25, 2.75).rvs(1000)
y = stats.lognorm(0.5).rvs(1000)
pp_x = sm.ProbPlot(x, fit=True)
pp_y = sm.ProbPlot(y, fit=True)
_ = pp_x.qqplot(line='45', other=pp_y)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Obcas ma dokonca zmysel prekryvat viacere ciary cez seba, na to aby ste zobrazili viacero atributov / metrik / casovych obdobi ...
Can you write Python code for it?


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Ocakavali sme, ze dostaneme hodnotu kurtosis okolo 3. V skutocnosti funkcia stats.kurtosis pocita pri predvolenych nastaveniach excess kurtosis
Can you write Python code for it?

sample_size = 100000

norm = stats.norm(0, 1)
x = np.linspace(-5, 5, 100)
sample = norm.rvs(sample_size)

plt.plot(x, norm.pdf(x))
plt.hist(sample, normed=True, bins=20)
plt.title("Normalne rozdelenie: ""Kurtosis %.5f" % (stats.kurtosis(sample, fisher=False), ))
# musime prestavit parameter fisher na False


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Ocividne je hovadina porovnavat tieto nase pozorovania s uplne inou distribuciou, ale moze sa to hodit, ked mame ine pozorovania
Can you write Python code for it?

_ = sm.ProbPlot(lognorm_sample, dist=stats.lognorm, fit=True).qqplot(line='45')


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Osobne mam celkom rad spojenie boxplotu a histogramu do Violinplotu pretoze prehladne ukazuje tvar rozdelenia.
Castejsie sa ale pouziva vykreslenie dvoch obrazkov (aj histogram a aj boxplot). Spolu obsahuju viac informacii ako len jeden violinplot
Can you write Python code for it?


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Pozor sklon regresnej ciary nehovori o sile korelacie. Len o smere.
Can you write Python code for it?

regr = lm.LinearRegression()
regr.fit(car_data.horsepower.values.reshape(len(car_data),1), car_data.mpg)

seaborn.regplot(x="horsepower", y="mpg", data=car_data)
print("Pearson correlation: %.3f, Regresion coefficient: %.3f" % (car_data.horsepower.corr(car_data.mpg), regr.coef_[0]))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Samozrejme sa da aj pairplot ofarbit triedou
Can you write Python code for it?


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Stacked bar chart
Can you write Python code for it?


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Toto je extremny pripad, kde korelaciu nevieme spocitat, kedze jedna hodnota je len konstanta (nulova variancia), pricom je to pekna rovna ciara so sklonom 0.
Can you write Python code for it?

x = np.arange(100)
y = x * (-3.14) + 3

synth_data = pd.DataFrame({
    'x': x,
    'y': y
})

regr = lm.LinearRegression()
regr.fit(synth_data.x.values.reshape(100,1), synth_data.y)

seaborn.regplot(x="x", y="y", data=synth_data)
print("Pearson correlation: %.3f, Regresion coefficient: %.3f" % (synth_data.x.corr(synth_data.y), regr.coef_[0]))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Tu je perfektne linearny vztah, kde je jasne vidiet, ze ten sklon je uplne iny.
Can you write Python code for it?

x = np.arange(100)
y = x + stats.norm(0,1).rvs(100)

synth_data = pd.DataFrame({
    'x': x,
    'y': y
})

regr = lm.LinearRegression()
regr.fit(synth_data.x.values.reshape(100,1), synth_data.y)

seaborn.regplot(x="x", y="y", data=synth_data)
print("Pearson correlation: %.3f, Regresion coefficient: %.3f" % (synth_data.x.corr(synth_data.y), regr.coef_[0]))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Velmi pekny prispevok o tom ako interpretovat QQ-plot
https://stats.stackexchange.com/questions/101274/how-to-interpret-a-qq-plot

Analyza po jednom - kategoricke atributy
Tu je najcastejsim sposobom zobrazenia frekvencna tabulka zobrazujuca bud pocty pozorovani per unikatna hodnota atrinutu alebo pomer voci celkovemu poctu pozoorvani.
Graficka vizualizacia je bar plot
Can you write Python code for it?

diamonds = pd.read_csv('data/diamonds.csv')
diamonds.head()

diamonds.color.value_counts()

diamonds.color.value_counts() / len(diamonds)

diamonds.color.value_counts().plot(kind='bar')


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Vysledok tohto testu je, ze namerana hodnota chi-kvadrat statistiky je mensia ako kriticka hodnota a nam sa nepodarilo doukazat zavislost medzi tymito dvoma premennymi.
Co dava celkom zmysel, kedze sme data generovali z dvoch uplne nezavislich nahodnych premennych.
Nepodaril osa nam teda vyvratit nulovu hypotezu a nemame dostatok dokazov na to, aby sme povedali, ze existuje zavislost medzi tymito dvoma premennymi
Spojity - Kategoricky
Tu sa najcastejsie pouziva rozdelovanie podla kategorickej hodnoty a zobrazovanie rozdeleni podmnozin numerickych hodnot napriklad pomocou histogramov alebo box-plotov.
Cize viacnasobne pouzitie vizualizacii, ktore sa pouzivaju na zobrazenie spojitych atributov
Can you write Python code for it?


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
Vzorec na vypocet chi-kvadrat statistiky
$$ \sum{\frac{(observerd - expected)^2}{expected}}$$
Can you write Python code for it?

observed = minnesota_table

national_ratios = national_table/len(national)  # pomery pre celu populaciu (referencne rozdelenie)
print('Ratios:', national_ratios)

expected = national_ratios * len(minnesota)   # ocakavane hodnoty ak by mala sledovana vzorka rovnake rozdelenie ako cela populacia
print("Expected:", expected) # ak su z rovnakeho rozdelenia, tak toto by malo byt velmi podobne ako obsah premennej minnesota_table

chi_squared_stat = (((observed-expected)**2)/expected).sum()

print("Chi-squared", chi_squared_stat) # vysledna hodnota chi-kvadrat statistiky


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
ak by som chcel manualne vytvarat nejake pravidla na klasifikaciu, tak z tohto obrazku by som uz nejake vedel vyrobit
Scatterplot sa da pouzit nie len na vizualizovanie zhlukov ale aj korelacii / zavislosti.

Nedokaze vsak kvantifikovat silu vztahu. Na to potrebujeme nejaku inu metriku - korelaciu.
Zdroj obrazku: https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/#one
Korelacia
Hodnota v rozsahu [-1, 1], ktora hovori o tom, aky silny linearny vztah je medzi atributmi.

-1 perfektna negativna korelacia
0 ziadna korelacia
1 perfektna kladna korelacia

Pearsnov korelacny koeaficient:
$$ corr(X, Y) = \frac{cov(X,Y)}{E[X]E[Y]} = \frac{E[(X-E[X])(Y-E[Y])]}{E[X]E[Y]}$$
Can you write Python code for it?

car_data = pd.read_csv('data/auto-mpg.data', delim_whitespace=True,
                       names = ['mpg', 'cylinders', 'displacement','horsepower',
                                'weight', 'acceleration', 'model_year', 'origin', 'name'],
                        na_values='?'
                      )
car_data.head()

car_data.describe()

car_data = car_data.dropna()

seaborn.regplot(x="horsepower", y="mpg", data=car_data)
print("Pearson correlation: %.3f" % car_data.horsepower.corr(car_data.mpg))


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
p-hodnota je menej ako 0.01, takze mozeme povedat, ze sme nasli statisticky vyznamny rozdiel v priemernych hodnotach tychto dvoch vzoriek
Parovy t-test
Can you write Python code for it?

np.random.seed(11)
before= stats.norm.rvs(scale=30, loc=250, size=100)
after = before + stats.norm.rvs(scale=5, loc=-1.25, size=100) # Upravime povodnu vzorku.
                                                              # Simulujeme tak nejaky proces, po ktorom chceme overit zmenu
weight_df = pd.DataFrame({"weight_before":before,
                          "weight_after":after,
                          "weight_change":after-before})

weight_df.describe()

stats.ttest_rel(a = before,
                b = after)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
p-hodnota nieje mensia ako 0.01, takze mozeme povedat, ze sme nenasli dostatok dokazov na zamietnutie nulovej hypotezy a teda nevieme povedat, ci je tam rozdiel v strednych hodnotach
ANOVA

zdroj obrazku: https://xkcd.com/882/
Na zistenie rozdielov medzi viacerymi skupinami by sme nemali robit opakovany t-test len tak. Hrozi nam, ze najdeme nevyznamny vysledok len kvoli nahode. P-hodnota znamena, ze s nejakou pravdepodobnostou dosiahnuty vysledok moze byt nahoda. Casta hranica je 5% alebo 1%. Ale aj toto je pravdepodobnost, ktora obcas nastane. Ak test opakujeme viac krat, tak sa nam to moze realne stat.
Can you write Python code for it?

np.random.seed(12)

races =   ["asian","black","hispanic","other","white"]

# Generate random data
voter_race = np.random.choice(a= races,
                              p = [0.05, 0.15 ,0.25, 0.05, 0.5],
                              size=1000)

# Use a different distribution for white ages
white_ages = stats.poisson.rvs(loc=18,
                              mu=32,
                              size=1000)

voter_age = stats.poisson.rvs(loc=18,
                              mu=30,
                              size=1000)

voter_age = np.where(voter_race=="white", white_ages, voter_age)

# Group age data by race
voter_frame = pd.DataFrame({"race":voter_race,"age":voter_age})
groups = voter_frame.groupby("race").groups

# Extract individual groups
asian = voter_age[groups["asian"]]
black = voter_age[groups["black"]]
hispanic = voter_age[groups["hispanic"]]
other = voter_age[groups["other"]]
white = voter_age[groups["white"]]

# Perform the ANOVA
stats.f_oneway(asian, black, hispanic, other, white)


I am working on the file "Explorativna_analyza.ipynb".
The first task is:
toto vychadza z vety o sucine nezavislich premennych:
Ak A a B su nezavisle premenne take ze P(A) > 0 a P(B) > 0, tak
$$ P(A \cap B) = P(A) \times P(B)$$
expected obsahuje ocakavane pocetnosti v pripade ak by tato nezavislost platila.
Can you write Python code for it?

chi_squared_stat = (((observed-expected)**2)/expected).sum().sum()

print(chi_squared_stat)

crit = stats.chi2.ppf(q = 0.95, # Find the critical value for 95% confidence*
                      df = 8)   # pocet stupnov volnosti je sucin poctu kategorii pre kazdu premenny - 1.
                                # Pocty su 3 a 5, teda 2 * 4 = 8

print("Critical value:", crit )

p_value = 1 - stats.chi2.cdf(x=chi_squared_stat,  # Find the p-value
                             df=8)
print("P value:", p_value)

# znova, existuje predpripravena funkcia, ktorej stacia pozorovane pocetnosti a vrati nam
# chi-kvadrat statistiku, p-hodnotu, pocet stupnov volnosti a ocakavane pocetnosti
stats.chi2_contingency(observed=observed)


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
3 cylinder cars are all Mazdas with rotary engines.  These actually have 3 chambers per rotor and 2 rotors per engine - so 3 x 2 = 6.  As plotted by the box plot they do indeed appear to share fuel economy characteristics with 6 cylinder engines.  Possibly in a machine learning model it might be best to replace these by 6 cylinders, especially since there are so few of them.
Lets try a plot to check vehicle weight vs fuel economy (mpg) and add a regression line to the plot.
Can you write Python code for it?

# fit the line
z = np.polyfit(df['weight'], df['mpg'], 1)
y_poly = [z[0] * x + z[1] for x in range(1500, 5600, 100)]
x_poly = [x for x in range(1500, 5600, 100)]

# plot
plt.plot(df['weight'], df['mpg'], ".", color=colors[0])
plt.plot(x_poly, y_poly, "-", color=colors[1])
plt.xlim(1500, 5500)
plt.ylim(0, 50)
plt.xlabel("Vehicle Weight")
plt.ylabel("MPG")
plt.title("Weight vs. MPG")
plt.show()
plt.close()


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
A scatter plot for mpg vs. weight with 2nd order regression fit curves by vehicle origin.  Shows that vehicle origin may have some effect on fuel economy even when vehicle weight is considered.  It is a complex relationship since lighter Japanese cars have higher fuel economy and the heavier amongst the European cars have higher fuel economy than their equivalent American cars.
Can you write Python code for it?

df_usa = df.loc[df['origin_name'] == 'US']
z = np.polyfit(df_usa['weight'], df_usa['mpg'], 2)
x_poly_usa = [x for x in range(np.min(df_usa['weight']), np.max(df_usa['weight'] + 100), 100)]
y_poly_usa = [z[0] * x ** 2 + z[1] * x + z[2] for x in range(np.min(df_usa['weight']),
                                                          np.max(df_usa['weight'] + 100), 100)]

df_eur = df.loc[df['origin_name'] == 'EUR']
z = np.polyfit(df_eur['weight'], df_eur['mpg'], 2)
x_poly_eur = [x for x in range(np.min(df_eur['weight']), np.max(df_eur['weight'] + 100), 100)]
y_poly_eur = [z[0] * x ** 2 + z[1] * x + z[2] for x in range(np.min(df_eur['weight']),
                                                          np.max(df_eur['weight'] + 100), 100)]

df_jpn = df.loc[df['origin_name'] == 'JPN']
z = np.polyfit(df_jpn['weight'], df_jpn['mpg'], 2)
x_poly_jpn = [x for x in range(np.min(df_jpn['weight']), np.max(df_jpn['weight'] + 100), 100)]
y_poly_jpn = [z[0] * x ** 2 + z[1] * x + z[2] for x in range(np.min(df_jpn['weight']),
                                                          np.max(df_jpn['weight'] + 100), 100)]

plt.plot(df['weight'], df['mpg'], ".", color=colors[0], alpha=0.33)
plt.plot(x_poly_usa, y_poly_usa, "-", color=colors[1])
plt.plot(x_poly_eur, y_poly_eur, "-", color=colors[2])
plt.plot(x_poly_jpn, y_poly_jpn, "-", color=colors[4])
plt.xlim(1500, 5500)
plt.ylim(0, 50)
plt.xlabel("Vehicle Weight")
plt.ylabel("MPG")
plt.title("Weight vs. MPG, Fitted by Vehicle Origin")
plt.show()
plt.close()


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
Check for normal distributions of the data - the test fails - note the p-values are far below the normal level of significance of 0.05.  All three cylinder types fail the normality test.
Can you write Python code for it?

s, p = stats.normaltest(df468['mpg'].loc[df468['cylinders'] == 4])
print 'Normality test for mpg with 4 cylinders, p-value: %.4f' % p
s, p = stats.normaltest(df468['mpg'].loc[df468['cylinders'] == 6])
print 'Normality test for mpg with 6 cylinders, p-value: %.4f' % p
s, p = stats.normaltest(df468['mpg'].loc[df468['cylinders'] == 8])
print 'Normality test for mpg with 8 cylinders, p-value: %.4f' % p


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
Even with non-normal data Levene's test can be used to check for equal variances.  This test shows low p-values for the 4 cylinder vehicles against 6 and 8 cylinder vehicles, so we cannot assume equal variances going forward.
Can you write Python code for it?

W, p = stats.levene(df468['mpg'].loc[df468['cylinders'] == 4], df468['mpg'].loc[df468['cylinders'] == 6])
print 'Levene test for equal variances, 4 vs 6 cylinder vehicles, p-value: %.4f' % p
W, p = stats.levene(df468['mpg'].loc[df468['cylinders'] == 4], df468['mpg'].loc[df468['cylinders'] == 8])
print 'Levene test for equal variances, 4 vs 8 cylinder vehicles, p-value: %.4f' % p
W, p = stats.levene(df468['mpg'].loc[df468['cylinders'] == 6], df468['mpg'].loc[df468['cylinders'] == 8])
print 'Levene test for equal variances, 6 vs 8 cylinder vehicles, p-value: %.4f' % p


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
In the plot above the orange line shows weight vs. mpg for cars from the US.  The regression fit curve covers the weight range for US cars.  The green line is similar for European cars and the maroon line for Japanese cars with the curves covering the weight range for their respective origin codes.
Can we verify statistically what the box plot for mpg by cylinders apears to show - that cylinders are significant as a predictor of fuel economy?  I will work with only the 4, 6 and 8 cylinder cars here since 3 and 5 cylinder cars are rare.
Set up data frame for 4, 6 and 8 cylinder vehicles.Leave out diesel vehicles since they are likely to be outliers and there are not many of them.  Thus, this checks the effect of cylinders for gasoline engine cars.
Can you write Python code for it?

df468 = df.loc[df.cylinders != 3]
df468 = df468.loc[df468.cylinders != 5]
df468 = df468.loc[df.diesel != 1]


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
It appears that the relationship is not really linear.  So try a second order polynomial fit.
Can you write Python code for it?

# second order fit
z = np.polyfit(df['weight'], df['mpg'], 2)
y_poly2 = [z[0] * x ** 2 + z[1] * x + z[2] for x in range(1500, 5600, 100)]

# plot it
plt.plot(df['weight'], df['mpg'], ".", color=colors[0])
plt.plot(x_poly, y_poly2, "-", color=colors[1])
plt.xlim(1500, 5500)
plt.ylim(0, 50)
plt.xlabel("Vehicle Weight")
plt.ylabel("MPG")
plt.title("Weight vs. MPG")
plt.show()
plt.close()


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
Lets check the effect of the number of cylinders a vehicle has vs. the fuel economy (mpg) by using a box plot.
Can you write Python code for it?

mpg3cyl = df['mpg'][df['cylinders'] == 3]
mpg4cyl = df['mpg'][df['cylinders'] == 4]
mpg5cyl = df['mpg'][df['cylinders'] == 5]
mpg6cyl = df['mpg'][df['cylinders'] == 6]
mpg8cyl = df['mpg'][df['cylinders'] == 8]

cyl = [mpg3cyl, mpg4cyl, mpg5cyl, mpg6cyl, mpg8cyl]

bplot = plt.boxplot(cyl,
                    notch=False,  # notch shape
                    vert=True,  # vertical box alignment
                    patch_artist=True,  # fill with color
                    labels=['3 Cyl', '4 Cyl', '5 Cyl', '6 Cyl', '8 Cyl'])  # will be used to label x-ticks
for patch, color in zip(bplot['boxes'], colors):
        patch.set_facecolor(color)
        patch._alpha = 0.5
for line in bplot['medians']:
    line._color = 'k'

plt.xlabel("Number of Engine Cylinders")
plt.ylabel("MPG")
plt.title("Engine Cylinders vs. MPG")
plt.show()


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
Mean mpg by the region of origin of a vehicle.  The origin code is translated to a more human readable format.  Origin is either related to fuel economy or lighter vehicles come from Japan and Europe than from the US in this time period (1970's to 1980's).
Can you write Python code for it?

df['origin_name'] = df['origin'] - 1
df['origin_name'] = df['origin_name'].astype('category')
df['origin_name'] = pd.Series(pd.Categorical.from_codes(df['origin_name'], categories=['US', 'EUR', 'JPN']))
mean_mpg = df.groupby('origin_name', as_index=False)['mpg'].mean()
print 'Mean value of mpg, by vehicle origin:'
print mean_mpg.to_string(index=False)


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
Mood's Median Test, another way of checking that the number of cylinders may be predictive of mpg. Or put another way, that automobiles with differing numbers of cylinders have significantly different mpg.  This is a comparison for 4, 6 and 8 cylinder cars.
Can you write Python code for it?

stat, p, med, tbl = stats.median_test(d_data[4], d_data[6], d_data[8])
print "Mood's median p-value: %.4f" % p
print "Mood's median value: %.4f (median mpg)" % med


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
Set up a color palette reputed to be color blind friendly.  From: https://gist.github.com/thriveth/8560036.
Can you write Python code for it?


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
The Kruskal ANOVA test can be used when conditions of normality and equal variances are not satisfied.  The p-value is near zero, so cylinders seem have an effect on mpg, though we may need to consider multi-collinearity with other variables such as weight and displacement.
Can you write Python code for it?

grps = pd.unique(df468.cylinders.values)
d_data = {grp: df468['mpg'][df468.cylinders == grp] for grp in grps}
kruskal = stats.kruskal(d_data[4], d_data[6], d_data[8])
print 'kruskal'
print kruskal


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
We can observe that more cylinders generally lead to increasing weight and lower mpg.  Multi-collinearity could be a problem in this data set, here the relationship between weight and cylinders.
A look at displacement vs. horsepower, another source of possible multi-collinearity in the data since there is a generally linear relationship.
Can you write Python code for it?

z = np.polyfit(df['displacement'], df['horsepower'], 1)
y_poly = [z[0] * x + z[1] for x in range(0, 600, 50)]
x_poly = [x for x in range(0, 600, 50)]

plt.plot(df['displacement'], df['horsepower'], '.', color=colors[0])
plt.plot(x_poly, y_poly, '-', color=colors[1])
plt.xlim(0, 500)
plt.ylim(0, 300)
plt.xlabel("Displacement")
plt.ylabel("Horsepower")
plt.title("Displacement vs. Horsepower")
plt.show()
plt.close()


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
We can use the stats models multiple tests function to correct the p-values for the three test pairs so that we can see which pairs are significant.  The adjustment for multiple tests gives a better result than simple pairwise comparison since it adjusts for the loss of power in doing multiple tests.  Here the p-values are all still zero so each combination of engine cylinders has a significantly different median mpg value.
Can you write Python code for it?

stat, p46, med, tbl = stats.median_test(d_data[4], d_data[6])
stat, p48, med, tbl = stats.median_test(d_data[4], d_data[8])
stat, p68, med, tbl = stats.median_test(d_data[6], d_data[8])

rej, pvc, aS, aB = mc.multipletests((p46, p48, p68))
print pvc


I am working on the file "Exploratory Data Analysis UCI Auto MPG Data Set.ipynb".
The first task is:
What is with 3 cylinder vehicles, why do they break the pattern of decreasing fuel economy with more cylinders?
Can you write Python code for it?

df3cyl = df.loc[df['cylinders'] == 3]
print ' 3 cylinder cars'
print df3cyl['car_name']


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Compare undeclared payments by category
As above, but looking for which categories have the largest share of undeclared payments.
Can you write Python code for it?

# As above, EXCLUDE R&D.
df_agg_by_cat = df_agg_ex_rd.groupby('TOV Category').sum().reset_index()
df_agg_cat = df_agg_by_cat[['TOV Category',  'Amount']]
df_by_cat = df.groupby(('TOV Category')).sum().reset_index()
df_merged_cat = pd.merge(df_agg_cat, df_by_cat, on=['TOV Category'],
                     suffixes=(' Aggregate', ' Declared'))
df_merged_cat['Total Amount'] = df_merged_cat['Amount Declared'] + \
    df_merged_cat['Amount Aggregate']
df_merged_cat['Proportion Undeclared'] = df_merged_cat['Amount Aggregate'] / \
    df_merged_cat['Total Amount']
df_merged_cat.sort_values('Proportion Undeclared', ascending=False, inplace=True)
df_merged_cat.to_csv('declared_vs_aggregated_payments_by_category.csv')
df_merged_cat


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Distribution of payments
(Not very interesting) Most individual payments are very small. 25% are below £180, 50% are below £355, 70% are below £800.
See histogram and note the log scale on the y-axis.
Can you write Python code for it?

%pylab inline
plt.figure()
bin_range = np.arange(0, 1100000, 10000)
df['Amount'].plot(kind='hist', bins=bin_range, color='blue', alpha=0.6)
plt.ylabel("Number of payments")
plt.xlabel("Payment size")
plt.yscale('log', nonposy='clip')
plt.title("Payments by amount")
plt.grid()


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Most payments are small! 50% are below £355, 75% are below £800. Only 1,164 payments are more than £10,000.
Can you write Python code for it?

print df['Amount'].describe()
print len(df[df['Amount'] > 10000]), 'payments overall are more than £10,000'

print '\nPayments to organisations:'
print df_hcos['Amount'].describe()
print '\nPayments to individuals:'
print df_hcps['Amount'].describe()


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Payments by organisation
Again, these are the declared payments - we can't break down the aggregate payments this way.
Can you write Python code for it?

df_by_org = df.groupby('Organisation Name').agg(['sum', 'mean', 'median', 'count'])
df_by_org.sort_values(by=('Amount', 'sum'), ascending=False).head(10)


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Payments by pharma co
These are the declared payments, not the undeclared payments. Not sure how meaningful this is really.
Can you write Python code for it?

df_by_co = df.groupby('Company Name').agg(['sum', 'mean', 'median', 'count'])
df_by_co.sort_values(by=('Amount', 'sum'), ascending=False).head(10)


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Payments by pharma companies by category
Break down the above table by category, too. Again, this is only declared payments.
Can you write Python code for it?

#  df[['Company Name', 'TOV Category', 'Amount']].head()
df.groupby(('Company Name', 'TOV Category')).sum().sort_values('Amount', ascending=False).head(10)


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Payments by pharma companies by speciality
Break down the above table by speciality, too. Again, this is only declared payments.
Can you write Python code for it?

df_by_co_and_speciality = df.groupby(('Company Name', 'Speciality')).sum()\
    .sort_values('Amount', ascending=False).reset_index()

total = df_by_co_and_speciality.groupby('Company Name')['Amount'].transform('sum')
df_by_co_and_speciality['% of Co Spend'] = df_by_co_and_speciality['Amount']/total * 100

print df_by_co_and_speciality.head(10)
# Validate % calculation
# print df_by_co_and_speciality[df_by_co_and_speciality['Company Name'] == 'AstraZeneca']['% of Company Total Spend'].sum()
df_by_co_and_speciality.to_csv('payments_by_co_and_speciality.csv')


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Payments by speciality
Again, these are the declared payments - we can't break down the aggregate payments this way.
Nurses get more payments than anyone else, but they certainly don't get the highest mean payment.
Write the data to a CSV file for further analysis.
Can you write Python code for it?

df_by_spec = df.groupby('Speciality').agg(['sum', 'mean', 'median', 'count']).sort_values(by=('Amount', 'sum'), ascending=False)
print df_by_spec.head(10)
df_by_spec.to_csv('by_speciality.csv')


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Payments by specialty, with custom groupings
Create three groups: Nurses, Pharmacists, Doctors, and Other, and run the same analysis
Can you write Python code for it?

# Normalise the specialties
def specialty_classifier(row):
    other = ['Healthcare Administration',
             'Microbiology',
             'Laboratory - Medical Analysis',
             'Research',
             'Occupational Therapist',
             'Wholesaler',
             'Miscellaneous']
    pharmacists = ['Clinical Pharmacology', 'Pharmacist']
    nurses = ['Nurse']
    classification = 'HCO'
    if row['Speciality'] in other:
        classification = 'Other'
    elif row['Speciality'] in pharmacists:
        classification = 'Pharmacists'
    elif row['Speciality'] in nurses:
        classification = 'Nurses'
    elif str(row['Speciality']) != "nan":
        # XXX there must be a nicer way to filter out pd.nan values...
        classification = 'Doctors'
    return classification

df['Normalised specialty'] = df.apply(specialty_classifier, axis=1)

# Re-run the analysis
df.groupby('Normalised specialty').agg(['sum', 'mean', 'median', 'count']).sort_values(by=('Amount', 'sum'), ascending=False)


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
The same thing, with custom specialty groupings
Can you write Python code for it?

df_individual = df.groupby(('Last Name', 'First Name', 'Normalised specialty')).sum().reset_index()
df_individual.groupby('Normalised specialty').count()[['Amount']].sort_values("Amount", ascending=False)


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Unique individuals by speciality
We can work out how many uniquely-named individuals there are in each speciality as follows.
"General Medicine" probably != GPs, since Raj Acquilla is in this category and we know him to be a plastic surgeon.
Can you write Python code for it?

df_individual = df.groupby(('Last Name', 'First Name', 'Speciality')).sum().reset_index()
df_individual.groupby('Speciality').count()[['Amount']].sort_values("Amount", ascending=False)


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
What are the biggest single payments to individuals?
Print the top individual payments... they're all 'Fees for service and consultancy'.
Can you write Python code for it?


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Why are there a few negative payments?
Can you write Python code for it?

df.sort_values('Amount').head()[['Company Name', 'Organisation Name', 'TOV Type', 'Amount']]


I am working on the file "Exploratory work on ABPI Disclosure data.ipynb".
The first task is:
Work out how many HCOs/HCPs are missing, by company
With the limited and redacted data we have, perhaps the most meaningful thing we can look at is the amount of data that is undeclared!
The only thing we know about the undeclared payments is the amount of it for each company, in each TOV category, and the number of HCOs/HCPs which the undeclared payments cover.
We can therefore derive, by comparing this with the payments that are declared, per-company values for the total amount that was undeclared, and the total number of HCOs/HCPs that chose not to declare themsleves.
Perhaps this will reveal interesting differences.
Can you write Python code for it?

# Get all aggregate payments but EXCLUDE R&D.
# This is because NO R&D payments are included in the payments sheet, so
# including them in our per-company analyses will distort results.
# Lump both HCPs and HCOs together.
df_agg_by_co = df_agg_ex_rd.groupby(('Company Name')).sum().reset_index()
df_agg_by_co.rename(columns={'No of HCP HCO in Aggregate': 'No of HCP HCO'}, inplace=True)
df_agg = df_agg_by_co[['Company Name', 'Amount', 'No of HCP HCO']]

# We've already created dataframes for HCOs and HCPs: now get the sums
# and counts of these, and merge them. There's probably a better way to do this.
df_hcos_by_co = df_hcos.groupby('Company Name')['Amount'].agg(['sum','count']).reset_index()
df_hcps_by_co = df_hcps.groupby('Company Name')['Amount'].agg(['sum','count']).reset_index()
df_m = pd.merge(df_hcos_by_co, df_hcps_by_co, on='Company Name', suffixes=('_hco', '_hcp'))
df_m['Amount'] = df_m['sum_hco'] + df_m['sum_hcp']
df_m['No of HCP HCO'] = df_m['count_hco'] + df_m['count_hcp']
df_declared = df_m[['Company Name', 'Amount', 'No of HCP HCO']]

df_combined = pd.merge(df_declared, df_agg, on='Company Name',
                        suffixes=(' Declared', ' Undeclared'))

# Calculate totals, proportions etc.
df_combined['Total Amount'] = df_combined['Amount Declared'] + df_combined['Amount Undeclared']
df_combined['Total HCOs/HCPs'] = df_combined['No of HCP HCO Declared'] + df_combined['No of HCP HCO Undeclared']
df_combined['Proportion of names undeclared'] = df_combined['No of HCP HCO Undeclared'] / \
    df_combined['Total HCOs/HCPs'] * 100
df_combined['Proportion of total amount undeclared'] = df_combined['Amount Undeclared'] / \
    df_combined['Total Amount'] * 100

# Print summary stats about the proportion of undeclared payments.
# Note that this number is lower than the 77% above, because we've excluded R&D.
print 'Total % names undeclared:', "{:.2f}%".format(df_combined['No of HCP HCO Undeclared'].sum() / \
    df_combined['Total HCOs/HCPs'].sum() * 100)
print 'Total % amount...


I am working on the file "ExploratoryCode/simplest-ever-multicause-RBM.ipynb".
The first task is:
I will set weights such that patterns 01 and 10  on the visible units get piles of sand (are "memories"), and 00 and 11 aren't.
Can you write Python code for it?


I am working on the file "ExploratoryCode/simplest-ever-multicause-RBM.ipynb".
The first task is:
Note: we show the rbmA state alongside the orbm's Pr(A) here because in theory the orbm probabilities do depend on the existing state. However it is extremely heartening that they don't strongly. Perhaps because this example is just so symmetrical.
Can you write Python code for it?

show_reconstructions('reconstructions', w, v=np.array([[1,1]]), version='EXACT')

def logL_orbm(w, train, num_Gibbs_iters=10):
    # estimate the logL from samples.

    # get sample proportions

    # KL divergence

def samples_from_orbm_METHOD1(w, num_samples=1000, num_Gibbs_iters=10):
    # make shitloads of samples in the "naive" way, using two entirely separate RBMs, which combine at the last moment.
    samples = np.zeros((num_samples, 2), dtype = int)

    for j in range(num_samples):
        uA = (0.5 > rng.random(size=(1,2))).reshape(1,2)
        uB = (0.5 > rng.random(size=(1,2))).reshape(1,2)
        for i in range(num_Gibbs_iters):
            # set the hidden units in each network, which is a regular RBM.
            psiA, psiB = np.dot(uA,w), np.dot(uB,w)
            hA_prob, hB_prob = sigmoid(psiA), sigmoid(psiB)
            hA = (hA_prob > rng.random(size=hA_prob.shape)).reshape(2,1)
            hB = (hB_prob > rng.random(size=hB_prob.shape)).reshape(2,1)
            # set the "latent" (u) units in the two regular RBMs
            phiA, phiB = np.dot(w,hA).T,  np.dot(w,hB).T
            uA_prob, uB_prob = sigmoid(phiA), sigmoid(phiB)
            uA = (uA_prob > rng.random(size=uA_prob.shape)).reshape(1,2)
            uB = (uB_prob > rng.random(size=uB_prob.shape)).reshape(1,2)
        # finally, combine them to make a visible pattern: our sample
        phiA, phiB = np.dot(w,hA).T,  np.dot(w,hB).T
        vis_prob = sigmoid(phiA + phiB).reshape(1,2)
        v = (vis_prob > rng.random(size=vis_prob.shape))
        samples[j,:] = v
    return samples

def samples_from_orbm_METHOD2(w, num_samples=1000, num_Gibbs_iters=10):
    # make shitloads of samples, by sampling using our scheme...
    samples = np.zeros((num_samples, 2), dtype = int)

    for j in range(num_samples):
        v = (0.5 > rng.random(size=(1,2))).reshape(1,2)  # initial vis pattern
        hA = (0.5 > rng.random(size=(1,2))).reshape(1,2)  # initial vis pattern
        hB = (0.5 > rng.random(size=(1,2))).reshape(1,2) ...


I am working on the file "ExploratoryCode/simplest-ever-multicause-RBM.ipynb".
The first task is:
So with weights of 1 that was pretty affirming: (i) the two methods agree, and (ii) the result seems sensible.
But with larger weights (*5) there's a difference - why?
Learning from scratch
Fuckity fuck: something's wrong here.
Can you write Python code for it?

def learn_orbm(w, train, num_train_iters=10, num_Gibbs_iters=10):

    eta, momentum = .01, 0.25 # learning parameters
    prev_dw = np.zeros(shape=w.shape)
    for step in range(num_train_iters):
        dw = np.zeros(shape=w.shape)
        # Wake phase
        for v in train: # 'clamped'
            v = v.reshape(1,2)
            #print('shape of v should be 1,2: ',v.shape)
            # SAMPLE from hA and hB
            # Start by initialising as if it were a vanilla RBM
            psiA, psiB = np.dot(v,w), np.dot(v,w)
            hA_prob, hB_prob = sigmoid(psiA), sigmoid(psiB)
            hA = (hA_prob > rng.random(size=hA_prob.shape)).reshape(2,1)
            hB = (hB_prob > rng.random(size=hB_prob.shape)).reshape(2,1)
            #print('shape of hA should be 2,1: ',hA.shape)
            for t in range(num_Gibbs_iters):
                # update hA
                tmp, hA = update_hidden_layer(hA, hB, w, v, 'EXACT')
                # update hB
                tmp, hB = update_hidden_layer(hB, hA, w, v, 'EXACT')

            phiA, phiB = np.dot(w,hA).T,  np.dot(w,hB).T
            phiA0 = phiA - hA*w
            phiB0 = phiB - hB*w
            sigAB_toA  = sigmoid(phiA0 + .5*w + phiB)
            sigAB_toB  = sigmoid(phiB0 + .5*w + phiA)
            dwA = sigmoid(phiA)*hA + (v - sigAB_toA)*hA
            dwB = sigmoid(phiB)*hB + (v - sigAB_toB)*hB
            #COMBINE
            dw = dw + (dwA + dwB)

        # Sleep phase
        for v in train: # 'free'
            vA = rng.randint(0,2,(1,2))
            vB = rng.randint(0,2,(1,2))
            for t in range(num_iters):
                # visibles to hiddens
                psiA, psiB = np.dot(vA,w), np.dot(vB,w)
                hA_prob, hB_prob = sigmoid(psiA), sigmoid(psiB)
                hA = (hA_prob > rng.random(size=hA_prob.shape)).reshape(2,1)
                hB = (hB_prob > rng.random(size=hB_prob.shape)).reshape(2,1)
                # hiddens to visibles
                phiA, phiB = np.dot(w,hA), np.dot(w,hB)
               ...


I am working on the file "ExploratoryNotebooks/Change_counter.ipynb".
The first task is:
Read data and tree.
Can you write Python code for it?

data = pd.read_csv('../Data/PyronParityData.csv', index_col=0, header=False)

taxa = dendropy.TaxonSet()
mle = dendropy.Tree.get_from_path('../TotalOpt/annotatedTO_0param_2598364.dated', 'newick', taxon_set=taxa, preserve_underscores=True)


I am working on the file "ExploratoryNotebooks/CountedPlots.ipynb".
The first task is:
Pivot the table to group the data by tree.
Can you write Python code for it?

table = pivot_table(data, index=['Tree'], columns=['Parameter'])
table


I am working on the file "ExploratoryNotebooks/CountedPlots.ipynb".
The first task is:
Plot using native pandas plotting.
Can you write Python code for it?

table.plot(kind='bar', width=.7, sort_columns=True).set_ylim(0,75)

plt.tight_layout()
plt.savefig('exTotal.svg', bbox_inches='tight', dpi=300)


I am working on the file "ExploratoryNotebooks/CountedPlots.ipynb".
The first task is:
Read data using pandas.
Can you write Python code for it?

data = pd.read_csv("../Data/Histogram/pared_down.csv")
data

data.columns


I am working on the file "ExploratoryNotebooks/RF_checking.ipynb".
The first task is:
Calculate RF score, pairwise over all trees in sample.
Can you write Python code for it?

n = len(uotrees)
udiffarray = np.zeros((n,n))

for i, ele1 in enumerate(uotrees):
    for j, ele2 in enumerate(uotrees):
        if j >= i:
            break # Since the matrix is symmetrical we don't need to
                  # calculate everything
        difference = ele1.symmetric_difference(ele2)
        udiffarray[i, j] = difference
        udiffarray[j, i] = difference

diffarray

diffarray

o_tl = [tree.length() for tree in otrees]
print o_tl

uo_tl = [mle.length() for mle in uotrees]
print uo_tl


I am working on the file "ExploratoryNotebooks/RF_checking.ipynb".
The first task is:
Calculate Robinson-Foulds distance, and tree length.
Can you write Python code for it?

non_ml = dendropy.Tree.get_from_path('../Trees/MLE/ExaML_result.SquamataPyron.MLE.2b', 'newick', taxon_set = taxa)
print ours.length()
print non_ml.length()
print ours.symmetric_difference(non_ml)

taxa = dendropy.TaxonSet()

pb = dendropy.Tree.get_from_path('../Trees/TotalOptimization/Ranked/2598364', 'nexus', taxon_set=taxa)

rfs = [tree.symmetric_difference(pb_o) for tree in uotrees]

rfs

olist = find_files(top='garli_opt/', filename_filter='*.tre')
print olist

otrees = [dendropy.Tree.get_from_path(filename, "nexus") for filename in olist]


I am working on the file "ExploratoryNotebooks/RF_checking.ipynb".
The first task is:
Read in data.
Can you write Python code for it?

taxa = dendropy.TaxonSet()

ours = dendropy.Tree.get_from_path('../best.phy', 'newick', taxon_set=taxa)


I am working on the file "ExploratoryNotebooks/biopython.ipynb".
The first task is:
List of loci in our dataset that we would like to add for these species.
Can you write Python code for it?


I am working on the file "ExploratoryNotebooks/biopython.ipynb".
The first task is:
Search loop: using Entrez database, pull DNA records matching our organisms and genes.
Can you write Python code for it?

for i in sp:
    for j in ge:
        handle = Entrez.esearch(db="nucleotide", term = "{0}[Orgn] & {1}[Gene]".format(i, j))
        record = Entrez.read(handle)
        handle.close()
        if len(record["IdList"])!=0:
            print record["IdList"]
            results.append(record["IdList"])
print results

print


I am working on the file "ExploratoryNotebooks/heatmap.ipynb".
The first task is:
First, let's get some data.
Can you write Python code for it?

data = pd.read_csv("../Data/Heatmap/TO.csv", index_col = False, header = False)
data.columns = ['A','B','C']

data


I am working on the file "ExploratoryNotebooks/heatmap.ipynb".
The first task is:
If you're like me, generally, you store data one variable per column. This isn't ideal for heatmaps. Matplotlib's heatmap assumes data have one variable down the x-axis of the spreadsheet and one along the top, or y-axis. Pandas can melt our data to be in this format.
Can you write Python code for it?

h_data = data.pivot(index='A', columns='B', values='C')
h_data.ix[0.1,0.1] = .036
h_data

plt.pcolor(h_data,cmap=plt.cm.RdBu,edgecolors='k')
plt.xticks(np.arange(0.5, len(h_data.columns), 1), h_data.columns)
plt.yticks(np.arange(0.5, len(h_data.index), 1), h_data.index)
cbar = plt.colorbar()
plt.tight_layout()
plt.savefig('TO.svg', bbox_inches='tight', dpi=300)


I am working on the file "Explore.ipynb".
The first task is:
Embark Location
Here are the two missing values for embarkment:
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
Exploratory Analysis and Feature Engineering
Here, we'll explore the features of the dataset. Since Sex and PClass are rather clear-cut and have been explored in many other kernels, we will not explore those for now. We'll explore the related features SibSp and Parch together, as a "family size" feature group. Due to the large amount of missing values for Age, we will explore it last - after looking at the other features, we may come up with strategies for imputation.
Finally, we'll create several derived features if necessary.
Names
This doesn't seem like a very promising feature, but take a look:
Can you write Python code for it?