I am working on the file "Strava_Running_Data_Example.ipynb".
The first task is:
We're going to only look at my activities that are of type 'Run'
The data I want to visualize is the total distance I run for each activity
stravalib has a unithelper to do some metric conversion
Can you write Python code for it?

activities = [(activity.start_date, unithelper.kilometers(activity.distance))
              for activity in client.get_activities() if activity.type=='Run']

activities

plt.figure(figsize=(15, 15))
ax = plt.subplot(111)
plt.plot([x[0] for x in activities], [x[1] for x in activities], 'o-', label='runs')
plt.legend(loc=4)
plt.ylabel('Kilometers')
plt.xlabel('Date')
plt.grid()
plt.show()


I am working on the file "Stresses_disk_point_loads.ipynb".
The first task is:
Displacements
Can you write Python code for it?

C1, a, b = symbols("C1 a b")

ux, uy = symbols("ux uy", cls=Function)

eq1 = diff(ux(x), x) - Exx
eq1

from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "Stresses_disk_point_loads.ipynb".
The first task is:
Strains
Can you write Python code for it?

E, nu = symbols("E nu")
G = E/(2*(1 + nu))

Exx = simplify(1/E*(Sxx - nu*Syy))
Eyy = simplify(1/E*(Syy - nu*Sxx))
Exy = simplify(1/G*Sxy)

display(Exx)

display(Eyy)

display(Exy)


I am working on the file "String Comparison.ipynb".
The first task is:
Cosine Similarity
Can you write Python code for it?

import re, math
from collections import Counter


I am working on the file "String Comparison.ipynb".
The first task is:
Cosine works fine with whole words and word transposition but will start to trip up on CO vs COMPANY and when too much extraneous text is introduced.
DIFFLIB (Python Module)
String comparison using difflib - https://docs.python.org/3/library/difflib.html
Can you write Python code for it?

import difflib
from difflib import SequenceMatcher

m = SequenceMatcher(None, text1, text2)
print (m.ratio())


I am working on the file "String Comparison.ipynb".
The first task is:
Fuzzywuzzy has an interesting "process" function:
choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)
Jaccard
Can you write Python code for it?

import numpy as np

# Jaccard Similarity J (A,B) = | Intersection (A,B) | / | Union (A,B) |

def compute_jaccard_similarity_score(x, y):
    intersection_cardinality = len(set(x).intersection(set(y)))
    union_cardinality = len(set(x).union(set(y)))
    return intersection_cardinality / float(union_cardinality)

score = compute_jaccard_similarity_score(text1, text2)

print ("Jaccard Similarity Score: ",score)


I am working on the file "String Comparison.ipynb".
The first task is:
May have some issues where it comes to partial string matches http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/
FuzzyWuzzy (Python Module)
Background on FuzzyWuzzy - http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/
Package install should only need to be done once, unless the cluster was reset - this will install locally (using --user parameter), so variables need to be set
Can you write Python code for it?

## %%sh
## pip install fuzzywuzzy --user


I am working on the file "String Comparison.ipynb".
The first task is:
taken from https://codegists.com/code/python%20jaccard/
Jellyfish
Testing Jellyfish library, with the following algorithms
String comparison:

Levenshtein Distance
Damerau-Levenshtein Distance
Jaro Distance
Jaro-Winkler Distance
Match Rating Approach Comparison
Hamming Distance

Phonetic encoding:

American Soundex
Metaphone
NYSIIS (New York State Identification and Intelligence System)
Match Rating Codex

https://github.com/jamesturk/jellyfish
Can you write Python code for it?

## %%sh
## pip install jellyfish --user

import jellyfish

jellyfish.levenshtein_distance(text1,text2)

jellyfish.damerau_levenshtein_distance(text1,text2)

jellyfish.jaro_distance(text1,text2)

jellyfish.jaro_winkler(text1,text2)

jellyfish.match_rating_comparison(text1,text2)

jellyfish.hamming_distance(text1,text2)

jellyfish.soundex(text1)

jellyfish.soundex(text2)

soundexenc = ''
sentence=text1.split()
for word in sentence:
    soundexenc = soundexenc+' '+jellyfish.soundex(word)
print(soundexenc)

jellyfish.metaphone(text1)

jellyfish.metaphone(text2)

jellyfish.metaphone(text1) == jellyfish.metaphone(text2)

jellyfish.nysiis(text1)

jellyfish.nysiis(text2)


I am working on the file "StringIO.ipynb".
The first task is:
We can also write to it:
Can you write Python code for it?

f.write(' Second line written to file like object')

# Reset cursor just like you would a file
f.seek(0)

# Read again
f.read()


I am working on the file "StringProcessing.ipynb".
The first task is:
Another common scenarios is to check whether or not a string starts with a particular word.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Basically match() only looks for a match at the beginning of the string to be searched. For more information see the
search() vs match() section in the Python documentation.
Finally if you are using the same regular expression many times you may find it advantageous to compile the regular expression. This may speed up your program.
Can you write Python code for it?

cat_regex = re.compile(r"cat")
cat_regex.search("my cat has a hat")


I am working on the file "StringProcessing.ipynb".
The first task is:
If we search for something that exists more than once we get the index of the first instance.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Individual groups can also be accessed. Note that the first group includes everything matched by the regular expression.
Can you write Python code for it?

match.group(0)

match.group(1)

match.group(2)


I am working on the file "StringProcessing.ipynb".
The first task is:
It is also possible to find all matches. However, note that this returns strings as opposed to regular expression match objects.
Can you write Python code for it?

matches = re.findall(r"\wat", some_text)
for m in matches:
    print(m)


I am working on the file "StringProcessing.ipynb".
The first task is:
It is possible to do this only for the left/right hand side of the string.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Let us have a look at a common pitfall when using regular expressions in Python: the difference between the methods
search() and match().
Can you write Python code for it?

re.search(r"cat", "my cat has a hat")

print( re.match(r"cat", "my cat has a hat") )

re.match(r"my", "my cat has a hat")


I am working on the file "StringProcessing.ipynb".
The first task is:
Let us search for particular words within our string.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Let us search for the word "cat".
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Note how horrible and incomprehensible the regular expression is.
It took me a couple of attempts to get this one right as I forgot that | is a regular expression meta character that needs to be escaped using a backslash \.
However, we can now access the groups specified by the parenthesis.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Note that the above returns False as the words have different capitalisation. One can work around this by forcing the input text to be lower case.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Note the extra items at the begining and end of the line arising from the extra white spaces. One can get around this by using the strip() function.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
Now suppose that we wanted the first letter to be any alphanumberic character. We can achieve this using the regular expression "word" meta character \w.
Can you write Python code for it?

match = re.search(r"\wat", some_text)
if match:
    print(match.string[match.start():match.end()])


I am working on the file "StringProcessing.ipynb".
The first task is:
Similarly we can use regular expressions to perform substitutions.
Can you write Python code for it?


I am working on the file "StringProcessing.ipynb".
The first task is:
There are many variaitons on the string operators described above. It is useful to familiarise yourself with the
Python documentation on strings.
Regular expressions
Regular expressions can be defined as a series of characters that define a search pattern.
Regular expressions can be very powerful. However, they can be difficult to build up. Often it is a process of trial and error. This means that once they have been created, and the trial and error process has been forgotten, it can be extremely difficult to understand what the regular expression does and why it is constructed the way it is.
Use regular expressions only as a last resort!
To use regular expressionsions in Python we need to import the re module.
Can you write Python code for it?

import re
some_text = "  Postman Pat has a cat named Jess.  "


I am working on the file "StringProcessing.ipynb".
The first task is:
This function is particularly useful when dealing with csv files.
Can you write Python code for it?


I am working on the file "Strings.ipynb".
The first task is:
String Basics
We can also use a function called len() to check the length of a string!
Can you write Python code for it?


I am working on the file "Strings.ipynb".
The first task is:
String Indexing
We know strings are a sequence, which means Python can use indexes to call parts of the sequence. Let's learn how this works.
In Python, we use brackets [] after an object to call its index. We should also note that indexing starts at 0 for Python. Let's create a new object called s and the walk through a few examples of indexing.
Can you write Python code for it?

# Assign s as a string
s = 'Hello World'

#Check
s

# Print the object
print(s)


I am working on the file "Strings.ipynb".
The first task is:
There are many more methods than the ones covered here. Visit the advanced String section to find out more!
Print Formatting
We can use the .format() method to add formatted objects to printed string statements.
The easiest way to show this is through an example:
Can you write Python code for it?

'Insert another string with curly brackets: {}'.format('The inserted string')


I am working on the file "Strings.ipynb".
The first task is:
We can also use negative indexing to go backwards.
Can you write Python code for it?

# Last letter (one index behind 0 so it loops back around)
s[-1]

# Grab everything but the last letter
s[:-1]


I am working on the file "Strings.ipynb".
The first task is:
We can use a print statement to print a string.
Can you write Python code for it?

print 'Hello World 1'
print 'Hello World 2'
print 'Use \n to print a new line'
print '\n'
print 'See what I mean?'


I am working on the file "Strings.ipynb".
The first task is:
We can use the multiplication symbol to create repetition!
Can you write Python code for it?


I am working on the file "Strips-on-sphere.ipynb".
The first task is:
Instead of applying this rotation successively to each point in the basic strip, we make use of the numpy function einsum (Einstein summation), and rotate simultaneously   all points with coordinates in  two equally shaped meshgrids (x,y, in our case):
Can you write Python code for it?

def rotate_z_mesh(x, y, alpha):
    return np.einsum('ji, mni -> jmn', Rot(alpha), np.dstack([x, y]))


I am working on the file "Strips-on-sphere.ipynb".
The first task is:
The colorscale to plot the sphere's strips:
Can you write Python code for it?

cofee_green=[[0.0,u'#543005'],
[0.1,u'#8c510a'],
[0.2, u'#bf812d'],
[0.3, u'#dfc27d'],
[0.4, u'#f6e8c3'],
[0.5, u'#f5f5f5'],
[0.6, u'#c7eae5'],
[0.7, u'#80cdc1'],
[0.8, u'#35978f'],
[0.9, u'#01665e'],
[1.0, u'#003c30']]

import plotly.plotly as py
import plotly.tools as tls
from plotly.graph_objs import *

def make_trace(x,y,z, colsc):# this function creates the trace for each strip on sphere
    return Surface(
        x=x,
        y=y,
        z=z,
        colorscale=colsc,
        name=''
    )

dt=[make_trace(x,y,z, cofee_green)]#initial strip
for k in range(1,12):
    rez=rotate_z_mesh(x, y, k*np.pi/6)
    dt+=[make_trace(rez[0], rez[1], z, cofee_green)]

data=Data(dt)

axis = dict(
showbackground=True,
backgroundcolor='rgb(20, 20, 20)',
gridcolor='rgb(75, 75, 75)',
zerolinecolor='rgb(75, 75, 75)'
    )

layout = Layout(
         title='Strips on sphere',
         width=800,
         height=800,
         scene=Scene(
         xaxis=XAxis(axis),
         yaxis=YAxis(axis),
         zaxis=ZAxis(axis),
        )
        )

fig = Figure(data=data, layout=layout)
py.sign_in('empet', 'my_api_key')
py.plot(fig, filename='spherical-strips')


I am working on the file "Strips-on-sphere.ipynb".
The first task is:
We generate a sphere from 12 strips. The basic strip is parameterized and discretized below:
Can you write Python code for it?

theta=np.linspace(0, 5*np.pi/36, 30)  #5pi/36=25 degrees
phi=np.linspace(-np.pi/2, np.pi/2, 100)
theta, phi=np.meshgrid(theta, phi)
x=np.cos(theta)*np.cos(phi)
y=np.sin(theta)*np.cos(phi)
z=np.sin(phi)


I am working on the file "Strips-on-sphere.ipynb".
The first task is:
https://plot.ly/~empet/4759
Can you write Python code for it?

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "StructuredSparsity.ipynb".
The first task is:
Create dimacs file from adjacency matrix
Can you write Python code for it?

def convert_W_to_dimacs(W, dimacs_fname):
    num_edges = np.sum(W)
    print "Considering %d edges" % num_edges

    num_features = W.shape[0]

    with open(dimacs_fname, 'w') as g:
        g.write("p max %d %d\n" % (num_features, num_edges))
        for u, v in zip(np.nonzero(W)[0], np.nonzero(W)[1]):
            g.write("a %d %d %d\n" % ((u+1), (v+1), int(W[u, v])))
        g.close()

w_fh, w_fname = tempfile.mkstemp(suffix='network.dimacs')

print w_fname

convert_W_to_dimacs(W, w_fname)


I am working on the file "StructuredSparsity.ipynb".
The first task is:
In the StructuredSparsityDataGeneration.ipynb notebook, I have simulated data with 1000 features and only 150 observations.
The features are binary (as motivated by GWAS data).
I am also assuming a network structure on the features, captured by the adjacency matrix W. The network is built as a collection of fully connected subnetworks of 10 features, each connected to the next. One module is causal, meaning that the outcome y of interest is generated as a linear combination of the features of that module.
Can you write Python code for it?

num_feats = 1000
num_obsvs = 150

mod_size = 10

num_causl = 10


I am working on the file "StructuredSparsity.ipynb".
The first task is:
Load generated data
Can you write Python code for it?

data_rep = 'data/struct_spars'
X_fname = '%s/X.data' % data_rep
y_fname = '%s/y.data' % data_rep
W_fname = '%s/W.data' % data_rep
causl_fname = '%s/causl.data' % data_rep
wghts_fname = '%s/w_causl.data' % data_rep

X = np.loadtxt(X_fname, dtype='int')
y = np.loadtxt(y_fname)
W = np.loadtxt(W_fname)
causl = list(np.loadtxt(causl_fname, dtype='int'))
w_causl = np.loadtxt(wghts_fname)


I am working on the file "StructuredSparsity.ipynb".
The first task is:
New variables
Can you write Python code for it?

S = np.dot(evecs, np.diag(evals))

y_new = np.hstack((y_train, np.zeros((num_feats, ))))

l1 = 0.001
l2 = 10.

gamma = l1/(np.sqrt(l2+1))

X_new = 1/(np.sqrt(l2+1)) * np.vstack((X_train, np.sqrt(l2)*S.T))


I am working on the file "StructuredSparsity.ipynb".
The first task is:
Once the correct features are selected, the linear regression can make good predictions.
Clean up
Can you write Python code for it?

import os
os.remove(c_fname)
os.remove(w_fname)


I am working on the file "StructuredSparsity.ipynb".
The first task is:
Plot selected features
Can you write Python code for it?

cd ..

non_selected = [ix for ix in range(num_feats) if ix not in selected]

fig = plt.figure(figsize(12, 8))

plt.scatter(selected, # x = SNP position
            np.ones(shape=(len(selected), 1)),
           color=def_colors[0], s=200)

plt.scatter(non_selected,
            np.zeros(shape=(len(non_selected), 1)),
           color=def_colors[0], s=200)

plt.scatter(causl,
            np.ones(shape=(len(causl), )),
            color=def_colors[1], s=200)

plt.xlabel("features", fontsize=28)
plt.ylabel("sfan selection", fontsize=28)
plt.xlim([0, num_feats])

plt.savefig('structured_sparsity/sfan_selected.png', bbox_inches='tight')


I am working on the file "StructuredSparsity.ipynb".
The first task is:
Run sfan
Can you write Python code for it?

lbd = 5.
eps = 1.

vv = !python multitask_sfan.py -k 1 -w $w_fname -r $c_fname -l $lbd -e $eps
print vv


I am working on the file "StructuredSparsity.ipynb".
The first task is:
T-test
Can you write Python code for it?

import statsmodels.api as sm

pvalues = []
for feat_idx in range(num_feats):
    myX = X_train[:, feat_idx]
    myX = sm.add_constant(myX)
    est = sm.regression.linear_model.OLS(y_train, myX)
    est2 = est.fit()
    pvalues.append(est2.pvalues[1])
pvalues = np.array(pvalues)

fig = plt.figure(figsize(12, 8))

plt.scatter(range(num_feats), # x = SNP position
            -np.log10(pvalues), # y = -log10 p-value
            s=200)

# Plot the causal SNPs in red
plt.scatter(causl,
            -np.log10(pvalues[causl]),
            color=def_colors[1], s=200)

# significance threshold according to Bonferroni correction
t = -np.log10(0.05/num_feats)
plt.plot([0, num_feats], [t, t], lw=4, c='w')

plt.xlabel("feature", fontsize=28)
plt.ylabel("-log10 p-value", fontsize=28)
plt.xlim([0, num_feats])

plt.savefig('structured_sparsity/t_test.png', bbox_inches='tight')


I am working on the file "StructuredSparsity.ipynb".
The first task is:
The predictivity has slightly improved.
Network-Connected Lasso
This method is developed in https://academic.oup.com/bioinformatics/article/24/9/1175/206444
As detailed in the paper, this can be reformulated so as to be solved as a Lasso problem.
Laplacian
Can you write Python code for it?

degree = np.sum(W, axis=0)
L = np.diag(degree) - W

# spectral decomposition
evals, evecs = np.linalg.eigh(L)

# correct for numerical errors:
# eigenvalues of 0 might be computed as small negative numbers
evals = np.maximum(0, evals)


I am working on the file "StructuredSparsity.ipynb".
The first task is:
The solution is sparse, but many non-causal features have non-zero weights and several causal features have a weight of 0.
If you increase lambda, you'll lose  more causal features.
Validation
Can you write Python code for it?

y_pred = model.predict(X_test)

rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))

fig = plt.figure(figsize(12, 12))

plt.scatter(y_test, y_pred, s=200)

plt.xlabel("true y", fontsize=28)
plt.ylabel("prediction", fontsize=28)
plt.xlim([np.min(y_test)-0.05, np.max(y_test)+0.05])
plt.ylim([np.min(y_test)-0.05, np.max(y_test)+0.05])

plt.text(0, 0.6, 'RMSE = %.2f' % rmse, fontsize=28)

plt.savefig('structured_sparsity/lasso_pred.png', bbox_inches='tight')


I am working on the file "StructuredSparsity.ipynb".
The first task is:
True model
Can you write Python code for it?

fig = plt.figure(figsize(12, 8))

plt.scatter(range(num_feats), # x = SNP position
            np.zeros(shape=(num_feats, )), # weight = 0
            s=200) # marker size

# Plot the causal SNPs in red
plt.scatter(causl, w_causl, s=200)

plt.xlabel("feature", fontsize=28)
plt.ylabel("true feature weight", fontsize=28)
plt.xlim([0, num_feats])

plt.savefig('structured_sparsity/causl_weight.png', bbox_inches='tight')

y_pred = np.dot(X_test[:, causl], w_causl)
print y_pred.shape

from sklearn import metrics

rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))

fig = plt.figure(figsize(6, 6))

plt.scatter(y_test, y_pred)

plt.xlabel("y = Xw", fontsize=18)
plt.ylabel("y = Xw + noise", fontsize=18)
plt.xlim([np.min(y_test)-0.05, np.max(y_test)+0.05])
plt.ylim([np.min(y_test)-0.05, np.max(y_test)+0.05])

plt.text(0, 0.6, 'RMSE = %.2f' % rmse)

plt.savefig('structured_sparsity/true_pred.png', bbox_inches='tight')


I am working on the file "StructuredSparsity.ipynb".
The first task is:
ncLasso
Can you write Python code for it?

model = linear_model.Lasso(fit_intercept=True, alpha=gamma)
model.fit(X_new, y_new)

fig = plt.figure(figsize(12, 8))

plt.scatter(range(num_feats), # x = SNP position
            model.coef_, # y = -log10 p-value (the higher the more significant)
            s=200)

# Plot the causal SNPs in red
plt.scatter(causl, model.coef_[causl],
            color=def_colors[1], s=200)

plt.xlabel("features", fontsize=28)
plt.ylabel("ncLasso weight", fontsize=28)
plt.xlim([0, num_feats])

plt.savefig('structured_sparsity/nclasso_weights.png', bbox_inches='tight')


I am working on the file "StructuredSparsity.ipynb".
The first task is:
sfan selected exactly the right features!
Validation
Create predictive model from the selected features only
Can you write Python code for it?

model = linear_model.LinearRegression(fit_intercept=True)
model.fit(X_train[:, selected], y_train)

y_pred = model.predict(X_test[:, selected])

rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))

fig = plt.figure(figsize(12, 12))

plt.scatter(y_test, y_pred, s=200)

plt.xlabel("true y", fontsize=28)
plt.ylabel("prediction", fontsize=28)
plt.xlim([np.min(y_test)-0.05, np.max(y_test)+0.05])
plt.ylim([np.min(y_test)-0.05, np.max(y_test)+0.05])

plt.text(0, 0.6, 'RMSE = %.2f' % rmse, fontsize=28)

plt.savefig('structured_sparsity/sfan_pred.png', bbox_inches='tight')


I am working on the file "StructuredSparsityDataGeneration.ipynb".
The first task is:
Generate data
Can you write Python code for it?

# adjacency matrix
W = np.zeros((num_feats, num_feats))
for i in range(num_feats/mod_size):
    W[i*mod_size:(i+1)*mod_size, i*mod_size:(i+1)*mod_size] = np.ones((mod_size, mod_size))
    if not i == (num_feats/mod_size - 1):
        W[(i+1)*mod_size-1, (i+1)*mod_size] = 1
        W[(i+1)*mod_size, (i+1)*mod_size-1] = 1

# remove the diagonal
W = W - np.eye(num_feats)

# SNPs
X = np.random.binomial(1, 0.1, size=(num_obsvs, num_feats))

# Phenotype
w_causl = np.random.normal(loc=0.2, scale=0.05, size=(num_causl))
print w_causl

w = np.zeros((num_feats, ))
w[:num_causl] = w_causl

y = np.dot(X, w) + np.random.normal(loc=0., scale=0.1, size=(num_obsvs, ))


I am working on the file "StructuredSparsityDataGeneration.ipynb".
The first task is:
Save generated data
The data used in the StructuredSparsity.ipnb notebook is saved under data/struct_spars. Here it will be generated under data/my_struct_spars.
Can you write Python code for it?

data_rep = 'data/my_struct_spars'
X_fname = '%s/X.data' % data_rep
y_fname = '%s/y.data' % data_rep
W_fname = '%s/W.data' % data_rep
causl_fname = '%s/causl.data' % data_rep
wghts_fname = '%s/w_causl.data' % data_rep

np.savetxt(X_fname, X, fmt='%d')
np.savetxt(y_fname, y)
np.savetxt(W_fname, W, fmt='%.1f')
np.savetxt(causl_fname, causl, fmt='%d')
np.savetxt(wghts_fname, w_causl)


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
A more detailed view of correlation: Heatmap visualization

The stronger the correlation, the deeper the shade of purple, really just confirming our calcuations previously
Can you write Python code for it?


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Categorical Feature Analysis - Parent Satisfation & Relation [Separated by Class: H, M, L]

Now we want to pair those 'High, Low and Middle'grade value counts with other features that may indicate patterns in the distribution of grades and add value to our analysis.

Below are 4 bar graphs for which grades students received based on Parent satisfaction (good or bad) and Relation responsible for the student (Father or Mum)
There is a clear pattern where:
High grade = Parent schoool satisfaction (good) & Relation responsible (Mum)
Low grade = Parent schoool satisfaction (bad) & Relation responsible (Father)
Can you write Python code for it?

import matplotlib.pyplot as plt
fig = plt.figure(figsize=(18,4))

plt.subplot(141)
good_sat= data.Class[data.ParentschoolSatisfaction== 'Good'].value_counts()
good_sat.plot(kind='bar')
plt.title('Parent school satisfaction = Good')

plt.subplot(142)
bad_sat = data.Class[data.ParentschoolSatisfaction== 'Bad'].value_counts()
bad_sat.plot(kind='bar')
plt.title('Parent school satisfaction = Bad')

plt.subplot(143)
survey_no = data.Class[data.Relation == 'Father'].value_counts()
survey_no.plot(kind='bar')
plt.title('Father Responsible for Student')

plt.subplot(144)
survey_yes = data.Class[data.Relation == 'Mum'].value_counts()
survey_yes.plot(kind='bar')
plt.title('Mother Responsible for Student')


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Categorical Feature Analysis Cont.

Another good feature that may add value to our analysis of factors that contributes to a student's class is their attendance record.
Again, looking at value counts for the separate classes, we see that the number of students who had under-7 absences and received a 'Low' class/grade is very low. (and vice versa for above-7 absences & 'H')
Can you write Python code for it?

import matplotlib.pyplot as plt

attendance = pd.crosstab(index=data['StudentAbsenceDays'], columns=[data['Class']], normalize='columns')

attendance.plot(kind='bar', figsize=(6,6), stacked=True)


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Data Visualization & Exploratory Analysis

Libraries used: Matplotlib & Seaborn

First step in exploring the data to classify the grades of students is to look at a simple value count.
Next we visualize the number of students in each of the separate classes to get an idea of how evenly spread the 'H, L, and M's are among the students in the dataset.
As the bar graph shows, distribution is fairly even, no major skews toward any individual class.
Can you write Python code for it?

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

data.Class.value_counts().plot(kind='bar')

data.Class.value_counts()


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Exploring correlations between multidimensional data & plotting all pairs of values against each other

We see the relationship between class is strongest between 'raisedhands' and other features as well as 'visitedresources' and other features
The 2 weaker features seem to be 'announcements view' and 'discussion' where the points for the different 'H, M, L' classes seem to be scattered all over with no particular pattern.
Now that we have this analysis, we have a better understanding of which features will help us predict the target classes we want and which will be added noise. This is an importance part of the process of feature analysis.
Can you write Python code for it?

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

sns.pairplot(data, hue='Class', size=2.5);


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Gradient Boosting Classifier

Gradient Boosting is based on the the idea of boosting, which is a method of trying to modify a weak learner into a becoming a better one. It starts with filtering observations, leaving those observations that the weak learner can handle and focusing on developing new weak learns to handle the remaining difficult observations. For example, the model will build trees one at a time, where each new tree helps to correct errors made by previously trained tree.
With Gradient Boosting, the objective is to minimize the loss of the model by adding weak learners using a gradient descent like procedure. This type of algorithm can be described as a stage-wise additive model. This is because one new weak learner is added at a time and existing weak learners in the model are frozen and left unchanged.
Pros:

Can easily handle qualitative (categorical) features
Very powerful and performs well in most cases

Cons:

Training generally takes longer because of the fact that trees are built sequentially
Harder to fit/tune parameters
Can you write Python code for it?

# Gradient Boosting Classifer: Ensemble method No.2

feature_cols = ['raisedhands', 'VisITedResources', 'ParentschoolSatisfaction_Bad',
                'ParentschoolSatisfaction_Good', 'StudentAbsenceDays_Above-7', 'Relation_Father',
                               'Relation_Mum',
                'StudentAbsenceDays_Under-7']

X = data_dummies[feature_cols]
y = target_value

# train/test split
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.15)

from sklearn.preprocessing import StandardScaler
scl = StandardScaler()
scl.fit_transform(X_train, y_train)
scl.transform(X_test)

from sklearn.ensemble import GradientBoostingClassifier

gbc = GradientBoostingClassifier(learning_rate = 0.15, n_estimators = 300, max_depth = 8, min_samples_leaf = 3,
                                 max_features = 'log2')
gbc.fit(X_train,y_train)

y_pred = gbc.predict(X_test)

from sklearn.metrics import accuracy_score
print ('Prediction Accuracy:', accuracy_score(y_test, y_pred))

from sklearn.cross_validation import cross_val_score
scores = cross_val_score(estimator = pipe_lr, X= X_train, y = y_train, cv = 10, n_jobs =1)
print ('Cross-validated Scores: %s' %scores)
print("CV Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

from sklearn.metrics import classification_report
print('Scores', classification_report(y_test, y_pred))


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Here is a visualization of the above pivot table. The combination of two features do not vary too much within the 'H, M, L' classes, but there is a distict gap showing that students in 'H' class raised their hands at an average of around 3 times as often as 'L' students.
Can you write Python code for it?

parents_hands = data.pivot_table('raisedhands', index = ['ParentschoolSatisfaction','Relation'],
                 columns = 'Class', aggfunc = 'mean')

parents_hands.plot()
plt.ylabel('Average number of times student raised hand')


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Logistic Regression Classifer

The logistic regression classifier is a method used to generalizes logistic regression to multiclass problems (applicable here since we are trying to predict more than two possible discrete outcomes). It is a model that is used to predict the probabilities of the different possible outcomes of a categorically distributed dependent variable
Dependent variable (y values) =  Class ('H, L, M')
 Independent variables (or features / x values) =  'raisedhands', 'VisITedResources', 'ParentschoolSatisfaction_Bad',  'Relation_Father', Relation_Mum','ParentschoolSatisfaction_Good', 'StudentAbsenceDays_Above-7','StudentAbsenceDays_Under-7'
In Logistic Regression modeling, input values (X) are combined linearly using weights or coefficient values to predict an output value (y)--- or more specifically, the probability that an input (X) belongs to the default class.
Pros:

Low variance
Provides probabilities for outcomes
works well with diagonal (feature) decision boundaries

Cons:

Doesn’t perform well when feature space is too large
High bias
Relies on entire data
Can you write Python code for it?

feature_cols = feature_cols = ['raisedhands', 'VisITedResources',  'ParentschoolSatisfaction_Bad',  'Relation_Father',
                               'Relation_Mum','ParentschoolSatisfaction_Good', 'StudentAbsenceDays_Above-7',
                                'StudentAbsenceDays_Under-7']

X = data_dummies[feature_cols]
y = target_value

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.15)

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

pipe_lr = Pipeline([('rs', StandardScaler()), ('pca', PCA(n_components = 4)),
                    ('logreg', LogisticRegression(C=1e9))])

pipe_lr.fit(X_train, y_train)

y_pred = pipe_lr.predict(X_test)

from sklearn.metrics import accuracy_score
print ('Prediction Accuracy:', accuracy_score(y_test, y_pred))

from sklearn.cross_validation import cross_val_score
scores = cross_val_score(estimator = pipe_lr, X= X_train, y = y_train, cv = 10, n_jobs =1)
print ('Cross-validated Scores: %s' %scores)
print("CV Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

from sklearn.metrics import classification_report
print(' ', classification_report(y_test, y_pred))


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Moving on to numerical feature analysis:

Look at the distribution and probability density function of the 4 numerical columns:
1) raised hands
2) Visited Resources
3) Announcements Views
4) Discussion
The graphs for 'raised hands' and 'visited resources' show the most promise in differentiating between 'H' and 'L' classes. The bimodal shape of their density curves show distinct peaks at the opposite ends of the graph.
Can you write Python code for it?

fig = plt.figure(figsize=(18,8))

plt.subplot(221)
data.raisedhands[data.Class == 'H'].plot(kind='kde')
data.raisedhands[data.Class == 'M'].plot(kind='kde')
data.raisedhands[data.Class == 'L'].plot(kind='kde')
plt.legend(('High', 'Middle','Low'),loc='best')
plt.title('Raised Hands')

plt.subplot(222)
data.VisITedResources[data.Class == 'H'].plot(kind='kde')
data.VisITedResources[data.Class == 'M'].plot(kind='kde')
data.VisITedResources[data.Class == 'L'].plot(kind='kde')
plt.legend(('High', 'Middle','Low'),loc='best')
plt.title('Visited Resources')

plt.subplot(223)
data.Discussion[data.Class == 'H'].plot(kind='kde')
data.Discussion[data.Class == 'M'].plot(kind='kde')
data.Discussion[data.Class == 'L'].plot(kind='kde')
plt.legend(('High', 'Middle','Low'),loc='best')
plt.title('Discussion')

plt.subplot(224)
data.AnnouncementsView[data.Class == 'H'].plot(kind='kde')
data.AnnouncementsView[data.Class == 'M'].plot(kind='kde')
data.AnnouncementsView[data.Class == 'L'].plot(kind='kde')
plt.legend(('High', 'Middle','Low'),loc='best')
plt.title('Viewed Announcements')


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Numerical Feature Analysis: Relationship & Correlation

Correlations can tell us about the direction, and the degree (strength) of the relationship between two variables (or features)
Using Pearson's correlation coefficient (Pearson's r) where r ranges from -1 (negative relationship) to 0 (no relationship to 1 (positive relationship).
We see that the only relatively strong relationship r = 0.69 is between the two strongest indications of class differences as noted above: 'raised hands' and 'visited resources'.
Can you write Python code for it?

raised_hands = data['raisedhands']
discussion = data['Discussion']
v_resources = data['VisITedResources']
v_announcements = data['AnnouncementsView']

def correlation(x,y):
    std_x = (x-x.mean())/x.std(ddof=0)
    std_y = (y-y.mean())/y.std(ddof=0)

    return (std_x * std_y).mean()

print ('Raised hands & Discussion: ', correlation(raised_hands, discussion))

print ('Visted Resources & Discussion: ', correlation(v_resources, discussion))

print ('Raised hands & Visited Resources: ', correlation(raised_hands, v_resources))


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Once again, by using the pandas groupby function, we can get a table showing the exact count of the distribution of students in H, L, M --- looking at the first feature "Parent School Satisfaction" only.
Can you write Python code for it?

grades_count = data.groupby(['ParentschoolSatisfaction','Class'])['Class'].aggregate('count').unstack()

grades_count


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Preprocessing Data:  Dummy/One-hot Encoding to transform the features values (x)

A common alternative approach to encoding categorical values is called dummy (or one-hot) encoding where the the basic strategy is to convert each category value into a new column and assigns a 1 or 0 (True/False) value to the column
Pandas supports this feature using get_dummies to create dummy/indicator variables (aka 1 or 0).
Can you write Python code for it?

data_dummies = pd.get_dummies(new_data, columns = ['ParentschoolSatisfaction', 'StudentAbsenceDays','Relation'])

print(data_dummies.shape)

data_dummies.head()


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Preprocessing Data:  Label Encoding Categorical Values to transform the prediction target (y)

Libraries used: Scikit-Learn
In Sklearn, machine learning algorithms require the input variables from the data to be real values. Therefore we must format (or transform) the data into the structure that allows us to feed it into the model.
Here, we use a label encoder to prepare the data where numerical dimensions are used to represent membership in the categories ('H, L, M')
As we can see, once encoded:

|Categorical Class | Numerical value|
|-----------------:| -------------- |
|H (High grade)    | 0              |
|L (Low grade)     | 1              |
|M (Middle grade)  | 2              |
Can you write Python code for it?

#Preprocessing data to encode categorical values for the y-target column

from sklearn import preprocessing
le = preprocessing.LabelEncoder()

target_value = le.fit_transform(new_data.Class)

print (target_value)


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Random Forest Classifier

A random forest is an ensemble method (combination of learning algorithms) that fits a number of decision tree classifiers on various sub-samples of the dataset and use averaging (majority of votes to make a prediction) to improve the predictive accuracy and control over-fitting.
At the root of random forest are decision trees, which are a type of flowchart which assist in the decision making process. Internal nodes represent tests on particular attributes, while branches exiting nodes represent a single test outcome, and leaf nodes represent class labels. The goal is to split on the attributes which create the purest child nodes possible.
Random forests are a way of averaging multiple deep decision trees, trained on different parts of the same training set, with the goal of reducing the variance. This comes at the expense of a small increase in the bias and some loss of interpretability, but generally greatly boosts the performance in the final model.
Pros:

Accurate and does not tend to overfit
Robust against outliers in the predictive variables
It gives estimates of what variables are important in the classification--> feature selection

Cons:

Not as easy to visually interpret
Slower runtime
Can you write Python code for it?

# RANDOM FOREST CLASSIFIER model -- ensemble method No.1

feature_cols = ['raisedhands', 'VisITedResources', 'ParentschoolSatisfaction_Bad',
                'ParentschoolSatisfaction_Good', 'StudentAbsenceDays_Above-7', 'Relation_Father',
                               'Relation_Mum',
                'StudentAbsenceDays_Under-7']

X = data_dummies[feature_cols]
y = target_value

# train/test split
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.15)

from sklearn.preprocessing import StandardScaler
scl = StandardScaler()
scl.fit_transform(X_train, y_train)
scl.transform(X_test)

from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(max_depth=3, n_estimators=200, oob_score = True, n_jobs = -1,
                                                   random_state=50)
rfc.fit(X_train, y_train)

y_pred = rfc.predict(X_test)
from sklearn.metrics import accuracy_score
print ('Prediction Accuracy:', accuracy_score(y_test, y_pred))

from sklearn.cross_validation import cross_val_score
scores = cross_val_score(estimator = pipe_lr, X= X_train, y = y_train, cv = 10, n_jobs =1)
print ('Cross-validated Scores: %s' %scores)
print("CV Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

from sklearn.metrics import classification_report
print('Scores', classification_report(y_test, y_pred))


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Scatterplot Visualization: Class ['H, M, L'] according to 'raisedhands' & 'visitedresources'
Zoom in on main features of importance:

To get a basic understanding of what we are trying to predict, we narrow down the analysis to the two main features that have thus far provided the most information in determining 'H, M, L' classes.
First we need a procedure to turn the class values (categorical) into numerical values in order to plot them and show the different points which are 'H'(large green points) or 'M' (medium yellow points) or 'L' (small red points).
Can you write Python code for it?

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

def new_grade(grade):
    if grade == 'H':
        return 100
    elif grade == 'M':
        return 50
    elif grade == 'L':
        return 10

def new_grades(grades):
    return grades.apply(new_grade)

print (new_grades(data['Class']).head())

converted_grades = new_grades(data['Class'])

fig = plt.figure(figsize=(12, 9))
plt.scatter(data['VisITedResources'], data['raisedhands'],c= converted_grades, s = converted_grades, cmap = 'RdYlGn')
plt.ylabel("Number of times Student Raised their hand in class")
plt.xlabel("Number of times Student Visited Resources")
plt.title ('Interaction Correlation by Class Marks')
plt.colorbar()


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Step #1 - Exploring/Cleaning the data

Summary of Statistics:

Use Pandas' describe function to generate descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution (for all numeric columns).
By looking at info: We have the total number of observations = 480, and we see there are no missing (non-null) values  (i.e., we have a clean data set).
Can you write Python code for it?


I am working on the file "Student Grades Classification Project.ipynb".
The first task is:
Visualization of Learning Curves: Test & Training Data

A learning curve shows the validation and training score of an estimator for varying numbers of training samples. It is a tool to find out how much we benefit from adding more training data and whether the estimator suffers more from a variance error or a bias error.
If both the validation score and the training score converge to a value that is too low with increasing size of the training set, we will not benefit much from more training data.

We can use the sklearn learning_curve function to generate the values that are required to plot such a learning curve (number of samples that have been used, the average scores on the training sets and the average scores on the validation sets)

Learning curve allows us to verify when a model has learning as much as it can about the data, indicated by: a) the performances on the training and testing sets reach a plateau and b) here is a consistent gap between the two error rates, as is consistent with our graph below.
Our Learning Curves show a decent model because a) the testing and training learning curves converge at similar values and b) the smaller the gap between curves, the better our model generalizes. The results of our graph represent moderate bias and low variance, which is an indication that we should increase model complexity--leading us to the ensemble methods we'll be using moving forward.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit

def plot_learning_curve(estimator, X, y, ylim=None, cv=None,
                        n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5)):
    plt.figure()
    plt.xlabel("Training examples")
    plt.ylabel("Score")
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    plt.grid()

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Testing score")

    plt.legend(loc="best")
    return plt

cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)
estimator = svc
plot_learning_curve(estimator, X, y, (0.7, 1.01), cv=cv, n_jobs=4)


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
20% test data and 80% train data
Can you write Python code for it?

# split into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print('Train: ', X_train.shape)
print('Test: ', X_test.shape)


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Build the model using 80% of the data
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_train_dtm, y_train)


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Confusion matrix:
Can you write Python code for it?


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Evaluate the most positive and most negative reviews on Giraffes
Can you write Python code for it?

giraffe_vect_dtm = vect.transform(giraffe_reviews['review'])

giraffe_vect_dtm.shape

giraffe_reviews['predicted_sentiment'] = logreg.predict(giraffe_vect_dtm)

giraffe_reviews['predicted_sentiment_prob'] = logreg.predict_proba(giraffe_vect_dtm)[:, 1]

giraffe_reviews.head()

giraffe_reviews = giraffe_reviews.sort_values('predicted_sentiment', ascending=False)


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Loading the data
Can you write Python code for it?

products = pd.read_csv('amazon_baby.csv')

products.head()

products.count()

products.shape

def cleanNaN(value):
    if pd.isnull(value):
        return ""
    else:
        return value


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Metrics - Accuracy, ROC AUC, Confusion Matrix
Can you write Python code for it?

# calculate accuracy and AUC
from sklearn import metrics


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Most bought product
Can you write Python code for it?

products['name'].value_counts()

giraffe_reviews = products[products['name'] == 'Vulli Sophie the Giraffe Teether']

len(giraffe_reviews)

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns

sns.distplot(giraffe_reviews['rating'])

#giraffe_reviews['rating'].plot(y='rating', orientation='horizontal', kind='hist', bins=5)


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Most of the reviews (68%) are rated 5
Can you write Python code for it?


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Predict using test data (20%)
Can you write Python code for it?

# class predictions and predicted probabilities
y_pred_class = logreg.predict(X_test_dtm)
y_pred_prob = logreg.predict_proba(X_test_dtm)[:, 1]


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
ROC AUC:
Can you write Python code for it?

roc_auc = metrics.roc_auc_score(y_test, y_pred_prob)
print(roc_auc)

roccurve = metrics.roc_curve(y_test, y_pred_prob)


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Split dataset
Can you write Python code for it?

X = products['review']
y = products['sentiment']


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
This data is imbalanced as we can see that 84% percent of the products tend to have Positive sentiment, i.e., rates equal or above 4.
Can you write Python code for it?


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Top 5 Negative reviews
Can you write Python code for it?


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Top 5 Positive reviews
Can you write Python code for it?


I am working on the file "Study Sentiment Analysis - Babies Products.ipynb".
The first task is:
Word Vector and document-term matrix
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer()
# learn training data vocabulary, then create document-term matrix
vect.fit(X_train)

#vocabulary contains the word count vector with all the words and how much they appear per review
vect.vocabulary_["great"]


I am working on the file "Study01.ipynb".
The first task is:
GeoTransが必要であることが分かりましたので、GeoTransのマニュアルも確認します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
Matrix-Marketフォーマットで出力してみます。
Can you write Python code for it?

K.save('mm', "K.mtx")
M.save('mm', "M.mtx")


I am working on the file "Study01.ipynb".
The first task is:
np.arrayについて詳細を確認したい場合は次のようにコマンドを入力します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
ここではMatrix Market形式(http://math.nist.gov/MatrixMarket/formats.html)
で書き出しを行ってみます。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
ここまでのまとめ

メッシュはGetfem++のMeshオブジェクトにより作成することができる。このメッシュはVTKファイルに出力しParaviewで確認することが可能である。
Meshオブジェクトに有限要素オブジェクトと積分法オブジェクトを割り当てることにより、それぞれMeshFemオブジェクトとMeshImオブジェクトを作成することができる。
MeshFemオブジェクトとMeshImオブジェクトから有限要素法により離散化した行列を得ることができる。この行列はSpmatオブジェクトであるが、ファイル出力をしてScipyで読み込むことで各種操作が可能である。

ギャラリー
今回、未紹介のもので出力が面白かったものをいくつか掲載しました。
電磁ポテンシャル
Can you write Python code for it?

"""
Demonstrates computation of gradient with matplotlib.tri.CubicTriInterpolator.
"""
from matplotlib.tri import Triangulation, UniformTriRefiner,\
    CubicTriInterpolator
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy as np
import math
%matplotlib inline

#-----------------------------------------------------------------------------
# Electrical potential of a dipole
#-----------------------------------------------------------------------------
def dipole_potential(x, y):
    """ The electric dipole potential V """
    r_sq = x**2 + y**2
    theta = np.arctan2(y, x)
    z = np.cos(theta)/r_sq
    return (np.max(z) - z) / (np.max(z) - np.min(z))

#-----------------------------------------------------------------------------
# Creating a Triangulation
#-----------------------------------------------------------------------------
# First create the x and y coordinates of the points.
n_angles = 30
n_radii = 10
min_radius = 0.2
radii = np.linspace(min_radius, 0.95, n_radii)

angles = np.linspace(0, 2*math.pi, n_angles, endpoint=False)
angles = np.repeat(angles[..., np.newaxis], n_radii, axis=1)
angles[:, 1::2] += math.pi/n_angles

x = (radii*np.cos(angles)).flatten()
y = (radii*np.sin(angles)).flatten()
V = dipole_potential(x, y)

# Create the Triangulation; no triangles specified so Delaunay triangulation
# created.
triang = Triangulation(x, y)

# Mask off unwanted triangles.
xmid = x[triang.triangles].mean(axis=1)
ymid = y[triang.triangles].mean(axis=1)
mask = np.where(xmid*xmid + ymid*ymid < min_radius*min_radius, 1, 0)
triang.set_mask(mask)

#-----------------------------------------------------------------------------
# Refine data - interpolates the electrical potential V
#-----------------------------------------------------------------------------
refiner = UniformTriRefiner(triang)
tri_refi, z_test_refi = refiner.refine_field(V, subdiv=3)

#-----------------------------------------------------------------------------
# Computes the electrical field (Ex, Ey) as gradient of...


I am working on the file "Study01.ipynb".
The first task is:
ここまでのまとめ

疎行列の計算にはScipyのsparseモジュールが便利である。
Scipyのioモジュールを使用すると、各種形式で行列の入出力が可能である。

Getfem++の学習
Can you write Python code for it?

import getfem as gf
import numpy as np
from scipy import io


I am working on the file "Study01.ipynb".
The first task is:
このようにScipyの疎行列に容易に変換できます。そのため、Scipyを使用した静解析や固有値解析を行うことにより有限要素法の解析が可能となります。
さて１次元メッシュで次のような階がnp.arrayオブジェクトで得られたとします。
Can you write Python code for it?

outou = np.array([1.0, 2.0, 3.0])
print outou


I am working on the file "Study01.ipynb".
The first task is:
これで、これらのパッケージはそれぞれ、np/sp/gf/pltとしてインポートされました。
各パッケージの機能は<module>.の後にタブキーを打つことで実行可能な関数などが表示されます。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
これをParaviewで確認すると4要素の方眼メッシュが作成されていることが分かります。
次に、先ほどの１次元のメッシュを使用して剛性マトリックスと質量マトリックスを作成してみます。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
これをベクトル情報付きのVTKファイルとして出力します。その際にはGetfem++のSliceオブジェクトを使用します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
ご覧のようにこの時点ではメッシュは空です。このメッシュに節点と要素を追加していきます。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
ちなみにこのMeshオブジェクトはVTKファイルにExportできます。
1次元のメッシュでは寂しいので、２次元の方眼メッシュを作成しましょう。
Can you write Python code for it?

m2 = gf.Mesh('cartesian', np.array([0.0, 1.0, 2.0]), np.array([0.0, 1.0, 2.0]))

m2.export_to_vtk?

m2.export_to_vtk('m2.vtk','ascii')


I am working on the file "Study01.ipynb".
The first task is:
また、これらの疎行列はファイルへの保存と読み込みが可能です。その際にはScipyのioモジュールを使用します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
また、固有値解析も少し複雑ですが以下のように計算できます。
Can you write Python code for it?

X = sp.rand(A.shape[0], 3)
linalg.lobpcg(A,X)


I am working on the file "Study01.ipynb".
The first task is:
また、既存のファイルを読み込んで新しく行列を作成することも可能です。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
もちろん、構造用のNavierの式の離散化だけではありません。laplacianやhelmholtzなどの関数があります。
つまり、１つのメッシュでさまざまな微分方程式の離散化が可能ということです。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
アニメーション出力
Can you write Python code for it?

from tempfile import NamedTemporaryFile
from IPython.display import HTML

VIDEO_TAG = """<video controls>
 <source src="data:video/x-m4v;base64,{0}" type="video/mp4">
 Your browser does not support the video tag.
</video>"""

def anim_to_html(anim):
    if not hasattr(anim, '_encoded_video'):
        with NamedTemporaryFile(suffix='.mp4') as f:
            anim.save(f.name, fps=20, extra_args=['-vcodec', 'libx264'])
            video = open(f.name, "rb").read()
        anim._encoded_video = video.encode("base64")

    return VIDEO_TAG.format(anim._encoded_video)

def display_animation(anim):
    plt.close(anim._fig)
    return HTML(anim_to_html(anim))

import pylab as plt
import numpy as np
from matplotlib import animation

# First set up the figure, the axis, and the plot element we want to animate
fig = plt.figure()
ax = plt.axes(xlim=(0, 2), ylim=(-2, 2))
line, = ax.plot([], [], lw=2)

# initialization function: plot the background of each frame
def init():
    line.set_data([], [])
    return line,

# animation function.  This is called sequentially
def animate(i):
    x = np.linspace(0, 2, 1000)
    y = np.sin(2 * np.pi * (x - 0.01 * i))
    line.set_data(x, y)
    return line,

# call the animator.  blit=True means only re-draw the parts that have changed.
anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=100, interval=20, blit=True)

# call our new function to display the animation
display_animation(anim)


I am working on the file "Study01.ipynb".
The first task is:
スライスオブジェクトを作成する際にはメッシュをスライスすることで断面などの表示か可能です。
しかし、今回は特に断面を見る必要もないので何もしません。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
マニュアルにある通り疎行列に代入するための値とそのインデックスを準備します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
メッシュmに２つの要素が追加されました。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
今回は１次のガウス積分を使用します。
Can you write Python code for it?

im = gf.Integ('IM_GAUSS1D(1)')

mim = gf.MeshIm(m, im)

gf.asm_linear_elasticity?

gf.asm_mass_matrix?

# ラメの弾性定数
Lambda = np.array([1.0, 1.0, 1.0])
Mu = np.array([1.0, 1.0, 1.0])
# 剛性行列の作成
K = gf.asm_linear_elasticity(mim, mfu, mfd, Lambda, Mu)
print K.full() # 剛性行列の出力
# 質量行列の作成
M = gf.asm_mass_matrix(mim, mfu)
print M.full() # 質量行列の出力


I am working on the file "Study01.ipynb".
The first task is:
以上のように各インデックスに値の入った疎行列が作成されます。このようにオブジェクトを作成した際には必ずなんのオブジェクトであるかを確認しておきましょう。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
以下のようにしてファイルから行列が読み込めていることを確認できます。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
作成したFem要素をMeshFemに設定します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
例えば、先ほど作成した行列$A$の逆行列を計算したい場合はinv関数を使用します。
Can you write Python code for it?

np.linalg.inv?

AA = np.linalg.inv(A)
print AA


I am working on the file "Study01.ipynb".
The first task is:
出力されたファイルはノートブックと同じディレクトリに保存されています。
Can you write Python code for it?

K = io.mmread("K.mtx")
M = io.mmread("M.mtx")

print "K = "
print K
print "M = "
print M


I am working on the file "Study01.ipynb".
The first task is:
変位用のMeshFemとデータ用のMeshFemを作成します。MeshFemはMesh上にFem要素を割り当てるためのオブジェクトです。
Can you write Python code for it?

mfu = gf.MeshFem(m,1) # 変位用MeshFem
mfd = gf.MeshFem(m,1) # データ用MeshFem


I am working on the file "Study01.ipynb".
The first task is:
密行列と同じ操作を疎行列の線形代数パッケージで実行します。
Can you write Python code for it?

A  = A.tocsc()
AA = linalg.inv(A)
print AA


I am working on the file "Study01.ipynb".
The first task is:
密行列の場合と同じ計算結果になることが確認できます。
それぞれの関数の説明は以下で確認してください。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
密行列の場合は行列積が"*"になりますので注意してください。先ほどと同じように単位行列が逆行列との積で得られます。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
座標 x = 0.0からx = 2.0の間に平行６面体要素を追加します。
Can you write Python code for it?

m.add_convex?

m.add_convex(gf.GeoTrans('GT_QK(1,1)'),[[0,2]])

m.add_convex(gf.GeoTrans('GT_QK(1,1)'),[[2,4]])


I am working on the file "Study01.ipynb".
The first task is:
本節ではメッシュ作成とそれを離散化した行列を作成するパッケージGetfem++について説明します。必要となるオブジェクトは以下の図に示すものになります。
<img src="http://download.gna.org/getfem/html/homepage/_images/getfem_structure1.png">
Femオブジェクト
有限要素で何を使用するかを指定する。(ラグランジュ要素Pk、Qkなど)
Meshオブジェクト
メッシュの指定(point:節点とconvex:要素などから作られる)
Integオブジェクト
積分方法を指定する。
MeshFemオブジェクト
メッシュの要素に対してFemを設定したオブジェクト、自由度数などの情報も設定する。
MeshImオブジェクト
メッシュの要素に対して積分方法を指定したオブジェクト
Assemblyオブジェクト
離散化した行列組み立てのためのオブジェクト。MeshFemとMeshImを基に作成する。
Modelオブジェクト
モデル全体の情報を設定するオブジェクト。今回は未使用。
マニュアルからわかるように、規則的なメッシュを作成したり、Gmshのメッシュファイルなどからインポートすることも可能です。ただ、今回はメッシュオブジェクトに慣れることを目的としますので、まずは１次元の空のMeshオブジェクトを作成します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
次は、積分要素を設定します。
Can you write Python code for it?


I am working on the file "Study01.ipynb".
The first task is:
疎行列の線形代数
以上では、Pythonによる密行列の線形代数までを紹介してきました。しかしながら、有限要素法計算の際にはメモリ節約のため疎行列計算が行われます。その際にはscipyのsparseパッケージとその下のlinalgパッケージを使用します。
Can you write Python code for it?

import scipy as sp
from scipy.sparse import linalg

linalg?


I am working on the file "Study01.ipynb".
The first task is:
要素を追加していきます。(add_convexメソッド)
Can you write Python code for it?


I am working on the file "StyleGuide.ipynb".
The first task is:
Long text can be wrapped in """
Can you write Python code for it?

quote_the_raven = """
Once upon a midnight dreary, while I pondered, weak and weary,
Over many a quaint and curious volume of forgotten lore—
While I nodded, nearly napping, suddenly there came a tapping,
As of some one gently rapping, rapping at my chamber door.
"Tis some visiter," I muttered, "tapping at my chamber door—
            Only this and nothing more."
"""

print(quote_the_raven)


I am working on the file "StyleGuide.ipynb".
The first task is:
|--------|---------|---------|---------|---------|---------|---------|---------
Function Names
Use the same guidelines as for variable names
Can you write Python code for it?

# Good

def find_kinetic_energy(mass_part, velocity_part):

    kinetic_energy = 0.5 * mass_part * velocity_part ** 2
    return(kinetic_energy)

# Bad

def KE(x,y):
    return(0.5*x*y**2)

# Good

mass_particle = 10.0
velocity_particle = 20.0

kinetic_energy = find_kinetic_energy(mass_particle,velocity_particle)

print(kinetic_energy)

# Bad

print(KE(10,20))


I am working on the file "StyleGuide.ipynb".
The first task is:
|--------|---------|---------|---------|---------|---------|---------|---------
Line Length

Limit all lines to a maximum of 79 characters
The piece that seperates the sections in this notebook is 79 characters long
If you have to scroll horizontally - your line is too long

The preferred way of wrapping long lines is by using Python's implied line continuation
inside parentheses (), brackets [] and braces {}.
Long lines can be broken over multiple lines by wrapping expressions in parentheses.
An Example:
Can you write Python code for it?

# Some variables to use in an equation

gross_wages = 50000
taxable_interest = 1000
dividends = 50
qualified_dividends = 10
ira_deduction = 2000
student_loan_interest = 3000
medical_deduction = 1000

# Good

income = (gross_wages
          + taxable_interest
          + (dividends - qualified_dividends)
          - ira_deduction
          - student_loan_interest
          - medical_deduction)

print(income)

# Bad

income = gross_wages + taxable_interest + (dividends - qualified_dividends) - ira_deduction - student_loan_interest - medical_deduction

print(income)


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Content loss
Content is the straight forward sum of squares between the base and the target image.
Can you write Python code for it?

def content_loss(base, combination):
    return K.sum(K.square(combination - base))


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Create a simple helper dictionary to get to the outputs of the layers in the model:
Can you write Python code for it?

outputs_dict = {layer.name: layer.output for layer in model.layers}


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Fetch the three images we need from the Internet if they aren't already loaded. Base Image is the image that we'll render a style transfered version of, style reference 1 & 2 are the style images to interpolate between.
Can you write Python code for it?

IMAGE_BASE = 'style_transfer/'

def fetch_image(url, fn=None):
    if not fn:
        fn = url.rsplit('/', 1)[-1]
    fn = IMAGE_BASE + fn
    if os.path.isfile(fn):
        return fn
    img = requests.get(url).content
    with open(fn, 'wb') as fout:
        fout.write(img)
    return fn

base_image_path = fetch_image('https://upload.wikimedia.org/wikipedia/commons/0/08/Okerk2.jpg')
style_reference_image_path_2 = fetch_image('https://upload.wikimedia.org/wikipedia/commons/9/99/Jan_van_Goyen_004b.jpg')
style_reference_image_path_1 = fetch_image('https://upload.wikimedia.org/wikipedia/commons/6/66/VanGogh-starry_night_ballance1.jpg')
Image(filename=base_image_path)


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Load the images into three variables and create a place holder for the resulting combination image. Then create a tensor that contains all four of them next to each other so we can process them together in the same way as we'd process a mini-batch when training.
Can you write Python code for it?

base_image = K.variable(preprocess_image(base_image_path))
style_reference_image_1 = K.variable(preprocess_image(style_reference_image_path_1))
style_reference_image_2 = K.variable(preprocess_image(style_reference_image_path_2))

# this will contain our generated image
if K.image_data_format() == 'channels_first':
    combination_image = K.placeholder((1, 3, img_nrows, img_ncols))
else:
    combination_image = K.placeholder((1, img_nrows, img_ncols, 3))

# combine the 4 images into a single Keras tensor
input_tensor = K.concatenate([base_image,
                              style_reference_image_1,
                              style_reference_image_2,
                              combination_image], axis=0)
input_tensor


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Now load the pretrained vgg16 model and load it up with the four images as input.
Can you write Python code for it?

model = vgg16.VGG16(input_tensor=input_tensor,
                    weights='imagenet', include_top=False)
print('Model loaded.')


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Some functions to convert from and to the internal format used. Make sure we end up with images of the right size and take care of the colors being at the position that the selected back end expects.
Can you write Python code for it?

def preprocess_image(image_path):
    img = load_img(image_path, target_size=(img_nrows, img_ncols))
    img = img_to_array(img)
    img = np.expand_dims(img, axis=0)
    img = vgg16.preprocess_input(img)
    return img

# util function to convert a tensor into a valid image

def deprocess_image(x):
    if K.image_data_format() == 'channels_first':
        x = x.reshape((3, img_nrows, img_ncols))
        x = x.transpose((1, 2, 0))
    else:
        x = x.reshape((img_nrows, img_ncols, 3))
    # Remove zero-center by mean pixel
    x[:, :, 0] += 103.939
    x[:, :, 1] += 116.779
    x[:, :, 2] += 123.68
    # 'BGR'->'RGB'
    x = x[:, :, ::-1]
    x = np.clip(x, 0, 255).astype('uint8')
    return x


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Some working variables and constants we are going to need. Change img_nrows to influence the size of the output picture. The various *_weight variables will determine how much influence the three loss functions have (see below). Iterations is the total of number of iterations, the first warm_up will be used to get to a stable version of the image, the rest will be used for the interpolation frames. result_prefix is the prefix used for the intermediate images
Can you write Python code for it?

width, height = load_img(base_image_path).size
img_nrows = 400
img_ncols = int(width * img_nrows / height)

total_variation_weight = 1.0
style_weight = 1.0
content_weight = 0.025

iterations = 120
warm_up = 30

result_prefix = 'star_goyen'


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
The combined loss function
Now combine the three loss functions from above into one that we can optimize for. We use selected layers from the neural network for style and content loss, while the variation loss is calculated on just the resulting image.
We introduce an extra place holder variable "step" indicating where we are transitioning from one style to another and calculate the style loss for both style images and scale the result accordingly between these two loss values. Finally we calculate the gradient with regard to the combination image.
Can you write Python code for it?

loss = K.variable(0.)
layer_features = outputs_dict['block4_conv2']
base_image_features = layer_features[0, :, :, :]
combination_features = layer_features[3, :, :, :]
loss += content_weight * content_loss(base_image_features,
                                      combination_features)
step = K.placeholder()
feature_layers = ['block1_conv1', 'block2_conv1',
                  'block3_conv1', 'block4_conv1',
                  'block5_conv1']
for layer_name in feature_layers:
    layer_features = outputs_dict[layer_name]
    style_reference_features_1 = layer_features[1, :, :, :]
    style_reference_features_2 = layer_features[2, :, :, :]
    combination_features = layer_features[3, :, :, :]
    sl_1 = style_loss(style_reference_features_1, combination_features) * step
    sl_2 = style_loss(style_reference_features_2, combination_features) * (1 - step)
    loss += (style_weight / len(feature_layers)) * (sl_1 + sl_2)
loss += total_variation_weight * total_variation_loss(combination_image)

# get the gradients of the generated image wrt the loss
grads = K.gradients(loss, combination_image)

outputs = [loss]
if isinstance(grads, (list, tuple)):
    outputs += grads
else:
    outputs.append(grads)

f_outputs = K.function([combination_image, step], outputs)


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
To display the result as an animated gif, we create a cycle from the frames variable and then use imageio to create an animated gif.
Can you write Python code for it?

cycled = frames + list(reversed(frames[1:-1]))
# Save them as frames into a gif
kargs = { 'duration': 0.1 }
imageio.mimsave(IMAGE_BASE + 'animated.gif', [imageio.imread(x) for x in cycled], 'GIF', **kargs)

HTML('<img src="%s">' % (IMAGE_BASE + 'animated.gif'))


I am working on the file "Style_Transfer_Interpolation.ipynb".
The first task is:
Variation Loss
Minimize the local variation by comparing effectively pixels next to each other and minimizing the variation. This keeps the resulting image somewhat fuzzy, but it avoids large jumps in the pixels and keeps the image locally coherent.
Can you write Python code for it?

def total_variation_loss(x):
    assert K.ndim(x) == 4
    if K.image_data_format() == 'channels_first':
        a = K.square(x[:, :, :img_nrows - 1, :img_ncols - 1] - x[:, :, 1:, :img_ncols - 1])
        b = K.square(x[:, :, :img_nrows - 1, :img_ncols - 1] - x[:, :, :img_nrows - 1, 1:])
    else:
        a = K.square(x[:, :img_nrows - 1, :img_ncols - 1, :] - x[:, 1:, :img_ncols - 1, :])
        b = K.square(x[:, :img_nrows - 1, :img_ncols - 1, :] - x[:, :img_nrows - 1, 1:, :])
    return K.sum(K.pow(a + b, 1.25))


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
Example
This example shows how to transfer the style of various images onto a portrait.
First we load the content-image which has the overall contours that we want in the mixed-image.
Can you write Python code for it?

content_filename = 'images/elon_musk.jpg'
content_image = load_image(content_filename, max_size=300)

filenamecontent='images/outputs_StyleTransfer/Content.jpg'
print(filenamecontent)
save_image(content_image, filenamecontent)


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
import PIL.Image


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
Loss Functions
These helper-functions create the loss-functions that are used in optimization with TensorFlow.
This function creates a TensorFlow operation for calculating the Mean Squared Error between the two input tensors.
Can you write Python code for it?

def mean_squared_error(a, b):
    return tf.reduce_mean(tf.square(a - b))

def create_content_loss(session, model, content_image, layer_ids):
    """
    Create the loss-function for the content-image.

    Parameters:
    session: An open TensorFlow session for running the model's graph.
    model: The model, e.g. an instance of the VGG16-class.
    content_image: Numpy float array with the content-image.
    layer_ids: List of integer id's for the layers to use in the model.
    """

    # Create a feed-dict with the content-image.
    feed_dict = model.create_feed_dict(image=content_image)

    # Get references to the tensors for the given layers.
    layers = model.get_layer_tensors(layer_ids)

    # Calculate the output values of those layers when
    # feeding the content-image to the model.
    values = session.run(layers, feed_dict=feed_dict)

    # Set the model's graph as the default so we can add
    # computational nodes to it. It is not always clear
    # when this is necessary in TensorFlow, but if you
    # want to re-use this code then it may be necessary.
    with model.graph.as_default():
        # Initialize an empty list of loss-functions.
        layer_losses = []

        # For each layer and its corresponding values
        # for the content-image.
        for value, layer in zip(values, layers):
            # These are the values that are calculated
            # for this layer in the model when inputting
            # the content-image. Wrap it to ensure it
            # is a const - although this may be done
            # automatically by TensorFlow.
            value_const = tf.constant(value)

            # The loss-function for this layer is the
            # Mean Squared Error between the layer-values
            # when inputting the content- and mixed-images.
            # Note that the mixed-image is not calculated
            # yet, we are merely creating the operations
            # for calculating the MSE between those two.
            loss =...


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
Save an image as a jpeg-file. The image is given as a numpy array with pixel-values between 0 and 255.
Can you write Python code for it?

def save_image(image, filename):
    # Ensure the pixel-values are between 0 and 255.
    image = np.clip(image, 0.0, 255.0)

    # Convert to bytes.
    image = image.astype(np.uint8)

    # Write the image-file in jpeg-format.
    with open(filename, 'wb') as file:
        PIL.Image.fromarray(image).save(file, 'jpeg')


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
The VGG-16 model is downloaded from the internet. This is the default directory where you want to save the data-files. The directory will be created if it does not exist.
Can you write Python code for it?

# vgg16.data_dir = 'vgg16/'

vgg16.maybe_download()


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
Then we define a list of integers which identify the layers in the neural network that we want to use for matching the content-image. These are indices into the layers in the neural network. For the VGG16 model, the 5th layer (index 4) seems to work well as the sole content-layer.
Can you write Python code for it?


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
Then we load the style-image which has the colours and textures we want in the mixed-image.
Can you write Python code for it?

style_filename = 'images/style9.jpg'
style_image = load_image(style_filename, max_size=300)

filenamestyle='images/outputs_StyleTransfer/Style.jpg'
print(filenamestyle)
save_image(style_image, filenamestyle)


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
This function plots a large image. The image is given as a numpy array with pixel-values between 0 and 255.
This function plots the content-, mixed- and style-images.
Can you write Python code for it?

def plot_image_big(image):
    # Ensure the pixel-values are between 0 and 255.
    image = np.clip(image, 0.0, 255.0)

    # Convert pixels to bytes.
    image = image.astype(np.uint8)

    # Convert to a PIL-image and display it.
    display(PIL.Image.fromarray(image))

def plot_images(content_image, style_image, mixed_image):
    # Create figure with sub-plots.
    fig, axes = plt.subplots(1, 3, figsize=(10, 10))

    # Adjust vertical spacing.
    fig.subplots_adjust(hspace=0.1, wspace=0.1)

    # Use interpolation to smooth pixels?
    smooth = True

    # Interpolation type.
    if smooth:
        interpolation = 'sinc'
    else:
        interpolation = 'nearest'

    # Plot the content-image.
    # Note that the pixel-values are normalized to
    # the [0.0, 1.0] range by dividing with 255.
    ax = axes.flat[0]
    ax.imshow(content_image / 255.0, interpolation=interpolation)
    ax.set_xlabel("Content")

    # Plot the mixed-image.
    ax = axes.flat[1]
    ax.imshow(mixed_image / 255.0, interpolation=interpolation)
    ax.set_xlabel("Mixed")

    # Plot the style-image
    ax = axes.flat[2]
    ax.imshow(style_image / 255.0, interpolation=interpolation)
    ax.set_xlabel("Style")

    # Remove ticks from all the plots.
    for ax in axes.flat:
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "Style_Transfer_Modifying_Function_ For_Denoising_Image_By_Shifting_By_50_Pixels.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Example
This example shows how to transfer the style of various images onto a portrait.
First we load the content-image which has the overall contours that we want in the mixed-image.
Can you write Python code for it?

content_filename = 'images/download.jpg'
content_image = load_image(content_filename, max_size=None)

filenamecontent='images/outputs_StyleTransfer/Content.jpg'
print(filenamecontent)
save_image(content_image, filenamecontent)


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Helper-functions for image manipulation
This function loads an image and returns it as a numpy array of floating-points. The image can be automatically resized so the largest of the height or width equals max_size.
Can you write Python code for it?

def load_image(filename, max_size=None):
    image = PIL.Image.open(filename)

    if max_size is not None:
        # Calculate the appropriate rescale-factor for
        # ensuring a max height and width, while keeping
        # the proportion between them.
        factor = max_size / np.max(image.size)

        # Scale the image's height and width.
        size = np.array(image.size) * factor

        # The size is now floating-point because it was scaled.
        # But PIL requires the size to be integers.
        size = size.astype(int)

        # Resize the image.
        image = image.resize(size, PIL.Image.LANCZOS)
        print(image)

    # Convert to numpy floating-point array.
    return np.float32(image)


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Imports
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Now perform the style-transfer. This automatically creates the appropriate loss-functions for the style- and content-layers, and
then performs a number of optimization iterations. This will gradually create a mixed-image which has similar contours as the content-image, with the colours and textures being similar to the style-image.
This can be very slow on a CPU!
Can you write Python code for it?

%%time
img = style_transfer(content_image=content_image,
                     style_image=style_image,
                     content_layer_ids=content_layer_ids,
                     style_layer_ids=style_layer_ids,
                     weight_content=1.5,
                     weight_style=10.0,
                     weight_denoise=0.3,
                     num_iterations=150,
                     step_size=10.0)

# Function for printing output image
filename='images/outputs_StyleTransfer/Mixed.jpg'
save_image(img, filename)


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Save an image as a jpeg-file. The image is given as a numpy array with pixel-values between 0 and 255.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
The VGG-16 model is downloaded from the internet. This is the default directory where you want to save the data-files. The directory will be created if it does not exist.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Then we define a list of integers which identify the layers in the neural network that we want to use for matching the content-image. These are indices into the layers in the neural network. For the VGG16 model, the 5th layer (index 4) seems to work well as the sole content-layer.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Then we define another list of integers for the style-layers.
Can you write Python code for it?

# The VGG16-model has 13 convolutional layers.
# This selects all those layers as the style-layers.
style_layer_ids = list(range(13))

# You can also select a sub-set of the layers, e.g. like this:
# style_layer_ids = [1, 2, 3, 4]


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
Then we load the style-image which has the colours and textures we want in the mixed-image.
Can you write Python code for it?

style_filename = 'images/style4.jpg'
style_image = load_image(style_filename, max_size=None)

filenamestyle='images/outputs_StyleTransfer/Style.jpg'
print(filenamestyle)
save_image(style_image, filenamestyle)


I am working on the file "Style_Transfer_Saving_Input_Output_Images.ipynb".
The first task is:
This function plots a large image. The image is given as a numpy array with pixel-values between 0 and 255.
This function plots the content-, mixed- and style-images.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
Helper-functions for image manipulation
This function loads an image and returns it as a numpy array of floating-points. The image can be automatically resized so the largest of the height or width equals max_size.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
Imports
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
Loss Functions
These helper-functions create the loss-functions that are used in optimization with TensorFlow.
This function creates a TensorFlow operation for calculating the Mean Squared Error between the two input tensors.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
Save an image as a jpeg-file. The image is given as a numpy array with pixel-values between 0 and 255.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
Then we define a list of integers which identify the layers in the neural network that we want to use for matching the content-image. These are indices into the layers in the neural network. For the VGG16 model, the 5th layer (index 4) seems to work well as the sole content-layer.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
Then we load the style-image which has the colours and textures we want in the mixed-image.
Can you write Python code for it?

style_filename = 'images/style7.jpg'
style_image = load_image(style_filename, max_size=300)

filenamestyle='images/outputs_StyleTransfer/Style.jpg'
print(filenamestyle)
save_image(style_image, filenamestyle)


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
This function plots a large image. The image is given as a numpy array with pixel-values between 0 and 255.
This function plots the content-, mixed- and style-images.
Can you write Python code for it?


I am working on the file "Style_Transfer_Saving_Mixed_Image_Every_10_Iterations.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "Style_Transfer_Without_Calculating_Denoising_Loss.ipynb".
The first task is:
Save an image as a jpeg-file. The image is given as a numpy array with pixel-values between 0 and 255.
Can you write Python code for it?


I am working on the file "Style_Transfer_Without_Calculating_Denoising_Loss.ipynb".
The first task is:
Then we define a list of integers which identify the layers in the neural network that we want to use for matching the content-image. These are indices into the layers in the neural network. For the VGG16 model, the 5th layer (index 4) seems to work well as the sole content-layer.
Can you write Python code for it?


I am working on the file "Style_Transfer_Without_Calculating_Denoising_Loss.ipynb".
The first task is:
Then we define another list of integers for the style-layers.
Can you write Python code for it?

# The VGG16-model has 13 convolutional layers.
# This selects all those layers as the style-layers.
# This is somewhat slow to optimize.
style_layer_ids = list(range(13))

# You can also select a sub-set of the layers, e.g. like this:
# style_layer_ids = [1, 2, 3, 4]

%%time
img = style_transfer(content_image=content_image,
                     style_image=style_image,
                     content_layer_ids=content_layer_ids,
                     style_layer_ids=style_layer_ids,
                     weight_content=1.5,
                     weight_style=10.0,
                     weight_denoise=0.3,
                     num_iterations=150,
                     step_size=10.0)

# Function for printing mixed output image
filename='images/outputs_StyleTransfer/Mixed.jpg'
save_image(img, filename)


I am working on the file "Style_Transfer_Without_Calculating_Denoising_Loss.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "Style_Transfer_Without_Style_Loss.ipynb".
The first task is:
Helper-functions for image manipulation
This function loads an image and returns it as a numpy array of floating-points. The image can be automatically resized so the largest of the height or width equals max_size.
Can you write Python code for it?


I am working on the file "Style_Transfer_Without_Style_Loss.ipynb".
The first task is:
Then we define a list of integers which identify the layers in the neural network that we want to use for matching the content-image. These are indices into the layers in the neural network. For the VGG16 model, the 5th layer (index 4) seems to work well as the sole content-layer.
Can you write Python code for it?


I am working on the file "Stylometry/Bag of Words/BagOfWords.ipynb".
The first task is:
Applying Bag of Words
To apply the Naive Bayes theorem to our dataset, we should convert all our data into numeric values since sklearn can't work with non numeric values. So, we created a frequency matrix(word frequency) of our dataset and apply Bayes theorem to that. So, we used a module called CountVectorizer to that.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

count_vector = CountVectorizer()

# Fit the training data and then return the matrix
training_data = count_vector.fit_transform(X_train)

# Transform testing data and return the matrix. Note we are not fitting the testing data into the CountVectorizer()
testing_data = count_vector.transform(X_test)


I am working on the file "Stylometry/Bag of Words/BagOfWords.ipynb".
The first task is:
Now, we should apply bayes technique for this dataset. So, we should import MultinomialNB module from sklearn. Here, we use Multinimial Naive Bayes because it good for classification with discrete values.
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB
naive_bayes = MultinomialNB()
naive_bayes.fit(training_data, y_train)

## Now that we trained our model, it's time to test the model with the dataset.

predictions = naive_bayes.predict(testing_data)


I am working on the file "Subreddit Analysis Direct t-SNE.ipynb".
The first task is:
As you can see the results don't look as good -- although if you go bakc up and remmove layers of the manipulations I performed on the conditional probability matrix and rerun things you'll see how much worse things get. I still believe this idea has merit, but making it work in practice involves going back to the drawing board to determine how to correctly manipulate the count matrix to make a suitable conditional probability matrix; hacking around, as I was doing here, will not cut it.
Finally, as usual for the experimental notebooks, I look at the actual content of the clusters.
Can you write Python code for it?

def is_nsfw(subreddit):
    return re.search(r'(nsfw|gonewild)', subreddit)

for cid in range(cluster_ids.max() + 1):
    subreddits = subreddit_map_df.subreddit[cluster_ids == cid]
    if np.any(subreddits.map(is_nsfw)):
        subreddits = ' ... Censored ...'
    else:
        subreddits = subreddits.values

    print '\nCluster {}:\n{}\n'.format(cid, subreddits)


I am working on the file "Subreddit Analysis PPMI.ipynb".
The first task is:
Next we compute the denominator by multiplying $P(A)$ by $P(B)$ pointwise across the whole matrix.
Can you write Python code for it?

pmi_denominator = row_normalized.copy()
pmi_denominator.data = row_normalized.data * col_normalized.data


I am working on the file "Subreddit Analysis PPMI.ipynb".
The first task is:
Things proceed the same as the other analyses up to the construction of the count matrix. Now, instead of row normalizing to get conditional probabilities we are going to compute the pointwise mutual information. Given events $A$ and $B$ we define the pointwise mutual information of A and B to be
$$
\text{PMI}(A, B) = \log\left(\frac{P(A, B)}{P(A)P(B)}\right);
$$
that is the PMI is the log of the ratio of the joint probability of $A$ and $B$ occuring with the product of the independent probabilties of $A$ and $B$. To start we'll just compute $P(A)$ and $P(B)$ by row and column normalizing.
Can you write Python code for it?

row_normalized = sklearn.preprocessing.normalize(count_matrix.tocsr(), norm='l1')
col_normalized = sklearn.preprocessing.normalize(count_matrix.tocsr(), norm='l1', axis=0)


I am working on the file "Subreddit Analysis PPMI.ipynb".
The first task is:
We can then compute the PMI by taking the log of the numerator over the denominator. One alternative I explored was simply forcing all values to be positive by adding the min value (of non-zero entries). It didn't help.
Can you write Python code for it?

pmi_matrix = pmi_numerator.copy()
pmi_matrix.data = np.log(pmi_numerator.data / pmi_denominator.data)
# pmi_matrix.data += pmi_matrix.min()


I am working on the file "Subreddit Analysis PPMI.ipynb".
The first task is:
We now return to our regularly scheduled programming ...
Can you write Python code for it?

reduced_vectors = TruncatedSVD(n_components=500,
                               random_state=0).fit_transform(ppmi_matrix)

reduced_vectors = sklearn.preprocessing.normalize(reduced_vectors[:10000], norm='l2')

seed_state = np.random.RandomState(0)
subreddit_map = tsne.bh_sne(reduced_vectors[:10000], perplexity=50.0, random_state=seed_state)

subreddit_map_df = pd.DataFrame(subreddit_map, columns=('x', 'y'))
subreddit_map_df['subreddit'] = subreddits[:10000]
subreddit_map_df.head()

import hdbscan

clusterer = hdbscan.HDBSCAN(min_samples=5,
                            min_cluster_size=20).fit(subreddit_map)
cluster_ids = clusterer.labels_

subreddit_map_df['cluster'] = cluster_ids

from bokeh.plotting import figure, show, output_notebook, output_file
from bokeh.models import HoverTool, ColumnDataSource, value
from bokeh.models.mappers import LinearColorMapper
from bokeh.palettes import viridis
from collections import OrderedDict

output_notebook()

color_mapper = LinearColorMapper(palette=viridis(256), low=0, high=cluster_ids.max())
color_dict = {'field': 'cluster', 'transform': color_mapper}

plot_data_clusters = ColumnDataSource(subreddit_map_df[subreddit_map_df.cluster >= 0])
plot_data_noise = ColumnDataSource(subreddit_map_df[subreddit_map_df.cluster < 0])

tsne_plot = figure(title=u'A Map of Subreddits',
                   plot_width = 700,
                   plot_height = 700,
                   tools= (u'pan, wheel_zoom, box_zoom,'
                           u'box_select, resize, reset'),
                   active_scroll=u'wheel_zoom')
tsne_plot.add_tools( HoverTool(tooltips = OrderedDict([('subreddit', '@subreddit'),
                                                       ('cluster', '@cluster')])))

# draw clusters
tsne_plot.circle(u'x', u'y', source=plot_data_clusters,
                 fill_color=color_dict, line_alpha=0.002, fill_alpha=0.1,
                 size=10, hover_line_color=u'black')
# draw noise
tsne_plot.circle(u'x', u'y', source=plot_data_noise,
                 fill_color=u'gray',...


I am working on the file "Subreddit Analysis t-SNE.ipynb".
The first task is:
Again with the hand-rolled normalisation. It was not hard in this case.
Can you write Python code for it?

reduced_vectors /= np.sqrt((reduced_vectors**2).sum(axis=1))[:, np.newaxis]


I am working on the file "Subreddit Analysis t-SNE.ipynb".
The first task is:
Clustering looks pretty much the same as it did in the final version. I played with parameters a little here, and also looked at leaf clustering as the cluster extraction method. In practice, however, the standard Excess of Mass approach was more than adequate.
Can you write Python code for it?

import hdbscan

clusterer = hdbscan.HDBSCAN(min_samples=5,
                            min_cluster_size=20).fit(subreddit_map)
cluster_ids = clusterer.labels_

subreddit_map_df['cluster'] = cluster_ids


I am working on the file "Subreddit Analysis t-SNE.ipynb".
The first task is:
I hadn't bothered to look if the relevant scikit-learn functions actually accepted sparse matrices when I was just playing, so I did the row normalization myself by hand.
Can you write Python code for it?

conditional_prob_matrix = count_matrix.tocsr()
row_sums = np.array(conditional_prob_matrix.sum(axis=1))[:,0]
row_indices, col_indices = conditional_prob_matrix.nonzero()
conditional_prob_matrix.data /= row_sums[row_indices]

reduced_vectors = TruncatedSVD(n_components=500,
                               random_state=0).fit_transform(conditional_prob_matrix)


I am working on the file "Subreddit Analysis t-SNE.ipynb".
The first task is:
Instead of LargeVis we can just use t-SNE. Some caveats: the tnse package is still quite a bit faster than t-SNE in scikit-learn, but iot only works with python 2.
Can you write Python code for it?

seed_state = np.random.RandomState(0)
subreddit_map = tsne.bh_sne(reduced_vectors[:10000], perplexity=50.0, random_state=seed_state)

subreddit_map_df = pd.DataFrame(subreddit_map, columns=('x', 'y'))
subreddit_map_df['subreddit'] = subreddits[:10000]
subreddit_map_df.head()


I am working on the file "Subreddit Analysis t-SNE.ipynb".
The first task is:
Onto the Bokeh plotting. This was still just experimenting with mapping and clustering so I hadn't honed down the plot code much. I don't do nice colormapping, for instance, but instead plot the noise and cluster points separately. There is also no adjustment of alpha channels based on zoom levels. It was good enough to view the map and mouse over regions to see how well things worked.
Can you write Python code for it?

from bokeh.plotting import figure, show, output_notebook, output_file
from bokeh.models import HoverTool, ColumnDataSource, value
from bokeh.models.mappers import LinearColorMapper
from bokeh.palettes import viridis
from collections import OrderedDict

output_notebook()

color_mapper = LinearColorMapper(palette=viridis(256), low=0, high=cluster_ids.max())
color_dict = {'field': 'cluster', 'transform': color_mapper}

plot_data_clusters = ColumnDataSource(subreddit_map_df[subreddit_map_df.cluster >= 0])
plot_data_noise = ColumnDataSource(subreddit_map_df[subreddit_map_df.cluster < 0])

tsne_plot = figure(title=u'A Map of Subreddits',
                   plot_width = 700,
                   plot_height = 700,
                   tools= (u'pan, wheel_zoom, box_zoom,'
                           u'box_select, resize, reset'),
                   active_scroll=u'wheel_zoom')
tsne_plot.add_tools( HoverTool(tooltips = OrderedDict([('subreddit', '@subreddit'),
                                                       ('cluster', '@cluster')])))

# draw clusters
tsne_plot.circle(u'x', u'y', source=plot_data_clusters,
                 fill_color=color_dict, line_alpha=0.002, fill_alpha=0.1,
                 size=10, hover_line_color=u'black')
# draw noise
tsne_plot.circle(u'x', u'y', source=plot_data_noise,
                 fill_color=u'gray', line_alpha=0.002, fill_alpha=0.05,
                 size=10, hover_line_color=u'black')

# configure visual elements of the plot
tsne_plot.title.text_font_size = value(u'16pt')
tsne_plot.xaxis.visible = False
tsne_plot.yaxis.visible = False
tsne_plot.grid.grid_line_color = None
tsne_plot.outline_line_color = None

show(tsne_plot);