I am working on the file "Construct Dataset.ipynb".
The first task is:
至此，数据集title_set1——Count>10构造完成！ used for classification model

stop words were not ignored
text was not tokenized and lowercased
no stemming was used

Now, let's contruct the dataset for computing the similarity
start from here
Can you write Python code for it?

import numpy as np
import pandas as pd
experts_count=pd.read_pickle('experts_count.pkl')
experts_count=experts_count.fillna('none')

train = experts_count[:80000]
test = experts_count[80000:]
title_train = train.drop(['Id','QuestionBody','CodeBody','Tag', \
                           'ExpertId','Count',],axis=1)
title_test = test.drop(['Id','QuestionBody','CodeBody','Tag', \
                         'ExpertId','Count',],axis=1)

title_train = title_train.groupby('Label',as_index=True).agg(lambda x: ' '.join(x))
title_test = title_test.set_index('Label')

title_train.info()

title_test.info()

train_unique_expert = train.ExpertId.unique()
print ("number of experts in train set: %r " % train_unique_expert.shape)

test_unique_expert = test.ExpertId.unique()
print ("number of experts in test set: %r" % test_unique_expert.shape)

print ("type : %r" % type(test_unique_expert))

l = np.intersect1d(train_unique_expert,test_unique_expert)
print ("the number of experts both in train set and test set: %r" % l.shape)
# 1417 - 698 = 719, 719 + 8285 = 9004. : 有719个专家是在测试集中新出现的。

title_train.to_csv('title_train_similarity',encoding='utf-8')
title_test.to_csv('title_test_similarity',encoding='utf-8')


I am working on the file "Construct Dataset.ipynb".
The first task is:
至此，数据集title_set构造完成！ used for similarity model

stop words were not ignored
text was not tokenized and lowercased
no stemming was used

去停用词
Can you write Python code for it?

import nltk
stopset = set(nltk.corpus.stopwords.words('english'))

texts = list(experts_count.Title)
# Tokenize the titles
texts = [nltk.word_tokenize(text) for text in texts]
# pos tag the tokens
txtpos = [nltk.pos_tag(texts) for texts in texts]
# for titles we only care about verbs and nouns
txtpos = [[w for w in s if (w[1][0] == 'N' or w[1][0] == 'V') and \
           w[0].lower() not in stopset]
                  for s in txtpos]

qbodys = list(dfFinal.QuestionBody)
#break into sentences
qsents = [nltk.sent_tokenize(text) for text in qbodys]
# Tokenize the question body
qbodys = [nltk.word_tokenize(text) for text in qbodys]
# attach tags to the body
qpos = [nltk.pos_tag(texts) for texts in qbodys]

import cPickle
cPickle.dump([train1, test1], open('data1.pkl','wb'))


I am working on the file "Constructing Scalar Models.ipynb".
The first task is:
Having found these modes, we can now use them as a basis for constructing models of our resonator which can describe all the dynamics in a relatively simple form.
Exact solution to extinction
Now we utilise this model to calculate the extinction cross section of each mode. First we setup the incident wave polarisation and direction and the frequency range of interest.
Can you write Python code for it?

num_freqs = 101
freqs = np.linspace(1e9, 30e9, num_freqs)

e_inc = np.array([1.0, 1.0, 0])
k_dir = np.array([0, 0, 1])
plane_wave = PlaneWaveSource(e_inc, k_dir, p_inc =1.0)

area = np.pi*(0.5*mesh.max_distance)**2

extinction = np.empty(num_freqs, np.complex128)
extinction_modes = np.empty((num_freqs, len(modes)), np.complex128)


I am working on the file "Constructing Scalar Models.ipynb".
The first task is:
Initialising simulation
Can you write Python code for it?

sim = openmodes.Simulation(notebook=True)
mesh = sim.load_mesh(osp.join(openmodes.geometry_dir, "SRR.geo"), parameters={'inner_radius': 2.5e-3}, mesh_tol=0.8e-3)
ring = sim.place_part(mesh)


I am working on the file "Constructing Scalar Models.ipynb".
The first task is:
Searching for modes
Can you write Python code for it?

start_freq = 2e9
start_s = 2j*np.pi*start_freq

num_modes = 4
estimates = sim.estimate_poles(start_s, modes=num_modes, cauchy_integral=False)
refined = sim.refine_poles(estimates, rel_tol=1e-12)
modes = refined.add_conjugates()

plt.figure()
plt.plot(modes.s.imag, np.abs(modes.s.real), 'x')
plt.xlabel('Frequency $j\omega$')
plt.ylabel('Damping rate $|\Omega|$')
plt.title('Complex eigenfrequencies of modes')
plt.tight_layout()
plt.show()


I am working on the file "Consul.ipynb".
The first task is:
Getting a key/value with python-consul
For full documentation, see python-consul documentation
Can you write Python code for it?

from consul import Consul

c = Consul('192.168.59.103')

index, data = c.kv.get('key1')

print(data['Value'].decode('utf8'))


I am working on the file "Content Analysis.ipynb".
The first task is:
Actors
Can you write Python code for it?

nz_actors_count = content_stats.Title_stats(nz_data).top_actors(21)
us_actors_count = content_stats.Title_stats(us_data).top_actors(21)

print '
'
print '
Actor Name	Count of Titles with Actor Name
'

for tup in nz_actors_count[1:]:
    print '
'
    print '        '
    print '            ', tup[0].lstrip()
    print '        	'
    print '    '
    print '              ', tup[1]
    print '        	'
    print '
'
print '
'

print
print '====================='

print '
'
print '
Actor Name	Count of Titles with Actor Name
'
for tup in us_actors_count[1:]:
    print '
'
    print '        '
    print '            ', tup[0].lstrip()
    print '        	'
    print '    '
    print '              ', tup[1]
    print '        	'
    print '
'

print '
'

print '
'
print '
	NZ Service		US Service
'
print '
Rank	Actor Name	Count of Titles with Actor Name	Actor Name	Count of Titles with Actor Name
'

for x in range(1,len(nz_actors_count)):
    print '
'
    print ''
    print str(x)
    print '	'
    print ''
    print nz_actors_count[x][0].lstrip()
    print '	'
    print ''
    print nz_actors_count[x][1]
    print '	'
    print ''
    print us_actors_count[x][0].lstrip()
    print '	'
    print ''
    print us_actors_count[x][1]
    print '	'
    print '
'

print '
'

print '
'
print '
New Zealand Service
'
print '
Movie Name	IMDb Score
'
nzx = []
for tup in top_nz_titles[:21]:
    print '
'
    print '',tup[0],'	','', tup[1],'	'
    print '
'
    nzx.append(tup[0])

print '
'

print
print
print '==============='

print '
'
print '
Movie Name	IMDb Score
'
usx = []
for tup in...


I am working on the file "Content Analysis.ipynb".
The first task is:
For all charts we present; the data we use is available for exploration and reuse.  There is a 'Play with this data' link at the bottom right hand side of each chart.
If you are running these iPython notebooks yourself please note we are embedding the graphs rather than creating them.  You can uncomment the creation code and comment out the embed code if you are using the notebooks interactively.
Library Size
The simplest test we can carry out is looking at library size.  We are looking at count of titles not count of discreet episodes or total viewing time.  Its not an unusfule test to begin to understand the libraries.
Can you write Python code for it?

total_titles_nz = len(nz_data)
total_titles_us = len(us_data)

data = (
    [Bar( x = ['NZ', 'USA'],
          y = [total_titles_nz, total_titles_us],
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )]
    )

layout = Layout(
    title ='Netflix Library Comparison- USA vs NZ - June 2015',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'Geographic Service'),

    )

fig = Figure(data=data, layout=layout)

# Run this to generate the plot.ly plot for yourself, once created we will embed it
# py.iplot(fig, filename = 'Netflix-Library-Comparison-June-2015')

# Run this to embed the plot after creation, this is necessary for rendering in Github in particular.

HTML('''

Link to interactive chart and data

'''
)


I am working on the file "Content Analysis.ipynb".
The first task is:
It appears the NZ library, while smaller has a greater proportion of higher quality content.
New Zealand as Country of Origin
We can see what New Zealand content is represented within our samples. This includes anything that has the identifier New Zealand in 'Country' via IMDb.  This can include co-productions.
Can you write Python code for it?

# test
nz_origin_nz = content_stats.Title_stats(nz_data).nz_origin()
nz_origin_us = content_stats.Title_stats(nz_data).nz_origin()

data = (
    [Bar( x = ['NZ', 'USA'],
          y = [nz_origin_nz, nz_origin_us],
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )]
    )

layout = Layout(
    title ='Netflix - Count of Titles with Country of Origin as New Zealand via IMDb',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'Geographic Service'),

    )

fig = Figure(data=data, layout=layout)

# py.iplot(fig, filename = 'Netflix-Library-Comparison-Country-June-2015')

HTML('''

    Link to interactive chart and data

''')


I am working on the file "Content Analysis.ipynb".
The first task is:
It seems NZ might have a smaller library but better quality (based on IMDB rankings).  Though on an absolute value this makes some sense.  We can look at a distribution to get a better understanding.
Can you write Python code for it?

hist_nz_titles = content_stats.Title_stats(nz_data).ratings_distribution()
hist_us_titles = content_stats.Title_stats(us_data).ratings_distribution()

trace1 = Histogram(
    x = hist_nz_titles,
    opacity=0.75,
    name = 'Count of NZ Titles'
)
trace2 = Histogram(
    x = hist_us_titles,
    opacity = 0.75,
    name = 'Count of US Titles'
)
data = Data([trace2, trace1])
layout = Layout(
    barmode ='overlay',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'IMdB Score (rounded down)'),
    title = 'Netflix Library Comparison June 2015 Distribution of IMDB scores'
)
fig = Figure(data=data, layout=layout)

# py.iplot(fig, filename = 'Netflix-Library-Comparison-June-2015-Distribution of IMDB scores')

HTML('''

    Link to interactive chart and data

''')

titles_score_count_nz = Counter(hist_nz_titles)
titles_score_count_us = Counter(hist_us_titles)

# make relative
nz_tot = sum(titles_score_count_nz.values())
titles_score_count_nz_relative = { k:round((float(v)/float(nz_tot))*100, 1) for (k,v) in titles_score_count_nz.items()}

us_tot = sum(titles_score_count_us.values())
titles_score_count_us_relative = { k:round((float(v)/float(us_tot))*100,1) for (k,v) in titles_score_count_us.items()}

nz_x_list = []
nz_y_list = []

for score, count in titles_score_count_nz_relative.iteritems():
    nz_x_list.append(score)
    nz_y_list.append(count)

us_x_list = []
us_y_list = []

for score, count in...


I am working on the file "Content Analysis.ipynb".
The first task is:
New Zealand may have a smaller catalgoue but it does seem to have a marginally higher average quality than the US.  We can look at what the top movies are (based on IMDb/OMDb ratings) between the two countries.
Top Movies
This give an interesting understanding of the top movies.  At first look we can se the titles are very different.
Can you write Python code for it?

top_nz_titles = content_stats.Title_stats(nz_data).top_movies(25)
top_us_titles = content_stats.Title_stats(us_data).top_movies(25)

print 'Top NZ Titles'
print

# Truncate to 5
for tup in top_nz_titles[:5]:
    print 'Title :', tup[0]
    print 'Rating: ', tup[1]
    print '========='

print
print
print 'Top US Titles'
print

# Truncate to 5
for tup in top_us_titles[:5]:
    print 'Title :', tup[0]
    print 'Rating: ', tup[1]
    print '========='

print
print

print 'Average score of top 25 Titles'
print
print 'NZ top 25 average'

score = 0
count = 0
for tup in top_nz_titles:
    count += 1
    score += tup[1]
print score/count

print

print 'US top 25 average'
score = 0
count = 0
for tup in top_us_titles:
    count += 1
    score += tup[1]
print score/count


I am working on the file "Content Analysis.ipynb".
The first task is:
The Bieber Effect
Across the top 25 both NZ and US are close, with the NZ library taking a slight lead.  How do we compare in terms of the bottom ranked content.
Can you write Python code for it?

bottom_nz_titles = content_stats.Title_stats(nz_data).bottom_movies(25)
bottom_us_titles = content_stats.Title_stats(us_data).bottom_movies(25)

print 'Bottom ranked NZ title', bottom_nz_titles[-1][0], ':' , bottom_nz_titles[-1][1]

poster = Image(nz_data[quote(bottom_nz_titles[-1][0])]['Poster'])
display(poster)

print 'Bottom ranked US title', bottom_us_titles[-1][0], ':' , bottom_us_titles[-1][1]

poster = Image(us_data[quote(bottom_us_titles[-1][0])]['Poster'])
display(poster)

print
print

print 'Average score of bottom 25 Titles'
print
print 'NZ bottom 25 average'

score = 0
count = 0
for tup in bottom_nz_titles:
    count += 1
    score += tup[1]
print score/count

print

print 'US bottom 25 average'
score = 0
count = 0
for tup in bottom_us_titles:
    count += 1
    score += tup[1]
print score/count


I am working on the file "Content Analysis.ipynb".
The first task is:
The smaller New Zealand library makes this a bit tricker to understand which service has newer and older content when put against the larger US library.
Can you write Python code for it?

total_us_titles = sum(us_y_list)
us_y_list = [(float(count)/(float(total_us_titles)) * 100) for count in us_y_list]

total_nz_titles = sum(nz_y_list)
nz_y_list = [(float(count)/(float(total_nz_titles)) * 100) for count in nz_y_list]

trace1 = (
    Bar( x = us_x_list,
          y = us_y_list,
        name = 'USA',
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )
    )

trace2 = (
    Bar( x = nz_x_list,
          y = nz_y_list,
        name = 'NZ',
            marker = Marker(
            color = 'rgba(255, 144, 33, 0.6)')
        )
    )

layout = Layout(
    title ='Netflix - Year of Titles First Release - US and NZ June 2015 - Percentage',
    yaxis = YAxis(title = 'Percentage of Titles'),
    xaxis = XAxis(title = 'Year'),
    barmode='group'

    )

data = Data([trace1, trace2])

fig = Figure(data=data, layout=layout)

py.iplot(fig, filename = 'Relative- Netflix-Library-Comparison-Release-US-NZ-June-2015')


I am working on the file "Content Analysis.ipynb".
The first task is:
We can do the same for the USA catalogue.
Can you write Python code for it?

release_year = content_stats.Title_stats(us_data).year_first_release_count()

x_list = []
y_list = []

for year, count in release_year.iteritems():
    x_list.append(year)
    y_list.append(count)

data = (
    [Bar( x = x_list,
          y = y_list,
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )]
    )

layout = Layout(
    title ='Netflix - Year of Titles First Release - US June 2015',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'Year'),

    )

fig = Figure(data=data, layout=layout)

# py.iplot(fig, filename = 'Netflix-Library-Comparison-Release-US-June-2015')
HTML('''

        Link to interactive chart and data

''')


I am working on the file "Content Analysis.ipynb".
The first task is:
We can represent this graphically again.
Can you write Python code for it?

trace1 = Bar(
        y=['Service'],
        x=[unique_nz],
        name='Unique to NZ',
        orientation = 'h',

        marker = Marker(
            color = 'rgba(255,127,14,1.0)'        )

        )

trace2 = Bar(
        y=['Service'],
        x=[common],
        name='Common',
        orientation = 'h',
            marker = Marker(
            color = 'rgba(44,160,44,1.0)'        )

        )

trace3 = Bar(
        y = ['Service'],
        x = [unique_us],
        name = 'Unique to USA',
        orientation = 'h',
            marker = Marker(
            color = 'rgba(39,119,180,1.0)'        )

        )

data = Data([trace1, trace2, trace3])
layout = Layout(
    barmode='stack',
    title = 'Content Unique to and Common Between US and NZ Netflix Services - June 2015',
    yaxis = YAxis(title = ''),
    xaxis = XAxis(title = 'Count of Titles')
    )

fig = Figure(data=data, layout=layout)

# used to create
#py.iplot(fig, filename = 'Netflix-Library-Comparison-June-2015-Uniqueness of Content')

# used to embed

HTML('''

    Link to interactive chart and data

''')


I am working on the file "Content Analysis.ipynb".
The first task is:
We could potentially display this visually.
We need to do some trickery as our dictionaries have quoted title names as the keys, so 'The%20Pink%20Panther' instead of 'The Pink Panther', so we need to 'quote' our moviename to get an image.
Top NZ Title Visually
Can you write Python code for it?

#truncate to top 5
for tup in top_nz_titles[:5]:
    print 'Title :', tup[0]
    print 'Title :', tup[1]
    try:
        poster = Image(nz_data[quote(tup[0])]['Poster'])
        display(poster)
    except:
        print '''
        |
        | No Poster
        |
        '''


I am working on the file "Content Analysis.ipynb".
The first task is:
We're pretty happy with these percentages for getting an understanding of the content that is available from a quality perpective.  We might not be able to be absolutely, absolute, but good enough.
We can now look at the difference in quality.
In our assessment of quality we are not looking at Netflix ratings, we are looking at the ratings held in IMDb.  It could be that Netflix have content more suited to its customers and they would rate titles higher.  Those rating IMDb may be skewed in a particular way as they may have more interest in esoteric aspects of movies and content.  Perhaps another useful metric would be Rotten Tomatoes scores, though we don't have complete information on this and we were declied access to the Rotten Tomatoes API.
Average Score
Can you write Python code for it?

# Average IMDB score of NZ geographic content
nz_avg_score = content_stats.Title_stats(nz_data).average_score()

# Average IMDB score of NZ geographic content
us_avg_score = content_stats.Title_stats(us_data).average_score()

print 'NZ Average Score via OMDB: ', round(nz_avg_score,2)
print 'US Average Score via OMDB: ', round(us_avg_score,2)


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
Did you guess right?
Note that odeint returns the values of both the function theta[0] = $\theta(t)$ and its derivative theta[1] = $\omega(t)$ at each time step.  So we now have the solutions to both of our first-order equations.  This is kind of nice because in one move we have obtained both the position $\theta(t)$ and angular velocity $\omega(t)$ as a function of time.  We could use/plot either or both.  Let's plot the position and compare it to our previous approximate result.
Can you write Python code for it?

plt.plot(t,theta_app(t),'b-',label="Approximate solution")
plt.plot(t,theta[:,0],'r--',label="Numerical solution")
plt.xlabel(r'$t (seconds)$',fontsize=20)
plt.ylabel(r'$\theta (rad)$',fontsize=20)
plt.ylim(-1,1.5)
plt.grid()
plt.legend()
plt.show()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
How different is the result from Euler's method?  We'll explore the variation between these methods in the accompanying exercises.
Midpoint Method
The midpoint method is even more accurate than the previous two because it uses the slope at the midpoint of the time interval to estimate the forward progress of the velocity.
$$
v_{i+1} = v_i + a_i\Delta t
$$
$$
y_{i+1} = y_i + \frac{1}{2}(v_{i}+v_{i+1})\Delta t
$$
These approaches would give an exact solution if the time interval $\Delta t$ we used were infinitesimal in size, because then we would be doing the exact integrals that determine the velocity and the position.
Another way to say this is that values we use in the Euler, Euler-Cromer and Midpoint method equations equal the instantaneous values in the limit of an infinitesimal time interval. But there's a problem with trying to do this: we'd have an infinite number of data points where we'd have to calculate $v$ and $y$!
So we want to use a time interval that is short enough to approximate the answer adequately, but long enough that we don't have an unreasonable number of data points.
So how short is short enough? That depends on how fast $v$ and $y$ are changing. A simple test that you can try when programming is to change $\Delta t$ (such as we did with the interact example above) and see if your solution changes at all.
The Runge-Kutta Family of Methods
The Runge-Kutta methods are a series of numerical methods for solving differential equations and systems of differential equations of which Euler and the Midpoint methods are a subset.  The more refined the approximation procedure used at each step determines the "order" of the method.  Euler is an order 1 method, Midpoint is order 2, and the most commonly used Runge-Kutta Method, RK4, is of order 4.  The distinguishing characteristic of each order method is the error per step associated with it.
RK4 takes the form
$$
y_{i+1} = y_i + \frac{\Delta t}{6} (k_1 + 2k_2 + 2k_3 + k_4)
$$
where
$$k_1 = f(y_i, t_i) $$
$$k_2 = f(y_i + \frac{\Delta t}{2}k_1,...
Can you write Python code for it?

# we want to solve the system dy/dt = f(y, t)
def derivPP(y,t):

    alpha = 0.7
    gamma = 0.5
    beta = 0.007
    eps = 0.1

    Zi = y[0]
    Li = y[1]

    dZdt = alpha*Zi - beta*Zi*Li
    dLdt = eps*beta*Zi*Li - gamma*Li

    return np.array([ dZdt, dLdt ])

#50 year span
t = np.arange(0.,50.+0.1,0.1)

#initial conditions
Z0 = 200
L0 = 50

#pass the function deriv to the solver with initial values and integration variable
result = odeint(derivPP,np.array([Z0,L0]),t)

Z = result[:,0]
L = result[:,1]

plt.plot(t,Z,label="Zebra population")
plt.plot(t,L,label="Lion population")
plt.xlabel("time (years)")
plt.ylabel("Population")
plt.legend()
plt.show()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
How does the percent difference in $y_{max}$ change as a function of the size of the timesteps?

For what value of the timestep does the percent difference just barely exceed 10%?

Set the timestep so that the percent difference is approximately 1%.  For what initial velocity $v_0$ does the ball return to earth at $t$ = 15 s?

Euler-Cromer Method
Varying the timesteps let's us improve the accuracy of our result, but to do an even better job, we could try modifying the method for calculating the new positions and velocities.  Cromer added the idea of calculating the new position using the just calculated new velocity (at the end of the interval) to Euler’s method, creating what is known as the Euler-Cromer Method.
$$
v_{i+1} = v_i + a_i\Delta t
$$
$$
y_{i+1} = y_i + v_{i+1}\Delta t
$$
Did you catch the difference in the last term of the second line?  Let's implement it.
Can you write Python code for it?

def EulerCromerFreefall():
    tmax = 10 # maximum time
    dt = 0.1 # time step
    a = -9.81 #g in m/s^2
    nsteps = int(tmax/dt)
    t = np.zeros(nsteps)
    v = np.zeros(nsteps)
    y = np.zeros(nsteps)

    # set initial conditions, redundant in this example
    t[0] = 0.
    v[0] = 0.

    for i in range(len(t)-1):
        t[i+1] = t[i] + dt
        v[i+1] = v[i] + dt * a
        y[i+1] = y[i] + dt * v[i+1] #use just calculated v value to compute y

    return t,v,y

tec,vec,yec = EulerCromerFreefall()

plt.plot(tec,yec,"r-",label="y(t)")
plt.plot(tec,vec,"b-",label="v(t)")
plt.xlabel("time (s)",fontsize=20)
plt.ylabel("y(t),v(t)",fontsize=20)
plt.grid()
plt.title("Euler-Cromer")
plt.legend(loc="best", fontsize=20)
plt.show()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
Introduction
Differential equations relate functions of one or more variables to their derivatives.  "Ordinary" differential equations refer to those functions that have only one independent variable. This is in contrast to "partial" differential equations, which may depend on several different independent variables.
There are many examples of ODEs in physics.  Some of them are linear equations with closed form analytic solutions that we can solve by hand.  Nonlinear equations are harder to solve.  Some have solutions that can be expressed analytically in terms of series expansions or integrals, while others have no closed-form solutions and must be solved numerically.  Computers and libraries of ODE solving algorithms make it possible to explore the dynamics of nonlinear equations quantitatively.
We'll explore a few interesting examples in this tour.
The Simple Pendulum as an example
Solving for the motion of a simple pendulum involves a second order differential equation that cannot be solved analytically.  It is ripe to be solved numerically.  Let's review the physics:
<img src="SimplePendulum.png" width=400>
If you use Newton's second law to write the sum of the forces on the mass, $m$, you get
$\sum\vec{F} = \vec{T}+\vec{w} = m\vec{a}$
which is a vector expression that needs to be broken up into its components.  Since the length of the pendulum is fixed, the acceleration in the radial direction must be zero, and we need only consider accelerations tangential to the path of the mass, $m$.  Given this, it makes the most sense to use cylindrical coordinates.
$F_{radial} = T - mg\cos\theta = ma_{radial} = 0$
$F_{tangential} = -mg\sin\theta = ma_{tangential}$
The linear acceleration along the tangential direction is related to $\theta$ through the arc-length $s$ that the mass makes as it swings:
$$s = \ell\theta$$
$$
v_{tangential} = \frac{ds}{dt} = \ell \frac{d\theta}{dt}
$$
$$
a_{tangential} = \frac{d^2s}{dt^2} = \ell \frac{d^2\theta}{dt^2}
$$
which gives the following differential equation that we...
Can you write Python code for it?

g = 9.8 #m/s^2
ell = 0.5 #m
th0 = 20.*np.pi/180 # 20 degrees, but must use radians

t = np.arange(0.,10.+0.01,0.01)
theta_app = lambda t: th0*np.cos(np.sqrt(g/ell)*t)

plt.plot(t,theta_app(t),label='Approximate solution')
plt.xlabel(r'$t (seconds)$',fontsize=20)
plt.ylabel(r'$\theta (rad)$',fontsize=20)
plt.ylim(-1,1.5)
plt.grid()
plt.legend()
plt.show()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
It looks like the approximate solution does a decent job for the first few cycles, but starts to diverge later on.  What if we start it off at a bigger initial angle?
Can you write Python code for it?

th0 = 50.*np.pi/180 # 50 degrees

th_init = np.array([th0, omega0]) # initial values
theta = odeint(deriv,th_init,t)

plt.plot(t,theta_app(t),'b-',label="Approximate solution")
plt.plot(t,theta[:,0],'r--',label="Numerical solution")
plt.xlabel(r'$t (seconds)$',fontsize=20)
plt.ylabel(r'$\theta (rad)$',fontsize=20)
plt.ylim(-1.,1.5)
plt.grid()
plt.legend()
plt.show()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
Note, this same function can be written more compactly as
python
    def deriv(vals,t):  # return derivatives of the array vals
        g = 9.8 #m/s^2
        ell = 0.5 #m
        return np.array([ vals[1], -(g/ell)*np.sin(vals[0]) ])
But writing it the way we did in the code cell makes it very explicit what each step involves.  The odeint solver will take the derivative function, along with an array of current values for $[\theta,\omega]$ and return the result of 1) differentiating them via deriv, which produces $[d\theta/dt,d\omega/dt]$ and then 2) integrating $[d\theta/dt,d\omega/dt]$ at each time step, to give back $[\theta(t),\omega(t)]$.
Many numerical ODE solvers do it exactly the same way (e.g. MATLAB).
Now we import odeint from scipy.integrate, set up our initial conditions, and solve:
Can you write Python code for it?

from scipy.integrate import odeint

#Try the help for more info
#help(odeint)

t = np.arange(0.,10.+0.01,0.01)
th0 = 20.*np.pi/180 # 20 degrees, but must use radians
omega0 = 0. #starting from rest, omega = 0

th_init = np.array([th0, omega0]) # initial values

#pass the function deriv to the solver with initial values and integration variable
theta = odeint(deriv,th_init,t)


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
Now use the sliders to explore this system.  What effect does changing the parameters have on the shape of the plot?

Breaking into the black box
In order to be able to solve for the motion of the simple pendulum using odeint, we had to convert the single second-order ODE into two first order ODEs.  Let's peer inside scipy's black box to understand why.
Euler's Method
The most basic numerical algorithm for computing the solutions to ODEs was invented by Leonhard Euler in the 18th century - way before microprocessors.  Back in those days, the solutions were cranked out by human computers.
Free-fall as an example
Let's start by thinking about the familiar free-fall problem where velocity is the derivative of position, and acceleration is the derivative of velocity. We can write this as a single second-order differential equation:
$$
a(t) = \frac{d^2y}{dt^2}
$$
OR as two first order differential equations:
$$
a(t) = \frac{dv}{dt}, \hspace{1cm} v(t) = \frac{dy}{dt}.
$$
Once the problem is first order, we can integrate instead of differentiating. Integrating the acceleration tells us the velocity at some later time if we know it at some earlier time; we can derive the position in a similar stepwise fashion from the velocity.
Starting at a time $t_0$ at some known position $y(t_0)$ and with some known velocity $v(t_0)$, we can calculate the change in velocity in going from a time $t_0$ to a time $t_1 = t_0 + \Delta t$
$$
v(t_1) = v(t_0) + a(t_0)\Delta t
$$
and for the position:
$$
y(t_1) = y(t_0) + v(t_0)\Delta t
$$
This is the Euler Method.  It is based on a tangent line approximation to the function in question.
Let's denote $v(t_1)$ as $v_1$ and so on. Applying the method successively on subsequent time steps up to $n$ we can write $t_{i+1} = t_i + \Delta t$ and $t_{i} = t_0 + i\Delta t$, so that
$$
v_{i+1} = v_i + a_i\Delta t
$$
$$
y_{i+1} = y_i + v_i\Delta t
$$
Note that the velocity/position depend on the previous velocity/position, the acceleration/velocity and the time interval. So the acceleration need...
Can you write Python code for it?

def EulerFreefall():
    tmax = 10 # maximum time
    dt = 0.1 # time step
    a = -9.81 #g in m/s^2
    nsteps = int(tmax/dt)
    t = np.zeros(nsteps)
    v = np.zeros(nsteps)
    y = np.zeros(nsteps)

    # set initial conditions, redundant in this example
    t[0] = 0.
    v[0] = 0.

    for i in range(len(t)-1):
        t[i+1] = t[i] + dt
        v[i+1] = v[i] + dt * a
        y[i+1] = y[i] + dt * v[i]

    return t,v,y

te,ve,ye = EulerFreefall()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
Now we can plot the position and velocity as a function of time:
Can you write Python code for it?

plt.plot(te,ye,"r-",label="y(t)")
plt.plot(te,ve,"b-",label="v(t)")
plt.xlabel("time (s)",fontsize=20)
plt.ylabel("y(t),v(t)",fontsize=20)
plt.grid()
plt.legend(loc="best", fontsize=20)
plt.show()


I am working on the file "Content/ODEs/ODEsTour.ipynb".
The first task is:
What do you think the shape of theta is?  Try to figure it out and then check your answer.
Can you write Python code for it?


I am working on the file "Contextual-Policy.ipynb".
The first task is:
The Contextual Bandits
Here we define our contextual bandits. In this example, we are using three four-armed bandit. What this means is that each bandit has four arms that can be pulled. Each bandit has different success probabilities for each arm, and as such requires different actions to obtain the best result. The pullBandit function generates a random number from a normal distribution with a mean of 0. The lower the bandit number, the more likely a positive reward will be returned. We want our agent to learn to always choose the bandit-arm that will most often give a positive reward, depending on the Bandit presented.
Can you write Python code for it?

class contextual_bandit():
    def __init__(self):
        self.state = 0
        #List out our bandits. Currently arms 4, 2, and 1 (respectively) are the most optimal.
        self.bandits = np.array([[0.2,0,-0.0,-5],[0.1,-5,1,0.25],[-5,5,5,5]])
        self.num_bandits = self.bandits.shape[0]
        self.num_actions = self.bandits.shape[1]

    def getBandit(self):
        self.state = np.random.randint(0,len(self.bandits)) #Returns a random state for each episode.
        return self.state

    def pullArm(self,action):
        #Get a random number.
        bandit = self.bandits[self.state,action]
        result = np.random.randn(1)
        if result > bandit:
            #return a positive reward.
            return 1
        else:
            #return a negative reward.
            return -1


I am working on the file "Contextual-Policy.ipynb".
The first task is:
Training the Agent
We will train our agent by getting a state from the environment, take an action, and recieve a reward. Using these three things, we can know how to properly update our network in order to more often choose actions given states that will yield the highest rewards over time.
Can you write Python code for it?

tf.reset_default_graph() #Clear the Tensorflow graph.

cBandit = contextual_bandit() #Load the bandits.
myAgent = agent(lr=0.001,s_size=cBandit.num_bandits,a_size=cBandit.num_actions) #Load the agent.
weights = tf.trainable_variables()[0] #The weights we will evaluate to look into the network.

total_episodes = 10000 #Set total number of episodes to train agent on.
total_reward = np.zeros([cBandit.num_bandits,cBandit.num_actions]) #Set scoreboard for bandits to 0.
e = 0.1 #Set the chance of taking a random action.

init = tf.global_variables_initializer()

# Launch the tensorflow graph
with tf.Session() as sess:
    sess.run(init)
    i = 0
    while i < total_episodes:
        s = cBandit.getBandit() #Get a state from the environment.

        #Choose either a random action or one from our network.
        if np.random.rand(1) < e:
            action = np.random.randint(cBandit.num_actions)
        else:
            action = sess.run(myAgent.chosen_action,feed_dict={myAgent.state_in:[s]})

        reward = cBandit.pullArm(action) #Get our reward for taking an action given a bandit.

        #Update the network.
        feed_dict={myAgent.reward_holder:[reward],myAgent.action_holder:[action],myAgent.state_in:[s]}
        _,ww = sess.run([myAgent.update,weights], feed_dict=feed_dict)

        #Update our running tally of scores.
        total_reward[s,action] += reward
        if i % 500 == 0:
            print("Mean reward for each of the " + str(cBandit.num_bandits) + " bandits: " + str(np.mean(total_reward,axis=1)))
        i+=1
for a in range(cBandit.num_bandits):
    print("The agent thinks action " + str(np.argmax(ww[a])+1) + " for bandit " + str(a+1) + " is the most promising....")
    if np.argmax(ww[a]) == np.argmin(cBandit.bandits[a]):
        print("...and it was right!")
    else:
        print("...and it was wrong!")


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
1. Load the crystallographic and docked protein-ligand structures in pymol
Docking programs often return as output only the coordinates of the docked ligand.
Therefore, in order to be able to perform the protein alignment,
we first need to load both the protein structure used for docking and the docked ligand structure in pymol
and merge the coordinates in a temporary PDB file, complex.pdb
Input files:
- docked_lig_file : name of file containing the coordinates of the docked ligand
- protein_file : name of file containing the the protein structure used for docking
- crystal_file: name of file containing the crystal structure of the complex
Can you write Python code for it?

# Function to merge the ligand and protein PDB files in a unique PDB file
def merge_prot_lig_pdbs(protein_file, docked_lig_file, output_file = 'complex.pdb'):
    cmd.reinitialize()
    cmd.load(protein_file, 'protein')
    cmd.load(docked_lig_file, 'docked_lig')
    cmd.save(output_file, "protein or docked_lig")

docked_lig_file = "data/docked_2c6e_SKE_pH74_netcharge1.pdb"
protein_file = "data/aurka_protein_2c6e.pdb"
crystal_file = "data/5dt0.pdb"

merge_prot_lig_pdbs(protein_file, docked_lig_file)


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
<img src="./figures/1C9.png" width=800>
In this figure, the missing groups are highlighted in green in the docked pose while the truncation point in the crystal structure is highlighted in red.
Align docking and crystal structures prior to the RMSD calculation
The function above takes as input pre-aligned ligand structures. In this section, we show how one could align crystallographic protein-ligand structure to the docked structure and extract the ligand coordinates from the crystal structure using Pymol.
Can you write Python code for it?


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
Alignment for Structures with Multiple Chains/Domains and Different/Multiple Ligands
The structure alignment and ligand extraction becomes more complicated for crystal structures that contain dimers, or higher oligomers, multi-protein complexes, multiple and different ligands!
In these case, the code below can be used. It reads as input the crystallographic and docked structures.
The function does the following:

check for the presence of organic molecules in the crystal structure.
check for the presence of organic molecules in each of the chains
if a chain contains an organic molecule, then that chain is aligned to the docked structure based on the C-alpha backbone atoms.
calculate the distance between the organic molecule in the aligned crystal structure and the organic molecule in the aligned docked structure.
if the distance is lower than a  given threshold (max_dist), the coordinates of the organic molecule from the aligned crystal structure are stored in a PDB file.
Can you write Python code for it?

from pymol import stored
import numpy as np

def find_organic_ligs(obj_name):
    cmd.select(f"ligs", f"{obj_name} and organic")
    stored.residues = set()
    cmd.iterate(f"ligs", 'stored.residues.add(resn)')
    ligs_resname = list(stored.residues)
    return list(set(ligs_resname) - set(['ACE', 'NME']))

def extract_lig_from_aligned_crystal(crystal_file, docked_file, max_dist = 7):
    count = 0
    # load structures
    cmd.reinitialize()
    cmd.load(docked_file, "docked")
    cmd.load(crystal_file, "crystal")
    # identify organic molecules in both the docked and crystal structures
    lig_docked_name = find_organic_ligs('docked')[0]
    lig_crystal_names = find_organic_ligs('crystal')
    # check the number of chains in the crystal structure
    chains = cmd.get_chains("crystal")
    print(f"The crystal structure {crystal_file} contains {len(chains)} chains and {len(lig_crystal_names)} ligands {lig_crystal_names}")
    # loop over the organic molecules
    for lig_crystal_name in lig_crystal_names:
        print(f"\n{lig_crystal_name}")
        # loop over the chains
        for chain in chains:
            sel = cmd.select(f"ligs{chain}", f"crystal and chain {chain} and resn {lig_crystal_name}")
            if sel == 0:
                continue
            else:
                cmd.align(f"crystal and chain {chain} and name CA", "docked and name CA")
                cmd.save("data/crystal_aligned.pdb", "crystal")
                cmd.select(f"ligs{chain}", f"crystal and chain {chain} and resn {lig_crystal_name}")
                stored.residues = set()
                cmd.iterate(f"ligs{chain}", 'stored.residues.add(resv)')
                lig_res = list(stored.residues)
                # measure the distance between the center of mass of the ligands in docked and crystal structures.
                center2 = cmd.centerofmass(f"docked and resn {lig_docked_name}")
                # there may be multiple ligand copies in the crystal structure. Loop over them.
                for lig1 in lig_res:
    ...


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
CalcLigRMSD
CalcLigRMSD calculates the Root-mean-square deviation (RMSD) between two prealigned molecules.
This jupyter notebook shows examples of how to use the CalcLigRMSD function.
This function is particularly useful in 2 cases:

when the atom names in the compared structures do not match, i.e. when the same atoms present different names in the coordinate files.
when one or both structures have missing atoms. This occurs, for example, in crystallographic structures when some atoms are not well defined in the electron density map.

Below, there are also examples of how to align protein structures and extract ligand coordinates using a python/pymol function. The code below may be used both when the protein structures have multiple chains/domains and when protein structures present multiple copies of the ligand of interest, either in the binding pocket or on the surface.
How to use the CalcLigRMSD function:
Can you write Python code for it?


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
Example where there are missing atoms in the crystal structure
Two crystal structures are available for the EGFR/1C9 complex. However, the electron density of the 1C9 is not well defined in any of the structures (see figure below). In such cases, one can calculate the RMSD between the docked and crystal structures by excluding the atoms which are not defined in the crystal structure.
This is achived by calculating the RMSD for the maximum common substructure.
Can you write Python code for it?

docked_ligand = Chem.MolFromPDBFile("data/docked_3w2p_1C9_pH74_netcharge1.pdb")
crystal_ligand = Chem.MolFromPDBFile("data/4i23_1C9.pdb")
rmsd2 = CalcLigRMSD(docked_ligand, crystal_ligand, rename_lig2 = True, output_filename = 'data/1C9_renamed.pdb')
print(f"RMSD: {rmsd2:.2f}")

docked_ligand = Chem.MolFromPDBFile("data/docked_3w2p_1C9_pH74_netcharge1.pdb")
crystal_ligand = Chem.MolFromPDBFile("data/4i24_1C9.pdb")
rmsd2 = CalcLigRMSD(docked_ligand, crystal_ligand, rename_lig2 = True, output_filename = 'data/1C9_renamed.pdb')
print(f"RMSD: {rmsd2:.2f}")


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
Now we can reinitialize Pymol and load both the docked and crystal complexes in Pymol. The step above can be skipped if the docked complex is directly available.
Can you write Python code for it?

# load both the docked and crystallographic protein-ligand structures
cmd.reinitialize()
cmd.load("complex.pdb", "docked")
cmd.load(crystal_file, "crystal")


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
Now, you can calculate the RMSD:
Can you write Python code for it?

docked_ligand = Chem.MolFromPDBFile(docked_lig_file)
crystal_ligand = Chem.MolFromPDBFile("data/lig_crystal_aligned.pdb")
rmsd2 = CalcLigRMSD(docked_ligand, crystal_ligand, rename_lig2 = True, output_filename = 'data/1C9_renamed.pdb')
print(f"RMSD: {rmsd2:.2f}")


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
RMSD between Pre-aligned Ligands with not Matching Atom Names
Basic Example
Complex between the kinase protein AURKA and the JVE ligand
Comparison between the crystal structure of JVE (pink, PDB code: 4UZH) and the docked pose of JVE (yellow).
JVE was docked into the the AURKA structure 2C6E using AutodockVina.
Can you write Python code for it?

docked_ligand = Chem.MolFromPDBFile("data/docked_2c6e_JVE_pH74_netcharge1.pdb")
crystal_ligand = Chem.MolFromPDBFile("data/4uzh_JVE.pdb")
rmsd2 = CalcLigRMSD(docked_ligand, crystal_ligand, rename_lig2 = True, output_filename = 'data/JVE_renamed.pdb')
print(f"RMSD: {rmsd2:.2f}")


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
The AKI ligand contains four symmetric groups (highlighted in the Figure below). Again, the code takes symmetry into consideration!

Example for a very symmetrical molecule

BEG is a very symmetrical molecule (see this ref). Not only it does it contain two benzene groups but the entire molecule is symmetrical. We have downloaded the crystal structure 1D4I and extracted the coordinates of the BEG molecule (1d4i_BEG.pdb). In a copy (BEG_test.pdb), we have then renamed the atoms randomly. If the alghoritm is able to correctly match the atoms, then it will return an RMSD of zero. And indeed:
Can you write Python code for it?

docked_ligand = Chem.MolFromPDBFile("data/BEG_test.pdb")
crystal_ligand = Chem.MolFromPDBFile("data/1d4i_BEG.pdb")
rmsd2 = CalcLigRMSD(docked_ligand, crystal_ligand, rename_lig2 = False, output_filename = 'data/AKI_renamed.pdb')
print(f"RMSD: {rmsd2:.2f}")


I am working on the file "Contrib/CalcLigRMSD/Examples_CalcLigRMSD.ipynb".
The first task is:
The N15 ligand contains two symmetric groups (highlighted in the Figure below). The algorithm takes symmetry into consideration: it calculates the RMSD between all possible "symmetrical" combinations and returns the minimum RMSD. The structures with matched atom names corresponding to the lowest RMSD are reported in the figure below.
<img src="./figures/N15.png" width=800>
Complex between the kinase protein AURKA and the AKI ligand
Can you write Python code for it?

docked_ligand = Chem.MolFromPDBFile("data/docked_2c6e_AKI_pH74_netcharge1.pdb")
crystal_ligand = Chem.MolFromPDBFile("data/3m11_AKI.pdb")
rmsd2 = CalcLigRMSD(docked_ligand, crystal_ligand, rename_lig2 = True, output_filename = 'data/AKI_renamed.pdb')
print(f"RMSD: {rmsd2:.2f}")


I am working on the file "Control Flow/The 'if', 'elif', and 'else' Statements.ipynb".
The first task is:
Imagine you've written a magic number program, and the system outputs a message to indicate your correctness. Here, the if checks the most specific condition -> if the user is right. As we move down, we become more general. This is the typical layout of control flow in programming.
Can you write Python code for it?

magic_number = 24
my_guess = 29

if my_guess == magic_number:
    print("Correct!")
elif magic_number == (my_guess-1) or magic_number == (my_guess+1):
    print("So, so close!")
else:
    print("Wrong!")


I am working on the file "Control Flow/The 'if', 'elif', and 'else' Statements.ipynb".
The first task is:
Let's output a message depending on where you are going.
Can you write Python code for it?

my_place = "Grocery store"

if my_place == "work":
    print("Clock out early!")
elif my_place == "Grocery store":
    print("Run those errands!")
else:
    print("Where are you going?")


I am working on the file "Control Flow/The 'if', 'elif', and 'else' Statements.ipynb".
The first task is:
We can nest ifs, elifs, and elses, too!
Can you write Python code for it?

# A simple pet and breed printer
pet = "Dog"
breed = "Labrador"

if pet == "Cat":
    print("Pet: cat")
    if breed == "Siberian":
        print("Breed: Siberian")
    else:
        print("Breed: Not specified clearly")
elif pet == "Dog":
    print("Pet: Dog")
    if breed == "Labrador":
        print("Breed: Labrador")
    elif breed == "Husky":
        print("Breed: Husky")
    else:
        print("Breed: Not specified clearly")
else:
    print("I have no idea what that is!")


I am working on the file "Control Flow/for-statement.ipynb".
The first task is:
Python has a range() function. Range simply allows us to specify a number range like this:
range(inclusive, exclusive)

Where, in a for loop, the item variable takes on every value from the incusive up until the exclusive number.
Can you write Python code for it?

# Ranges start from 0 at default
for num in range(8):
    odd_or_even = None
    if num % 2 == 0:
        odd_or_even = "Even"
    else:
        odd_or_even = "Odd"
    print("{x} is {y}".format(x=num, y=odd_or_even))


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
Basic Length Stats
Aggregated stats for the following metrics:
* number of messages (each messanger bubble is a separate message)
* total length of messages (in number of characters, special ones included)
* message average length (averaging total length by number of messages)
By Sender
Can you write Python code for it?

basicLengthStats = conv.stats.generateStats(IConvStats.STATS_NAME_BASICLENGTH)
basicLengthStats


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
By Day
Can you write Python code for it?

day_basic_stats = conv.stats.generateStatsByYearMonthDay(IConvStats.STATS_NAME_BASICLENGTH)

mplot.plotSingleBasicLengthStatHeatmap(day_basic_stats, 'lenMsgs', 'total', ['2014','2015'])

df = statsUtil.dateRangeTransform(day_basic_stats, {'sender':['total'], 'stat':['numMsgs']})

df.plot()


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
By Hour
Can you write Python code for it?

hours_basic_stats = conv.stats.generateStatsByHour(IConvStats.STATS_NAME_BASICLENGTH, stackStats=True)

mplot.plotBasicLengthStatsByHour(hours_basic_stats,
                                 targetSenders=['s1','s2'],
                                 targetStats=['lenMsgs'])


hour_lexical_stats = conv.stats.generateStatsByHour(IConvStats.STATS_NAME_LEXICAL, stackStats=True)

mplot.plotBasicLengthStatsByHour(hour_lexical_stats, targetStats=['lexicalRichness'], kind='point')


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
By Year and Month
Can you write Python code for it?

month_lexical_stats = conv.stats.generateStatsByYearAndMonth(IConvStats.STATS_NAME_LEXICAL)

mplot.plotRichnessVariation(month_lexical_stats, 'month', yearsToShow=['2014', '2015'])


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
Emoticons Stats
Can you write Python code for it?

emoticons_stats = conv.stats.generateStats(IConvStats.STATS_NAME_EMOTICONS)
emoticons_stats


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
Interactive Visualization [WIP]
Can you write Python code for it?

from ipywidgets import interact, widgets

import matplotlib.pyplot as plt
from IPython.display import display, clear_output

%matplotlib inline

hours_basic_stats = conv.stats.generateStatsByYearAndHour(IConvStats.STATS_NAME_BASICLENGTH, stackStats=True)

years = list(map(int, hours_basic_stats['year'].unique()))

@interact(sender=list(map(str, hours_basic_stats['sender'].unique())),
          stat=list(map(str, hours_basic_stats['stat'].unique())),
          #year=list(map(str, hours_basic_stats['year'].unique())))
          year=(years[0], years[-1]))
def myplot(sender, stat, year):
    mplot.plotSingleBasicLengthStatByYearAndHour(hours_basic_stats, stat,
                                                 yearsToShow=[str(year)],
                                                targetSenders=[sender])
    sns.plt.show()

class Dashboard():
    def __init__(self, data):
        self.data = data
        self.selectors = {}

    def create_selector(self, target_col, description):
        selector = widgets.SelectMultiple(
            options = list(map(str, self.data[target_col].unique())),
            description = description
        )
        self.selectors[target_col] = selector

    def get_selector_values(self, selector_key):
        return list(self.selectors[selector_key].value)

    def run(self, fun):
        for k, selector in self.selectors.items():
            selector.observe(fun, names='value')
            display(selector)

d = Dashboard(hours_basic_stats)
d.create_selector('year', 'Year:')
d.create_selector('sender', 'Sender:')

def my_plot(x):
    sns.plt.clf()
    mplot.plotSingleBasicLengthStatByYearAndHour(hours_basic_stats, 'lenMsgs',
                                                 yearsToShow=d.get_selector_values('year'),
                                                targetSenders=d.get_selector_values('sender'))

d.run(my_plot)


I am working on the file "Conversation Analyzer - Basic Stats.ipynb".
The first task is:
Interval Stats
Can you write Python code for it?

start, end, interval = conv.stats.getIntervalStats()
days = conv.stats.getDaysWithoutMessages()
print("Conversation started: {}".format(str(start)))
print("Conversation ended: {}".format(str(end)))
print("Conversation overall duration: {}".format(interval))

percentage = (len(days)/(interval.days+1))*100
print("{:.0f} days without messages ({:.2f}% of the total days duration)".format(len(days), percentage))


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
By Sender
Can you write Python code for it?

mplot.plotWordsCount(month_stats, ['17','your','youth'], sender='s1')

# get data to plot (target words)
target_words = ['your','youth']
stats_to_plot = statsUtil.transformStats(month_word_count[target_words], 'word', 'val')

mplot._genericFactorPlot(stats_to_plot, 'month', 'val', {}, "Word count", "count", col='year', row='word')


# get data to plot (target words)
target_words = ['17','your','youth']
stats_to_plot = statsUtil.transformStats(hour_word_count[target_words], 'word', 'val')

mplot._genericFactorPlot(stats_to_plot, 'hour', 'val', {}, "", "", row='word')


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Top N Words
Most frequent words, overall (total) and sender specific.
Top Words Total
Can you write Python code for it?

total_word_count = sender_stats.getWordsCount()
total_word_count.sort_values(ascending=False)[:5]


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Top Unbalanced Words
Words with higher frequency difference between senders
Can you write Python code for it?


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Top Words by Sender
Can you write Python code for it?


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Total
Can you write Python code for it?

# plot total for target words
mplot.plotWordsCount(hour_stats, ['17','your','youth'])


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Word Count by Month
Consider words usage along time aggregated by months.
Can you write Python code for it?

# load month stats
month_stats = conv.stats.generateStats(IConvStats.STATS_NAME_WORDCOUNT, groupByColumns=['year', 'month'])
month_word_count = month_stats.wordsCount
month_word_count.head()
#filepath = conv.statsFolder + '\\' + 'wordCount.txt'
#mio.printDataFrameToFile(stats, filepath)


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Word Count by Sender
Consider ovarall word usage comparisong between senders
Can you write Python code for it?

# load sender stats
sender_stats = conv.stats.generateStats(IConvStats.STATS_NAME_WORDCOUNT, ngram_range=(1,1))
sender_word_count = sender_stats.wordsCount
sender_word_count.head(3)


I am working on the file "Conversation Analyzer - Words Stats.ipynb".
The first task is:
Word Trend [WIP]
About words usage change along time.
Can you write Python code for it?

# load month stats
month_stats = conv.stats.generateStats(IConvStats.STATS_NAME_WORDCOUNT, groupByColumns=['year', 'month'])
month_word_count = month_stats.wordsCount


I am working on the file "Converting BioPAX to BEL.ipynb".
The first task is:
Conversion
A file path is given to pybel.from_biopax, which wraps indra.sources.biopax.biopax_api.process_owl.
Can you write Python code for it?

graph = pybel.from_biopax(apoptosis_path)

to_jupyter(graph)


I am working on the file "Converting BioPAX to BEL.ipynb".
The first task is:
Data
The apoptosis pathway is downloaded from WikiPathways to the current user's downloads folder.
Can you write Python code for it?

apoptosis_url = 'https://www.wikipathways.org//wpi/wpi.php?action=downloadFile&type=owl&pwTitle=Pathway:WP254&oldid=95107'

apoptosis_path = os.path.join(os.path.expanduser('~'), 'Downloads', 'WP254_95107.owl')

if not os.path.exists(apoptosis_path):
    urlretrieve(apoptosis_url, apoptosis_path)


I am working on the file "Converting BioPAX to BEL.ipynb".
The first task is:
Dependencies
Can you write Python code for it?

pybel.utils.get_version()

pybel_tools.utils.get_version()

pkg_resources.get_distribution("jnius-indra").version

indra.util.get_version.get_version()


I am working on the file "Converting BioPAX to BEL.ipynb".
The first task is:
Environment
Can you write Python code for it?

%%bash
java -showversion 2>&1 | head -n 4

%%bash
locale

print(sys.version)

print(time.asctime())

os.environ['CLASSPATH']


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
And even further?
Can you write Python code for it?

neuron = LeakyIntegratorSpiking(threshold=0.2, tau_rc=20, spike_height=0.01)

for i in range(steps):
    output[i] = neuron.step(stim[i])

plt.figure(figsize=(14,4))
plt.plot(stim, label='stimulus (J)')
plt.plot(output_nonspiking, label='output (non-spiking)')
plt.plot(output, label='output (spiking)')
plt.xlabel('time steps')
plt.legend()
plt.title(f'spike_height={neuron.spike_height}')
plt.show()


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
And further?
Can you write Python code for it?

neuron = LeakyIntegratorSpiking(threshold=0.2, tau_rc=20, spike_height=0.1)

for i in range(steps):
    output[i] = neuron.step(stim[i])

plt.figure(figsize=(14,4))
plt.plot(stim, label='stimulus (J)')
plt.plot(output_nonspiking, label='output (non-spiking)')
plt.plot(output, label='output (spiking)')
plt.xlabel('time steps')
plt.legend()
plt.title(f'spike_height={neuron.spike_height}')
plt.show()


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
Now let's run the model with some random input just to get some basic behaviour out of the model
Can you write Python code for it?

steps = 500
# use Nengo to generate a random input signal
stim = nengo.processes.WhiteSignal(high=10, period=2, rms=0.3, seed=2).run_steps(steps, dt=0.001)

# make a neuron
neuron = LeakyIntegrator(threshold=0.2, tau_rc=20)

# run the simulation
output_nonspiking = np.zeros(steps)
for i in range(steps):
    output_nonspiking[i] = neuron.step(stim[i])

plt.figure(figsize=(14,4))
plt.plot(stim, label='stimulus (J)')
plt.plot(output_nonspiking, label='output')
plt.xlabel('time steps')
plt.legend()
plt.show()


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
So that's the basic behaviour we want to emulate with a spiking neuron.  The general approach is just to round the output!  We take whatever the neuron model is doing already, but we take the output and add it into an accumulator.  Then we check if the accumulator is above 1.  If it is, we emit a spike and subtract 1 from the accumulator.  If it isn't we don't emit a spike.  Here's the result:
Can you write Python code for it?

class LeakyIntegratorSpiking:
    def __init__(self, threshold, tau_rc=20):
        self.threshold = threshold
        self.tau_rc = tau_rc
        self.v = 0
        self.output_accumulator = 0

    def step(self, J):
        if self.v > self.threshold:
            output = self.v - self.threshold
        else:
            output = 0

        dv = -self.v + output + J
        self.v += dv / self.tau_rc

        self.output_accumulator += output
        output = int(self.output_accumulator)
        self.output_accumulator -= output

        return output

output = np.zeros(steps)
neuron = LeakyIntegratorSpiking(threshold=0.2, tau_rc=20)

for i in range(steps):
    output[i] = neuron.step(stim[i])

plt.figure(figsize=(14,4))
plt.plot(stim, label='stimulus (J)')
plt.plot(output_nonspiking, label='output (non-spiking)')
plt.plot(output, label='output (spiking)')
plt.xlabel('time steps')
plt.legend()
plt.show()


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
This notebook goes through an example of a general technique for converting a non-spiking neuron model into a spiking neuron model.  This technique is from https://arxiv.org/abs/2002.03553
We start with a particular neuron model we want to convert.  This could be anything at all, but here we're going to use a leaky integrator model defined like this:
Can you write Python code for it?

class LeakyIntegrator:
    def __init__(self, threshold, tau_rc=20):
        self.threshold = threshold
        self.tau_rc = tau_rc
        self.v = 0

    def step(self, J):
        if self.v > self.threshold:
            output = self.v - self.threshold
        else:
            output = 0

        dv = -self.v + output + J
        self.v += dv / self.tau_rc
        return output


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
We could even slow the spiking down even farther:
Can you write Python code for it?

neuron = LeakyIntegratorSpiking(threshold=0.2, tau_rc=20, spike_height=5)

for i in range(steps):
    output[i] = neuron.step(stim[i])

plt.figure(figsize=(14,4))
plt.plot(stim, label='stimulus (J)')
plt.plot(output_nonspiking, label='output (non-spiking)')
plt.plot(output, label='output (spiking)')
plt.xlabel('time steps')
plt.legend()
plt.title(f'spike_height={neuron.spike_height}')
plt.show()


I am working on the file "Converting non-spiking neurons to spiking neurons.ipynb".
The first task is:
Yikes!  What happened there?  Now when the peak happens, it's actually emitting 2 spikes per time step!  This is because the accumulator has a value in it that is twice the threshold, and they way I implemented the threshold detection to emit a spike was just like this:
self.output_accumulator += output / self.spike_height
        output = int(self.output_accumulator)
        self.output_accumulator -= output
That casting to an integer is a nice shortcut, and if our values are small enough, then output will only ever be 0 or 1.  But, if the output is supposed to be large enough that we should output 2 spikes on this same timestep, then this algorithm will go ahead and do so.
(Interestingly, if you do this trick to a neuron that can output negative numbers, then this algorithm will also go ahead and produce negative spikes!)
So what happens if we reduce this value even further?
Can you write Python code for it?

neuron = LeakyIntegratorSpiking(threshold=0.2, tau_rc=20, spike_height=0.3)

for i in range(steps):
    output[i] = neuron.step(stim[i])

plt.figure(figsize=(14,4))
plt.plot(stim, label='stimulus (J)')
plt.plot(output_nonspiking, label='output (non-spiking)')
plt.plot(output, label='output (spiking)')
plt.xlabel('time steps')
plt.legend()
plt.title(f'spike_height={neuron.spike_height}')
plt.show()


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
Convolution and Probability
But convolutions are much more than a simple way of expressing what simple transformations do to signals. In fact, they arise whenever we need to keep track of multiple possible contributions to a given value.
Say we want to know the probability of getting exactly two heads in three coin tosses.
First, we need to know the probability that the coin lands heads up. That plot appears below.
Can you write Python code for it?

pmfCoin = np.asarray([1/2,1/2])

plt.figure(); locs = [0,2]; edge = 4
labels = ['0H','1H']
plt.bar(locs,pmfCoin,align='center');

probabilityPlot(plt.gca(),locs,edge,labels)
plt.suptitle("One Coin Flip",size=24,weight='bold',y=1.);


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
HTML-based animation display
Can you write Python code for it?

# from http://jakevdp.github.io/blog/2013/05/12/embedding-matplotlib-animations/

from tempfile import NamedTemporaryFile

VIDEO_TAG = """<video controls>
 <source src="data:video/x-m4v;base64,{0}" type="video/mp4">
 Your browser does not support the video tag.
</video>"""

def anim_to_html(anim,fps=1):
    if not hasattr(anim, '_encoded_video'):
        with NamedTemporaryFile(suffix='.mp4') as f:
            anim.save(f.name, fps=fps,
                      extra_args=['-vcodec', 'libx264', '-pix_fmt', 'yuv420p'])
            video = open(f.name, "rb").read()
        anim._encoded_video = base64.b64encode(video).decode('utf-8')

    return VIDEO_TAG.format(anim._encoded_video)

from IPython.display import HTML

def display_animation(anim,fps=1):
    plt.close(anim._fig)
    return HTML(anim_to_html(anim,fps=fps))


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
Notice that the third filter's kernel is just the sum of the first two kernels. The response of the third filter is also a sum: it's the sum of the responses to the first two kernels.
This is also true if we put two signals in: the response to a sum of inputs is just the sum of the responses to the individual inputs.
Can you write Python code for it?

twoPointSignal = [1,0,1,0,0,0]
twoPointSignals = [twoPointSignal]

for kernel in kernels:
    twoPointSignals.append(np.convolve(twoPointSignal,kernel))

convolutionPlot(twoPointSignals,'two points',kernels,kernelNames)


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
Now, lets apply these kernels to our "unit impulse" -- one point. Notice what comes out: it's just the kernel!
Can you write Python code for it?

simpleSignal = [1]
simpleSignals = [simpleSignal]

for kernel in kernels:
    simpleSignals.append(np.convolve(simpleSignal,kernel))

convolutionPlot(simpleSignals,'just one point',kernels,kernelNames)


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
Plotting Functions
Can you write Python code for it?

def plotSignal(signal,signalName):
    plt.figure(figsize=(16,2));
    plt.plot(signal,'-o',color='k');
    pF.cleanPlot(plt.gca());
    plt.xlim(-len(signal)/10,len(signal)+len(signal)/10);
    pF.addAxis(plt.gca(),'horizontal');
    plt.title(signalName);

def plotSignalAsDelta(signal,signalName,color='r'):
    plt.figure(figsize=(16,4)); plt.subplot(2,1,1)
    plt.plot(signal,'-o',color='k');
    pF.cleanPlot(plt.gca());
    plt.xlim(-len(signal)/10,len(signal)+len(signal)/10);
    pF.addAxis(plt.gca(),'horizontal');
    plt.title(signalName);
    plt.subplot(2,1,2)
    deltaPlot(signal,color=color);
    pF.cleanPlot(plt.gca());
    plt.xlim(-len(signal)/10,len(signal)+len(signal)/10);
    pF.addAxis(plt.gca(),'horizontal');
    plt.title('also ' +signalName);

def deltaPlot(inp,color='r'):
    plt.scatter(np.arange(0,len(inp)),inp,
                linewidth=0,marker='o',s=36,color=color,
                zorder=10);
    plt.vlines(np.arange(0,len(inp)),0,inp,)

def plotKronecker():
    pad=[0,0]; padLen = len(pad)
    deltaPlot(pad+[1]+pad,color='b')
    pF.cleanPlot(plt.gca());
    plt.xlim(-padLen/10,2*padLen+1+padLen/10)
    pF.addAxis(plt.gca(),'horizontal');
    plt.title('the delta function')

def kernelsPlot(kernels,kernelNames):
    numKernels = len(kernels)
    plt.figure(figsize=(16,4));

    for idx,(kernel,name) in enumerate(zip(kernels,kernelNames)):
        plt.subplot(1,numKernels,idx+1)
        deltaPlot(kernel)
        pF.cleanPlot(plt.gca());
        plt.xlim(-len(kernel)/10,len(kernel)+len(kernel)/10);
        pF.addAxis(plt.gca(),'horizontal');
        plt.title(name);

def convolutionPlot(signals,signalName,kernels,kernelNames):
    for idx,(signal,kernel,kernelName) in enumerate(zip(signals[1:],kernels,kernelNames)):
        plt.figure(figsize=(16,4));

        plt.subplot(1,3,1)
        #Plot the original signal for reference
        deltaPlot(signals[0],color='blue');
        plt.title(signalName); plt.ylim(-1,1.5);
       ...


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
Since our signal is just an array, we can think of it as a collection of points. The zeroth element of the array is a point at $t = 0$, the next is a point at $t = 1$, etc. The height of the point is determined by the number in the array.
Put another way, we can break a signal of length $N$ down into $N$ components. If we were to represent the component at the timepoint $i$ as a function $e_i$, it would look like:
$$
e_i(t) = 1 \text{ if } t = i \
e_i(t) = 0 \text{ otherwise }
$$
This function is also called a "Kronecker's Delta", "the delta function", or the "unit impulse". If we were to draw it, it might look like:
Can you write Python code for it?


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
So if we have a more complicated signal, what we need to do to get the response to that signal is combine scaled, time-shifted copies of the kernel.
Put more formally: any signal can be broken up into scaled delta functions at different times, and the response of our simple transformation to a scaled delta function at some time $t$ is just a scaled copy of the kernel starting at that time $t$.
Let's put that into mathematical terms.
$$
\begin{align}
response_{filter}(signal) =& \sum_t response_{filter}(signal(t)) \
    =& \sum_t response_{filter}(impulse(t))*signal(t)
\end{align}
$$
Let's simplify our notation: we'll call the signal $s$ and the response of the filter $r_f$. We'll call the response to an impulse $k$. That makes our statement above:
$$
r_{f}(s) = \sum_t k*s(t)
$$
This gives us the overall response, but what if we wanted to know the response at a particular time $t$? In order to do that, we have to keep track of all the possible contributions to the response at that time. That is, the response at $t$ will include the beginning of the response to the point at time $t$ plus the ongoing responses to the points at time $t-1$, at time $t-2$, and so on. We'll call that time difference $\Delta$.
And what is the value of the response at $t$ to a point at $t-\Delta$? If $\Delta$ is $0$, then it's the zeroth* element of the kernel, scaled by $s(t)$. If $\Delta$ is $1$, then it's the first element of the kernel, scaled by $s(t-1)$. If $\Delta$ is $2$, then it's the second element of the kernel, scaled by $s(t-2)$.
There's an obvious pattern here: the response at $t$ to a point at $t-\Delta$ is just the $\Delta$th element of the kernel times the value of the signal at $t-\Delta$.
We should give a name to $t-\Delta$. The traditional one is $\tau$.
To reiterate: the response to a point at time $\tau$ is going to be a $s(\tau)$-scaled copy of the kernel response. At any given time $t$ after $\tau$, there will usually be several copies stacked on top of each other, each of them at a different point in their...
Can you write Python code for it?

# Define our filter kernels
pad = 1;  padding = [0]*pad
difference = padding+[1/2,0,-1/2]+padding;
average = padding+[1/2,1/2]+padding

kernels = [difference,average]
kernelNames = ['difference','average']
numKernels = len(kernels)

# Plot Our Kernels
kernelsPlot(kernels,kernelNames)
plt.suptitle('More Kernels!',fontsize=20,weight='bold',y=1.1);


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
We construct our signal from these components as follows: we multiply each $e_i$ by the signal $s$ at the timepoint $i$. The elements that are $0$ stay zero, while the element that is $1$ now has the value $s(i)$. We then add all $N$ of the $e_i$ together, and the result is our original signal.
If you've taken linear algebra, this process may sound familiar to you -- it's exactly the way that we construct a vector out of a set of basis vectors. This is a deep connection. Just as the canonical, or usual, basis set for a two-dimensional vector space is the set $\left{[1, \ 0], [0,\ 1]\right}$, the canonical basis set for signals is the set of all Kronecker delta functions, each at a different point in time or space.
Can you write Python code for it?


I am working on the file "Convolution/What's a Convolution?.ipynb".
The first task is:
Where $1H$ refers to getting a head and $0H$ refers to getting zero heads (aka a tails).
What are the possible ways we can get two heads in three coin tosses? A table appears below.
| First toss | Second Toss | Third Toss |
|:----------:|:-----------:|:----------:|
|      H     |      H      | T          |
|      H     |      T      | H          |
|      T     |      H      | H          |
How do we compute the probability of each of those outcomes? For an individual coin flip, the probability of a head or a tail is $1/2$. The probability of any pair $HH$, $HT$, etc. is $1/2*1/2 = 1/4$. When we look at combined events, all we have to do is multiply the probabilities.
And what is the overall probability that we get two heads? We just need to add up all the probabilities of the individual ways to get two heads.
To write that out mathematically, we split our result into two parts: the first two tosses, and the third toss. We'll call those components $A$ and $B$, and call our result $C$. In order to figure out the overall probability of our result $C$, we need to add up all the probabilities of combinations $A$ and $B$ that give us $C$. In our case, those combinations would be:
|  A | B |
|:--:|:-:|
| HH | T |
| HT | H |
| TH | H |
We can find the probability of any given combination by multiplying together the probabilities of its components $A$ and $B$, but to find the probability of our outcome $C$, we need to take all combinations into account:
$$
p(C) = \sum_{A+B=C} p(A)*p(B)
$$
Now doesn't that formula look familiar! It's a convolution, where the two functions being convolved are the probability functions of $A$ and $B$!
Note that this applies for any number of repetitions. If we want to know how likely it is to get any particular number $k$ of heads in some number of coin tosses $n$, we just need to look at the $n$th convolution of the coin flip probability distribution with itself.
The code block below generates an animated version of this process, showing you the result of each convolution in turn. Feel...
Can you write Python code for it?

pmfs = [pmfCoin]; #kernel = [1/2,1/2]
iters = 10; #change me to get different numbers of flips
mx = iters
locs = list(range(mx+2))
extendedPMF = np.hstack([pmfs[0],[0]*(mx+2-len(pmfs[0]))])
edge = 2
fig = plt.figure(figsize=(12,8)); pmfAx = plt.subplot(111);
pmfBars = pmfAx.bar(locs,extendedPMF,align='center')
labels = [str(n)+"H" for n in range(mx+1)]

probabilityPlot(plt.gca(),locs,edge,labels)
plt.suptitle("A Series of "+str(iters)+" Coin Flips",size=24,weight='bold',y=1.)

def init():
    return

def animate(_,pmfs):
    [pmfBars[idx].set_height(h)
         for idx,h in enumerate(pmfs[-1])]
    pmfs.append(np.convolve(pmfs[-1],pmfs[0])) #Convolution!
    return

anim = animation.FuncAnimation(fig, animate, init_func=init,
                                    fargs=[pmfs], frames=iters,
                                    interval=2)

display_animation(anim,fps=10) #change the FramesPerSecond here, for longer or shorter videos


I am working on the file "Convolutional Neural Network.ipynb".
The first task is:
Build TF Graph
To improve the prediction accuracy of the CovNet, I have decided on a fairly deep network utilizing a mix of convolution, pooling and fully connected layers.
Can you write Python code for it?

def cov_network(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """

    # Layer: Convolution #1-1
    net = layer_conv2d(x_tensor=x,
                       num_outputs=64,
                       k_size=(3,3),
                       strides=(1,1),
                       padding='SAME')
    net = tf.nn.dropout(x=net, keep_prob=keep_prob)

    # Layer: Convolution #1-2
    net = layer_conv2d(x_tensor=net,
                       num_outputs=64,
                       k_size=(3,3),
                       strides=(1,1),
                       padding='SAME')
    net = tf.nn.dropout(x=net, keep_prob=keep_prob)

    # Layer: Max Pool #1-1
    net = layer_max_pool(x_tensor=net,
                         k_size=(2, 2),
                         strides=(2, 2),
                         padding='SAME')
    net = tf.nn.dropout(x=net, keep_prob=keep_prob)

    # Layer: Convolution #2-1
    net = layer_conv2d(x_tensor=net,
                       num_outputs=128,
                       k_size=(3,3),
                       strides=(1,1),
                       padding='SAME')
    net = tf.nn.dropout(x=net, keep_prob=keep_prob)

    # Layer: Convolution #2-2
    net = layer_conv2d(x_tensor=net,
                       num_outputs=128,
                       k_size=(3,3),
                       strides=(1,1),
                       padding='SAME')
    net = tf.nn.dropout(x=net, keep_prob=keep_prob)

    # Layer: Max Pool #2-1
    net = layer_max_pool(x_tensor=net,
                         k_size=(2, 2),
                         strides=(2, 2),
                         padding='SAME')
    net = tf.nn.dropout(x=net, keep_prob=keep_prob)

    # Layer: Convolution #3-1
    net = layer_conv2d(x_tensor=net,
                       num_outputs=256,
                       k_size=(3,3),
                       strides=(1,1),
                      ...


I am working on the file "Convolutional Neural Network.ipynb".
The first task is:
Define layers
Before I can build my graph, I need to define the layers that I want to use. I am building a function for the following layers:

2D convolution
Max pooling
Flattening
Fully connected, and
Output
Can you write Python code for it?

def layer_conv2d(x_tensor, num_outputs, k_size, strides, padding='SAME'):
    """
    Apply convolution to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param num_outputs: Number of outputs for the convolutional layer
    :param k_size: kernal size 2-D Tuple for the convolutional layer
    :param strides: Stride 2-D Tuple for convolution
    :param padding: Desired padding - 'SAME' or 'VALID'
    : return: A tensor that represents convolution of x_tensor
    """

    # Define weight and bias
    W = tf.Variable(tf.truncated_normal(shape=[*k_size, x_tensor.get_shape().as_list()[3], num_outputs],
                                        stddev=0.1) * tf.sqrt(2.0/x_tensor.get_shape().as_list()[3]))
    b = tf.Variable(tf.truncated_normal(shape=[num_outputs],
                                        stddev=0.1))

    # Calculate Conv Layer
    conv_layer = tf.nn.conv2d(input=x_tensor,
                              filter=W,
                              strides=[1, *strides, 1],
                              padding=padding)
    conv_layer = tf.nn.bias_add(conv_layer, b)

    # Apply batch normalization before activation
    batch_mean, batch_var = tf.nn.moments(conv_layer, [0])
    scale = tf.Variable(tf.ones([num_outputs]))
    beta = tf.Variable(tf.zeros([num_outputs]))
    conv_layer = tf.nn.batch_normalization(conv_layer, batch_mean, batch_var, beta, scale, 0.000001)

    # Conv activation using ReLu
    conv_layer = tf.nn.relu(conv_layer)
    return conv_layer

def layer_max_pool(x_tensor, k_size, strides, padding='SAME'):
    """
    Apply max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param k_size: kernal size 2-D Tuple for pool
    :param strides: Stride 2-D Tuple for pool
    :param padding: Desired padding - 'SAME' or 'VALID'
    : return: A tensor that represents max pooling of x_tensor
    """

    # Apply max pooling
    conv_layer = tf.nn.max_pool(value=x_tensor,
                                ksize=[1, *k_size, 1],
                                strides=[1, *strides,...


I am working on the file "Convolutional Neural Network.ipynb".
The first task is:
Import data
I use the MNIST database of handwritten digits found @ http://yann.lecun.com/exdb/mnist/.
I am not reshaping the images as I need them in their original dimension in order to apply convolution and pooling.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist_data = input_data.read_data_sets("MNIST_data/", one_hot=True, reshape=False)


I am working on the file "Convolutional Neural Network.ipynb".
The first task is:
Launch TF Graph
Finally, a simple output to calculate the train, validation and test accuracy.
Can you write Python code for it?

# Calculate logits
logits = cov_network(x, keep_prob)

# Calculate loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits,
                                                              labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss=cost)

# Calculate accuracy percentage
correct_prediction = tf.equal(tf.argmax(logits, axis=1), tf.argmax(y, axis=1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy')

# Define global variable initializer
init = tf.global_variables_initializer()

# Launch session
with tf.Session() as sess:
    sess.run(init)

    for epoch in range(training_epochs):
        for batch in range(mnist_data.train.num_examples//batch_size):

            # Get x and y values for the given batch
            batch_x, batch_y = mnist_data.train.next_batch(batch_size)
            sess.run(optimizer, feed_dict={x: batch_x,
                                           y: batch_y,
                                           keep_prob: keep_probability})

            # Calculate batch loss and accuracy
            loss, training_accuracy = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                             y: batch_y,
                                             keep_prob: 1.})

            # Compute graph with respect to validation data
            validation_accuracy = sess.run(accuracy, feed_dict={
                x: mnist_data.validation.images[:valid_size],
                y: mnist_data.validation.labels[:valid_size],
                keep_prob: 1.})

            # Display logs per batch step
            if batch % print_loss_for_each_batch == 0:
                print('Epoch {:>2}, Batches {:>3}, Loss: {:>.4f}, Train Accuracy: {:.4f}, Val Accuracy: {:.4f}'.format(
                    epoch + 1, # epoch starts at 0
                    batch + 1, # batch starts at 0
                    loss,
                    training_accuracy,
                   ...


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Exercise: Now that you have J_content and J_style, compute the total cost J by calling total_cost(). Use alpha = 10 and beta = 40.
Can you write Python code for it?

### START CODE HERE ### (1 line)
J = total_cost(J_content, J_style, 10,40)
### END CODE HERE ###


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Expected Output:

            **J**

           35.34667875478276

What you should remember:
- The total cost is a linear combination of the content cost $J_{content}(C,G)$ and the style cost $J_{style}(S,G)$
- $\alpha$ and $\beta$ are hyperparameters that control the relative weighting between content and style
4 - Solving the optimization problem
Finally, let's put everything together to implement Neural Style Transfer!
Here's what the program will have to do:

Create an Interactive Session
Load the content image
Load the style image
Randomly initialize the image to be generated
Load the VGG16 model
Build the TensorFlow graph:
Run the content image through the VGG16 model and compute the content cost
Run the style image through the VGG16 model and compute the style cost
Compute the total cost
Define the optimizer and the learning rate

Initialize the TensorFlow graph and run it for a large number of iterations, updating the generated image at every step.

Lets go through the individual steps in detail.
You've previously implemented the overall cost $J(G)$. We'll now set up TensorFlow to optimize this with respect to $G$. To do so, your program has to reset the graph and use an "Interactive Session". Unlike a regular session, the "Interactive Session" installs itself as the default session to build a graph.  This allows you to run variables without constantly needing to refer to the session object, which simplifies the code.
Lets start the interactive session.
Can you write Python code for it?

# Reset the graph
tf.reset_default_graph()

# Start interactive session
sess = tf.InteractiveSession()


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Expected Output:

            **J_content**

           6.76559

What you should remember:
- The content cost takes a hidden layer activation of the neural network, and measures how different $a^{(C)}$ and $a^{(G)}$ are.
- When we minimize the content cost later, this will help make sure $G$ has similar content as $C$.
3.2 - Computing the style cost
For our running example, we will use the following style image:
Can you write Python code for it?

style_image = scipy.misc.imread("images/monet_800600.jpg")
imshow(style_image)


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Expected Output:

            **J_style_layer**

           9.19028

3.2.3 Style Weights
So far you have captured the style from only one layer. We'll get better results if we "merge" style costs from several different layers. After completing this exercise, feel free to come back and experiment with different weights to see how it changes the generated image $G$. But for now, this is a pretty reasonable default:
Can you write Python code for it?


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Let's load, reshape and normalize our "style" image (Claude Monet's painting):
Can you write Python code for it?

style_image = scipy.misc.imread("images/monet.jpg")
style_image = reshape_and_normalize_image(style_image)


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Let's load, reshape, and normalize our "content" image (the Louvre museum picture):
Can you write Python code for it?

content_image = scipy.misc.imread("images/louvre_small.jpg")
content_image = reshape_and_normalize_image(content_image)


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Note: At this point, a_G is a tensor and hasn't been evaluated. It will be evaluated and updated at each iteration when we run the Tensorflow graph in model_nn() below.
Can you write Python code for it?

# Assign the input of the model to be the "style" image
sess.run(model['input'].assign(style_image))

# Compute the style cost
J_style = compute_style_cost(model, STYLE_LAYERS)


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Note: In the inner-loop of the for-loop above, a_G is a tensor and hasn't been evaluated yet. It will be evaluated and updated at each iteration when we run the TensorFlow graph in model_nn() below.
<!--
How do you choose the coefficients for each layer? The deeper layers capture higher-level concepts, and the features in the deeper layers are less localized in the image relative to each other. So if you want the generated image to softly follow the style image, try choosing larger weights for deeper layers and smaller weights for the first layers. In contrast, if you want the generated image to strongly follow the style image, try choosing smaller weights for deeper layers and larger weights for the first layers
!-->

<font color='blue'>
What you should remember:
- The style of an image can be represented using the Gram matrix of a hidden layer's activations. However, we get even better results combining this representation from multiple different layers. This is in contrast to the content representation, where usually using just a single hidden layer is sufficient.
- Minimizing the style cost will cause the image $G$ to follow the style of the image $S$.
</font color='blue'>
3.3 - Defining the total cost to optimize
Finally, let's create a cost function that minimizes both the style and the content cost. The formula is:
$$J(G) = \alpha J_{content}(C,G) + \beta J_{style}(S,G)$$
Exercise: Implement the total cost function which includes both the content cost and the style cost.
Can you write Python code for it?

# GRADED FUNCTION: total_cost

def total_cost(J_content, J_style, alpha = 10, beta = 40):
    """
    Computes the total cost function

    Arguments:
    J_content -- content cost coded above
    J_style -- style cost coded above
    alpha -- hyperparameter weighting the importance of the content cost
    beta -- hyperparameter weighting the importance of the style cost

    Returns:
    J -- total cost as defined by the formula above.
    """

    ### START CODE HERE ### (≈1 line)
    J = alpha*J_content + beta*J_style
    ### END CODE HERE ###

    return J

tf.reset_default_graph()

with tf.Session() as test:
    np.random.seed(3)
    J_content = np.random.randn()
    J_style = np.random.randn()
    J = total_cost(J_content, J_style)
    print("J = " + str(J))


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Now, we initialize the "generated" image as a noisy image created from the content_image. By initializing the pixels of the generated image to be mostly noise but still slightly correlated with the content image, this will help the content of the "generated" image more rapidly match the content of the "content" image. (Feel free to look in nst_utils.py to see the details of generate_noise_image(...); to do so, click "File-->Open..." at the upper-left corner of this Jupyter notebook.)
Can you write Python code for it?

generated_image = generate_noise_image(content_image)
imshow(generated_image[0])


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
Run the following cell to generate an artistic image. It should take about 3min on CPU for every 20 iterations but you start observing attractive results after ≈140 iterations. Neural Style Transfer is generally trained using GPUs.
Can you write Python code for it?


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
The model is stored in a python dictionary where each variable name is the key and the corresponding value is a tensor containing that variable's value. To run an image through this network, you just have to feed the image to the model. In TensorFlow, you can do so using the tf.assign function. In particular, you will use the assign function like this:
python
model["input"].assign(image)
This assigns the image as an input to the model. After this, if you want to access the activations of a particular layer, say layer 4_2 when the network is run on this image, you would run a TensorFlow session on the correct tensor conv4_2, as follows:
python
sess.run(model["conv4_2"])
3 - Neural Style Transfer
We will build the NST algorithm in three steps:

Build the content cost function $J_{content}(C,G)$
Build the style cost function $J_{style}(S,G)$
Put it together to get $J(G) = \alpha J_{content}(C,G) + \beta J_{style}(S,G)$.

3.1 - Computing the content cost
In our running example, the content image C will be the picture of the Louvre Museum in Paris. Run the code below to see a picture of the Louvre.
Can you write Python code for it?

content_image = scipy.misc.imread("images/louvre.jpg")
imshow(content_image)


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
This painting was painted in the style of impressionism.
Lets see how you can now define a "style" const function $J_{style}(S,G)$.
3.2.1 - Style matrix
The style matrix is also called a "Gram matrix." In linear algebra, the Gram matrix G of a set of vectors $(v_{1},\dots ,v_{n})$ is the matrix of dot products, whose entries are ${\displaystyle G_{ij} = v_{i}^T v_{j} = np.dot(v_{i}, v_{j})  }$. In other words, $G_{ij}$ compares how similar $v_i$ is to $v_j$: If they are highly similar, you would expect them to have a large dot product, and thus for $G_{ij}$ to be large.
Note that there is an unfortunate collision in the variable names used here. We are following common terminology used in the literature, but $G$ is used to denote the Style matrix (or Gram matrix) as well as to denote the generated image $G$. We will try to make sure which $G$ we are referring to is always clear from the context.
In NST, you can compute the Style matrix by multiplying the "unrolled" filter matrix with their transpose:
<img src="images/NST_GM.png" style="width:900px;height:300px;">
The result is a matrix of dimension $(n_C,n_C)$ where $n_C$ is the number of filters. The value $G_{ij}$ measures how similar the activations of filter $i$ are to the activations of filter $j$.
One important part of the gram matrix is that the diagonal elements such as $G_{ii}$ also measures how active filter $i$ is. For example, suppose filter $i$ is detecting vertical textures in the image. Then $G_{ii}$ measures how common  vertical textures are in the image as a whole: If $G_{ii}$ is large, this means that the image has a lot of vertical texture.
By capturing the prevalence of different types of features ($G_{ii}$), as well as how much different features occur together ($G_{ij}$), the Style matrix $G$ measures the style of an image.
Exercise:
Using TensorFlow, implement a function that computes the Gram matrix of a matrix A. The formula is: The gram matrix of A is $G_A = AA^T$. If you are stuck, take a look at Hint 1 and Hint 2.
Can you write Python code for it?

# GRADED FUNCTION: gram_matrix

def gram_matrix(A):
    """
    Argument:
    A -- matrix of shape (n_C, n_H*n_W)

    Returns:
    GA -- Gram matrix of A, of shape (n_C, n_C)
    """

    ### START CODE HERE ### (≈1 line)
    GA = tf.matmul(A,tf.transpose(A))
    ### END CODE HERE ###

    return GA

tf.reset_default_graph()

with tf.Session() as test:
    tf.set_random_seed(1)
    A = tf.random_normal([3, 2*1], mean=1, stddev=4)
    GA = gram_matrix(A)

    print("GA = " + str(GA.eval()))


I am working on the file "Convolutional Neural Networks/Art+Generation+with+Neural+Style+Transfer+-+v2.ipynb".
The first task is:
To get the program to compute the content cost, we will now assign a_C and a_G to be the appropriate hidden layer activations. We will use layer conv4_2 to compute the content cost. The code below does the following:

Assign the content image to be the input to the VGG model.
Set a_C to be the tensor giving the hidden layer activation for layer "conv4_2".
Set a_G to be the tensor giving the hidden layer activation for the same layer.
Compute the content cost using a_C and a_G.
Can you write Python code for it?

# Assign the content image to be the input of the VGG model.
sess.run(model['input'].assign(content_image))

# Select the output tensor of layer conv4_2
out = model['conv4_2']

# Set a_C to be the hidden layer activation from the layer we have selected
a_C = sess.run(out)

# Set a_G to be the hidden layer activation from same layer. Here, a_G references model['conv4_2']
# and isn't evaluated yet. Later in the code, we'll assign the image G as the model input, so that
# when we run the session, this will be the activations drawn from the appropriate layer, with G as input.
a_G = out

# Compute the content cost
J_content = compute_content_cost(a_C, a_G)


I am working on the file "Convolutional Neural Networks/Autonomous+driving+application+-+Car+detection+-+v1.ipynb".
The first task is:
Expected Output:

            **scores[2]**

           138.791

            **boxes[2]**

           [ 1292.32971191  -278.52166748  3876.98925781  -835.56494141]

            **classes[2]**

           54

            **scores.shape**

           (10,)

            **boxes.shape**

           (10, 4)

            **classes.shape**

           (10,)

Summary for YOLO:
- Input image (608, 608, 3)
- The input image goes through a CNN, resulting in a (19,19,5,85) dimensional output.
- After flattening the last two dimensions, the output is a volume of shape (19, 19, 425):
    - Each cell in a 19x19 grid over the input image gives 425 numbers.
    - 425 = 5 x 85 because each cell contains predictions for 5 boxes, corresponding to 5 anchor boxes, as seen in lecture.
    - 85 = 5 + 80 where 5 is because $(p_c, b_x, b_y, b_h, b_w)$ has 5 numbers, and and 80 is the number of classes we'd like to detect
- You then select only few boxes based on:
    - Score-thresholding: throw away boxes that have detected a class with a score less than the threshold
    - Non-max suppression: Compute the Intersection over Union and avoid selecting overlapping boxes
- This gives you YOLO's final output.
3 - Test YOLO pretrained model on images
In this part, you are going to use a pretrained model and test it on the car detection dataset. As usual, you start by creating a session to start your graph. Run the following cell.
Can you write Python code for it?