I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Let's look at Canada's reported concussions:
Can you write Python code for it?

can_concussion = df[(df['Ctry'] == 'CAN') & (df['Injury'] == 'Concussion')]

can_concussion[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Let's look at how many of those Edmonton-born players were drafted before 2006
Can you write Python code for it?

df[(df['Birth City'] == 'Edmonton') & (df['Draft'] < 2006.0)].head()


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Let's look at how many of those Minneapolis-born players were drafted before 2006
Can you write Python code for it?

df[(df['Birth City'] == 'Minneapolis') & (df['Draft'] < 2006.0)].head()


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Let's look at how many of those Toronto-born players were drafted before 2006
Can you write Python code for it?

df[(df['Birth City'] == 'Toronto') & (df['Draft'] < 2006.0)].head()


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Let's look at how old the players are:
Can you write Python code for it?

birthdate = df[df['DOB']].replace("DOB", "")
birthdate

df['birthyear'] = df['DOB'].astype(str).str.split("'").str.get(1).astype(int)

df


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Lithuania Concussions
Can you write Python code for it?

ltu_concussion = df[(df['Ctry'] == 'LTU') & (df['Injury'] == 'Concussion')]

ltu_concussion[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Norway "Undisclosed" Injuries
Can you write Python code for it?

nor_mystery_injury = df[(df['Ctry'] == 'NOR') & (df['Injury'] == 'Undisclosed')]

nor_mystery_injury[["First Name", "Last Name"]]

df


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Old Players (36 years old or younger) for Canada:
Can you write Python code for it?

old_can_players = df[(df['Ctry'] == 'CAN') & (df['birthyear'] <= 80 )]

old_can_players[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Russia "Undisclosed Injuries"
Can you write Python code for it?

rus_mystery_injury = df[(df['Ctry'] == 'RUS') & (df['Injury'] == 'Undisclosed')]

rus_mystery_injury[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Russia Concussions:
Can you write Python code for it?

rus_concussion = df[(df['Ctry'] == 'RUS') & (df['Injury'] == 'Concussion')]

rus_concussion[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Sweden "Undisclosed" Injuries
Can you write Python code for it?

swe_mystery_injury = df[(df['Ctry'] == 'SWE') & (df['Injury'] == 'Undisclosed')]

swe_mystery_injury[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Sweden Concussions:
Can you write Python code for it?

swe_concussion = df[(df['Ctry'] == 'SWE') & (df['Injury'] == 'Concussion')]

swe_concussion[["First Name", "Last Name"]]


I am working on the file "Homework_8_DIY_Soma.ipynb".
The first task is:
Young Players (24 years old or younger) for the United States:
Can you write Python code for it?

young_usa_players = df[(df['Ctry'] == 'USA') & (df['birthyear'] >= 94 )]
young_usa_players[["First Name", "Last Name"]]


I am working on the file "Homework_Assignments/HW_11.7_Series_Testing_Strategy_[EC].ipynb".
The first task is:
Problem 1
Test whether the series is convergent or divergent:
$$\sum_{n=1}^{\infty} \frac{1}{n+7^n}$$
Can you write Python code for it?

import sympy as sp
from matplotlib import pyplot as plt

%matplotlib inline

# Customize figure size
plt.rcParams['figure.figsize'] = 25, 15
#plt.rcParams['lines.linewidth'] = 1
#plt.rcParams['lines.color'] = 'g'
plt.rcParams['font.family'] = 'monospace'
plt.rcParams['font.size'] = '16.0'
plt.rcParams['text.hinting'] = 'either'

f = lambda x: 1/(x + 7**x)

sp.mpmath.plot([f], xlim=[-5,25], ylim=[0,25], points=500)

# To check your work, use Sympy.mpmath.nsum()
# This gives the sum of the infinite series (if the series converges)
infty = sp.mpmath.inf
sum = sp.mpmath.nsum(f, [1, infty])
print('The sum of the series = {}'.format(sum))


I am working on the file "Homeworks/APPM5630_HW8_helper.ipynb".
The first task is:
... so we see that mySimpleSolver didn't have to do anything.  Now to get the information, we just ask for the value of objective.history as follows:
Can you write Python code for it?


I am working on the file "Homeworks/APPM5630_HW8_helper.ipynb".
The first task is:
One solution
Below is one way to use a class to do this
I'm not a software developer in python, so there may be much nicer ways, but this ought to be good enough.  There are many variants and extensions you can do, and of course, if you wanted to record some error function (like if you knew the true answer $x^\star$ and wanted to record $\|x-x^\star\|$ every iteration), you could easily incorporate that too.
Can you write Python code for it?

f = lambda x : x**2

class fcn:
  def __init__(self):
    self.history = []

  def evaluate(self,x):
    # Whatever objective function you're implementing
    # This also sees objective in the parent workspace,
    # so you can just call those
    fx = f(x)
    self.history.append(fx)
    return fx

  def reset(self):
    self.history = []

objective = fcn()
F = lambda x : objective.evaluate(x) # alternatively, have your class return a function
x = mySimpleSolver( F, 1 )
print(x)


I am working on the file "Homeworks/HW04/APPM5630_HW4_helperFunctions.ipynb".
The first task is:
Adjoint test
Let's make a different test of the adjoint
This one will scale easily to huge values of $N$
The adjoint of a linear operator $A$ is the operator (let's call it $T$, but later we'll denote it $A^*$ or $A^\dagger$ or $A^H$ in physics) such that for any inputs $x$ and $y$ of the right size,
$$ \langle Ax, y \rangle = \langle x, Ty \rangle. $$
This inner product we use np.vdot for, as it does the right thing (adds in a complex conjugat, and automatically vectorizes it, so it's the right thing to do for a matrix).  np.dot does the right thing only if our input is 1D and real-valued.
So, we'll just check this equation for several random choices of $x$ and $y$.  This is no way to prove that the adjoint is implemented correctly, but if we pass a lot of these checks, it's good evidence we've done it correctly.  If we fail a check, then we know for sure that there is a problem.
Note that for a matrix, there's also the formula
$$ A^ = \overline{A}^T$$
meaning the adjoint is the transpose of the complex conjugate. So for a real-valued matrix, the adjoint is just the transpose*.
Btw, do not confuse adjoint with "adjugate", which is a classical term used when finding the inverse of a matrix by hand
Can you write Python code for it?

def test_adjoint( A, At, domainSize, nRep = 10):
  rng   = np.random.default_rng()
  for rep in range(nRep):
    x  = rng.normal( size=domainSize )
    Ax = A(x)
    y  = rng.normal( size=Ax.shape)
    Aty= At(y)

    er  = np.vdot(Ax,y) - np.vdot(x,Aty)
    er  = np.abs(er)/np.sqrt( norm(x)*norm(y) ) # nice scaling
    print(f"Relative error in adjoint: {er:.2e}")

A   = np.arange(12).reshape(3,4) # test it on something rectangular (not square) to help expose more bugs

linop     = lambda x : A@x
linop_adj = lambda y : A.T@y

test_adjoint( linop, linop_adj, A.shape[1] )


I am working on the file "Homeworks/HW04/APPM5630_HW4_helperFunctions.ipynb".
The first task is:
Power method to estimate spectral norms
Can you write Python code for it?

A   = np.arange(12).reshape(3,4) # test it on something rectangular (not square) to help expose more bugs

normA = norm(A,ord=2)
print("Spectral norm of A is", normA )

def powerMethod(A, At, domainSize=None, x=None, iters=100, tol=1e-6):
  if x is None:
    if domainSize is None:
      raise ValueError("need domain size or x0 to be specified")
    rng   = np.random.default_rng()
    x = rng.normal( size=domainSize )
  normalization = norm( x.ravel() ) # Euclidean/Frobenius norm
  for k in range(iters):
    x = np.real_if_close( At(A(x)) )
    oldNormalization = normalization
    normalization = norm( x.ravel() )
    x /= normalization  # do this in-place
    if abs(oldNormalization - normalization)/np.max( [1e-12,normalization] ) < tol:
      print("Reached tolerance after",k,"iterations.")
      break
  return np.sqrt(normalization)

normA_v2 = powerMethod(linop,linop_adj, A.shape[1], iters=1000, tol=1e-12)
print("Spectral norm of A is", normA_v2, "using power method" )
print("  so the error in the power method is", abs(normA-normA_v2) )


I am working on the file "Homeworks/HW04/APPM5630_HW4_helperFunctions.ipynb".
The first task is:
implicit2explicit
We'll get a little bit fancy and allow functions that have matrix inputs (theoretically, this is no big deal, since matrices in $\mathbb{R}^{m\times n}$ are isomorphic to vectors in $\mathbb{R}^{mn}$ when we use the inner product $\langle X, Y\rangle = \text{trace}(X^Y)$ where $X^=\overline{X}^T$ is the adjoint.  This was not necessary for the homework problem.
Note that you can specify domainSize which could be something like n for a 1D vector of length $n$, or sometimes (n,1) depending on how the function expects it. For matrices, set domainSize to (m,n).
Can you write Python code for it?

def implicit2explicit( linearFcn, domainSize ):
    """ domainSize can be an integer, for a pure vector,
    or can be a tuple of integers """
    x = np.zeros( domainSize )
    # use ravel not flatten, as we don't want to make a copy

    # Determine sizes
    n = x.size
    x.ravel()[0] = 1  # first unit vector
    y = np.asarray( linearFcn( x ) ) # output may be scalar or list (not array) so convert it
    m = y.size
    x.ravel()[0] = 0 # undo it

    # Allocate array
    A = np.zeros( (m,n), dtype=y.dtype ) # allow for complex values
    A[:,0] = y.ravel()

    # Loop over all inputs (we already did the first one)
    for col in range(1,n):
      x.ravel()[col] = 1  # first unit vector
      y = linearFcn( x )
      A[:,col] = np.asarray(y).ravel()
      x.ravel()[col] = 0 # undo it

    return A


I am working on the file "Homeworks/HW04/APPM5630_HW4_helperFunctions.ipynb".
The first task is:
to fix this bug, we don't use A[col] and so on, we use A[:,col] to make sure we're getting a column, not a row:
Can you write Python code for it?

def implicit2explicit_square_fixed( linearFcn, n ):
  I   = np.eye(n)
  y   = linearFcn( I[0] )
  A   = np.zeros( (n,n), dtype=y.dtype )
  for col in range(0,n):
    A[:,col] = linearFcn( I[:,col] )
  return A

A_bugFixed = implicit2explicit_square_fixed( linearFcn, 4 )
print("Now it works:")
matprint(A_bugFixed)


I am working on the file "Homeworks/HW10/listen_to_Handel.ipynb".
The first task is:
Now if we downsample by factor of 2
Can you write Python code for it?

print("Downsampled by factor of 2")
Audio(y[::2],rate=Fs/2)


I am working on the file "Homeworks/HW10/listen_to_Handel.ipynb".
The first task is:
Periodogram
Can you you see the two slow "Hallelujah"s followed by the four fast "Halleluyah"s?
Can you write Python code for it?

fig = plt.figure(figsize=(8,8))
f, t, Sxx = sig.spectrogram(y, fs=Fs, window='barthann', nperseg=512,
    scaling='density')
plt.pcolormesh(f, t, dbPower(Sxx.T), vmin=-50,vmax=-30)
plt.colorbar()
plt.xlabel('Frequency [Hz]')
plt.ylabel('Time [sec]')
plt.show()


I am working on the file "Homeworks/HW10/listen_to_Handel.ipynb".
The first task is:
Plot spectrograms and periodograms
The filter we use
Can you write Python code for it?

w, h = sig.freqz(filt)
fig = plt.figure()
plt.title('Anti-alias Frequency Response')
plt.plot(w/np.pi*Fs/2, dbAmplitude(h) )
plt.ylabel('Amplitude [dB]')
plt.xlabel('Frequency [Hz]')

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(12,10),sharey=True)

# def with_logging(*args, **kwargs):
#     print(func.__name__ + " was called")
#     return func(*args, **kwargs)
def periodogram(*args,**kwargs):
  return sig.welch(*args,**kwargs,window='barthann',scaling='density')

f, Pxx = periodogram(y,fs=Fs,nperseg=1024)
ax1.plot(f, dbPower(Pxx))
ax1.set_title('Original signal')

f, Pxx = periodogram(y[::4], fs=Fs/4, nperseg=128)
ax2.plot(f, dbPower(Pxx))
ax2.set_title('Downsampled (x4) (aliased)')

f, Pxx = periodogram(lowpassFilter(y), fs=Fs, nperseg=1024)
ax3.plot(f, dbPower(Pxx))
ax3.set_title('Filtered')

f, Pxx = periodogram(lowpassFilter(y)[::4], fs=Fs/4, nperseg=128)
ax4.plot(f, dbPower(Pxx))
ax4.set_title('Filtered then Downsampled')

for ax in (ax1, ax2, ax3, ax4):
    ax.set_xlabel('Frequency [Hz]')
    ax.set_ylabel('Power Spectral Density [dB/Hz]')
plt.show()


I am working on the file "Homeworks/HW10/listen_to_Handel.ipynb".
The first task is:
Test the short-time DCT transform and its adjoint
We'll test if they are really adjoints of each other
Can you write Python code for it?

rng   = np.random.default_rng()

# This won't work: if x is 2D array, it'll mess things up
# x     = rng.normal(size=(2048,1))
# coeff, win = forwardShortTimeDCT(x.ravel() )

# Make x a 1D array
x     = rng.normal(size=100)
coeff, win = forwardShortTimeDCT(x)
# win   = np.ones(win.shape)

D   = lambda x : forwardShortTimeDCT(x, win=win )
Dt  = lambda y : adjointShortTimeDCT(y,win,x.size)

coeff = D(x)
print("sum(coeff) is", np.sum(coeff) )
xx = Dt(coeff)
print(f'Test 1: is Dt(Dt(x))=x? error is {norm(x-xx):.2e}')

y     = rng.normal( size=coeff.shape )
e     = np.vdot( D(x), y ) - np.vdot( x, Dt(y) ) # error
print(f"Error with adjoint is {e:.2e}")

# The windown that we use
plt.plot(win)
plt.show()


I am working on the file "Homeworks/HW10/listen_to_Handel.ipynb".
The first task is:
Test the utility functions
from python_functions.py
Test the l1 projection function
Can you write Python code for it?

# ?project_l1

rng   = np.random.default_rng()
x     = rng.normal(size=(100,1))
print('l1 norm of x is',norm(x,1))

# Test 1: projection shouldn't do anything
tau   = norm(x,1)*1.1
y     = project_l1( x, tau )
print(f"Test 1, error is {norm(y-x):.2e}")

# Test 2: projection should do something
tau   = norm(x,1)*0.8
import cvxpy as cvx
y     = cvx.Variable(x.shape)
obj   = cvx.Minimize( cvx.norm(y-x) )
constraints = [ cvx.norm(y,1) <= tau]
prob  = cvx.Problem(obj,constraints)
prob.solve(verbose=False,solver='ECOS',abstol=1e-15,reltol=1e-15,feastol=1e-15)

yy    = project_l1(x,tau)
print(f"Test 2, error is {norm(yy-y.value):.2e}")


I am working on the file "Homeworks/Homework6/Statistics and Probablity Homework.ipynb".
The first task is:
6.2.2  Calculate the actual probablities and compare it to your estimates for:
n= number of coins
k1 = min number of heads
k2 = upper limit of number of heads
m = the number of experiments
6.2.2.a n=100, k1 = 40, k2=60 m=100
6.2.2.b n=100, k1 = 40, k2=60 m=1000
6.3 Conditional probablity
In a recent study, the following data were obtained in response to the question"
    "Do you favor the proposal of the school’s combining the elementary and middle school students in one building?"
Answers = [Yes, No, No opinion]
Males = [75, 89, 10]
Females = [105, 56, 6]
If a person is selected at random, find these probabilities solving using python.
1. The person has no opinion
2. The person is a male or is against the issue.
3. The person is a female, given that the person opposes the issue.
6.4 Matrix creation
Write a 12 by 12 times table matrix shown below.
Do this
6.4.1 using nested for loops
6.4.2 using numpy fromfunction array constructor
6.4.3 using numpy broadcasting
Can you write Python code for it?


I am working on the file "Homeworks/Homework6/Statistics and Probablity Homework.ipynb".
The first task is:
6.5
Answer the following questions with respect to the
https://data.cdc.gov/NCHS/NCHS-Leading-Causes-of-Death-United-States/bi63-dtpu
How many patients were censored?
What is the correlation coefficient between state and Suicide for deaths above 100 ?
What is the average deaths for each state and type of cause ?
What is the year that was the most deadly for each cause name ?
Can you write Python code for it?

import pandas as pd
dfh = pd.read_csv(".\data\NCHS_-_Leading_Causes_of_Death__United_States.csv")
dfh.head()


I am working on the file "Homeworks/ME596_Homework_4.ipynb".
The first task is:
Polynomial Approximation
Can you write Python code for it?

# Polynomial approximation (4-point cubic)
# -Erin Schmidt

import numpy as np
from math import sqrt

# make an array with random values between 0.05 and 0.15 with 4 entries
x = (0.05 + np.random.sample(4)*0.15)

# make an array of function values at the 4 points of x
def f(x): # the objective function
    return (1.11 + 1.11*x**(-0.18))/(1 - x)

f_array = []
i = 0
while i <= len(x) - 1:
    f_array.append(f(x[i]))
    i += 1

# use the equations from Vanderplaats 1984 to solve coefficients
q1 = x[2]**3 * (x[1] - x[0]) - x[1]**3 * (x[2] - x[0]) + x[0]**3 * (x[2] - x[1])
q2 = x[3]**3 * (x[1] - x[0]) - x[1]**3 * (x[3] - x[0]) + x[0]**3 * (x[3] - x[1])
q3 = (x[2] - x[1]) * (x[1] - x[0]) * (x[2] - x[0])
q4 = (x[3] - x[1]) * (x[1] - x[0]) * (x[3] - x[0])
q5 = f_array[2] * (x[1] - x[0]) - f_array[1] * (x[2] - x[0]) + f_array[0] * (x[2] - x[1])
q6 = f_array[3] * (x[1] - x[0]) - f_array[1] * (x[3] - x[0]) + f_array[0] * (x[3] - x[1])

a3 = (q3*q6 - q4*q5)/(q2*q3 - q1*q4)
a2 = (q5 - a3*q1)/q3
a1 = (f_array[1] - f_array[0])/(x[1] - x[0]) - \
a3*(x[1]**3 - x[0]**3)/(x[1] - x[0]) - a2*(x[0] + x[1])
a0 = f_array[0] - a1*x[0] - a2*x[0]**2 - a3*x[0]**3
a = [a1, 2*a2, 3*a3] #coefficients of f'

# find the zeros of the f' polynomial (using the quadratic formula)
b = a2**2 - 3*a1*a3
X1 = (-a2 + sqrt(b))/(3*a3)
X2 = (-a2 - sqrt(b))/(3*a3)

print('roots = ', X1, X2)

# plot the results
plt.rc('pdf',fonttype=3)          # for proper subsetting of fonts
plt.rc('axes',linewidth=0.5)      # thin axes; the default for lines is 1pt
x = np.linspace( 0.05, 0.15, 500)
plt.plot(x, a0 +a1*x + a2*x**2 + a3*x**3, 'k--', label='Poly. approx.')
plt.plot(x, (1.11 + 1.11*x**(-0.18))/(1 - x), 'k', label='Objective func.')
plt.axis([0.05, 0.15, 3.09,3.17])
legend = plt.legend(loc='upper center', shadow=False, fontsize='large')
plt.ylabel("f")
plt.xlabel("x/D")
plt.show()

poly_root = [0.097620387704, 0.0985634827486, 0.0969340736066, \
             0.098775097463, 0.102426814371, 0.101638472077, \
             0.0991941169039,...


I am working on the file "Hopfield Network of memes--Storkey Learning Rule development.ipynb".
The first task is:
And indeed, they are the same!
Now let's show that the weights work, i.e., that feeding the network a noisy version of a trained pattern recovers the trained pattern itself. We'll just flip a certain number of random pixels on each row of the trained pattern.
Can you write Python code for it?

def noisify(pattern, numb_flipped):

    noisy_pattern = pattern.copy()

    for idx, row in enumerate(noisy_pattern):
        choices = np.random.choice(range(len(row)), numb_flipped)
        noisy_pattern[idx,choices] = -noisy_pattern[idx,choices]

    return noisy_pattern

noisy_test_pat = noisify(pattern=test_pat, numb_flipped=1)

plt.imshow(noisy_test_pat, cmap='Greys', interpolation='nearest')


I am working on the file "Hopfield Network of memes--Storkey Learning Rule development.ipynb".
The first task is:
As expected, the matrix math version -- which uses the optimized numpy data structures and operations -- is much, much faster than the for-loops verison. Now compare the weights of the two approaches.
Can you write Python code for it?

loop_weights = storkey_rule_loops(pats)
print(loop_weights)

matrix_weights = storkey_rule(pats[1], old_weights=storkey_rule(pats[0]))
print(matrix_weights)

matrix_weights == loop_weights


I am working on the file "Hopfield Network of memes--Storkey Learning Rule development.ipynb".
The first task is:
That algorithm is probably fairly opaque, owing to the matrix operations in it. Let's unpack it step-by-step. It's most illustrative to look at what happens when you are training a second pattern, because the weights of the first pattern are simply due to the hebbian term -- the other terms are just 0's. So, we'll start with old_weights from training the first test pattern.
Can you write Python code for it?

old_weights = storkey_rule(pats[0])

pattern = pats[1]

mem = pattern.flatten()
print(mem)

n = len(mem)
print(n)

if old_weights is None:
    old_weights = np.zeros(shape=(n,n))

hebbian_term  = np.outer(mem,mem)
print(hebbian_term)

net_inputs = old_weights.dot(mem)
print(net_inputs)

net_inputs = np.tile(net_inputs, (n, 1)) # repeat the net_input vector n times along the rows
                                         # so we now have a matrix
print(net_inputs)

# h_i and h_j should exclude input from i and j from h_ij
h_i = np.diagonal(old_weights) * mem # this obtains the input each neuron receives from itself
h_i = h_i[:, np.newaxis]             # turn h_i into a column vector so we can subtract from hij appropriately
print(h_i)

h_j = old_weights * mem              # element-wise multiply each row of old-weights by mem
np.fill_diagonal(h_j,0)
print(h_j)

hij = net_inputs - h_i - h_j
print(hij)

post_synaptic  = hij * mem
print(post_synaptic)

pre_synaptic = post_synaptic.T #equivalent to np.outer(hij,mem)
print(pre_synaptic)

new_weights = old_weights + (1./n)*(hebbian_term - pre_synaptic - post_synaptic)
print(new_weights)


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
Beautifully noisy. Can barely see anything in it. But now if we start with that, and apply the weights, it should recover the spurious pattern!
Can you write Python code for it?

steps = 100000

recovered_spurious_meme = flow(noisy_spurious_meme, average_weights, theta=theta, steps=steps)

plt.imshow(recovered_spurious_meme, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
I could force those 0 values to -1 or 1, but that biases the pattern towards deal and neil, respectively (at least, testing suggested this -- I think because Aang has more black pixels and Deal has more white pixels). So, I'll leave them in. I could probably solve this by randomly setting 0's to 1 or -1, but naw.
Can you write Python code for it?

#deal_with_neil[deal_with_neil == 0] = -1
#np.unique(deal_with_neil)

plt.imshow(deal_with_neil, cmap='Greys', interpolation='nearest');

recovered_deal_with_neil = flow(deal_with_neil, average_weights, theta = theta, steps = steps)

plt.imshow(recovered_deal_with_neil, cmap='Greys', interpolation='nearest');


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
Let's load in a meme. I'm partial to 'Deal with it'.
Can you write Python code for it?

#deal = 2 * np.random.binomial(1,.5,size=(5,5)) - 1
#deal = imread('obama.png', mode="L")
deal = imread('small-deal-with-it-with-text.jpg', mode="L")
print(deal.shape)
deal = deal.astype(int)

np.unique(deal)


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
Now we can start with that, and use the weights to update it. We'll update the units asynchronously (one at a time), and keep track of the energy of the network every so often.
Can you write Python code for it?

def flow(pattern, weights, theta=0, steps = 50000):

    pattern_flat = pattern.flatten()

    if isinstance(theta, numbers.Number):
        thetas = np.zeros(len(pattern_flat)) + theta

    for step in range(steps):
        unit = np.random.randint(low=0, high=(len(pattern_flat)-1))
        unit_weights = weights[unit,:]
        net_input = np.dot(unit_weights,pattern_flat)
        pattern_flat[unit] = 1 if (net_input > thetas[unit]) else -1
        #pattern_flat[unit] = np.sign(net_input)

        if (step % 10000) == 0:
            energy = -0.5*np.dot(np.dot(pattern_flat.T,weights),pattern_flat) + np.dot(thetas,pattern_flat)
            print("Energy at step {:05d} is now {}".format(step,energy))

    evolved_pattern = np.reshape(a=pattern_flat, newshape=(pattern.shape[0],pattern.shape[1]))

    return evolved_pattern

steps = 50000
theta = 0

noisy_deal_evolved = flow(noisy_deal, deal_weights, theta = theta, steps = steps)

plt.imshow(noisy_deal_evolved, cmap='Greys', interpolation='nearest');


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
Pretty noisy. Only Aang, and kiiiiinda the Deal with It, are visible. Now make a noisy version of that combination.
Can you write Python code for it?

noisy_spurious_meme = noisify(pattern=spurious_meme)

plt.imshow(noisy_spurious_meme, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
Sweet. So now we can try something like feeding it a pattern that is halfway between the two patterns -- it should eventually settle into one of them! Who has greater meme strength!??!
Can you write Python code for it?

deal_with_neil = (woah + deal) / 2
print(np.unique(deal_with_neil))


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
To convert this to a 1 bit image, I convert everything darker than some threshold to black (1), and everything else to white (-1). Experimenting a bit with the particular image of the 'deal with it meme' that I have, a threshold of 80 seemed to work reasonably. The resulting image is still a bit rough around the edges, but it's recognizable.
Can you write Python code for it?

bvw_threshold = 80

deal[deal <= bvw_threshold] = -1
deal[deal >  bvw_threshold] = 1
deal = -deal
deal

np.unique(deal)

plt.imshow(deal, cmap='Greys', interpolation='nearest');


I am working on the file "Hopfield Network of memes--Storkey Learning Rule.ipynb".
The first task is:
Voila.
Training the network on a second pattern
The cooler thing about the Hopfield networks is that they can encode multiple patterns (to a limit depending on the training regimen, and the number of units). So let's try another maymay.
I got the next meme from here, and then tweaked its levels in Mac's preview so that it'd translate nicely to a 1 bit (black or white) image.
Can you write Python code for it?

# woah = imread('woah.png', mode="L")
woah = imageio.imread('aang.jpg')[:,:,0]
woah = woah.astype(int)
woah[woah >= 1] = 1
woah[woah < 1] = -1
woah = -woah

np.unique(woah)

plt.imshow(woah, cmap='Greys', interpolation='nearest');


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
And it sure as heck did.
Animations
Let's make some nifty animations of the networks recovering a pattern (learned or spurious) from an input. Let's make the input more or less pure noise, by increasing numb_flipped.
Can you write Python code for it?

noise = noisify(pattern=shrek, numb_flipped=30)

plt.imshow(noise, cmap='Greys', interpolation='nearest')
plt.show()

steps = 50000

recovered_meme = flow(noise, average_weights, theta=theta, steps=steps)

plt.imshow(recovered_meme, cmap='Greys', interpolation='nearest')
plt.show()

steps = 50000
noisy_meme_flat = noise.flatten()

for step in range(steps):
    unit = np.random.randint(low=0, high=(len(noisy_meme_flat)-1))
    unit_weights = average_weights[unit,:]
    net_input = np.dot(unit_weights,noisy_meme_flat)

    noisy_meme_flat[unit] = 1 if (net_input > theta) else -1

    recovered_meme = np.reshape(a=noisy_meme_flat, newshape=(len(shrek),len(shrek)))
    if not (step % 1000):
        imageio.imsave('recovery_steps/shrek/recovered_meme{:05d}.jpg'.format(step), recovered_meme.astype(np.uint8))

filenames = sorted([x for x in os.listdir('recovery_steps/shrek') if x[-4:] == '.jpg'])
images = []
for filename in filenames:
    images.append(imageio.imread('recovery_steps/shrek/'+filename))
imageio.mimsave('recovery_steps/shrek/meme_movie.gif', images, duration=0.2)


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
As expected, energy decreases as the units' activations are updated. Now plot the 'evolved' pattern.
Can you write Python code for it?

plt.imshow(noisy_deal_evolved, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Assuming the cells/pixels of 0 were unaltered, if you run that a few times, you'll notice that sometimes it settles on Aang, and sometimes it settles on Deal!!!
Spurious patterns
Hopfield networks can also settle onto 'spurious patterns' (patterns that the network wasn't trained on). For each stored pattern x, -x is a spurious pattern. But also, any linear combination of the of the learned patterns can be a spurious pattern. So let's learn a third pattern, and then see the network stabilize on a simple combination of the three patterns.
Can you write Python code for it?

shrek = imageio.imread('shrek.jpg')
shrek = shrek.astype(int)

shrek_threshold = 200
shrek[shrek <  shrek_threshold] = -1
shrek[shrek >= shrek_threshold] = 1

shrek[120:,:] = 1

shrek = -shrek

plt.imshow(shrek, cmap='Greys', interpolation='nearest')
plt.show()

flattened_shrek = shrek.flatten()

flatlen = len(flattened_shrek)

shrek_weights = np.outer(flattened_shrek,flattened_shrek) - np.identity(len(flattened_shrek))

average_weights = (woah_weights + deal_weights + shrek_weights) / 3

noisy_shrek = noisify(pattern=shrek)

plt.imshow(noisy_shrek, cmap='Greys', interpolation='nearest')
plt.show()

recovered_shrek = flow(noisy_shrek, average_weights, theta=theta, steps=steps)

plt.imshow(recovered_shrek, cmap='Greys', interpolation='nearest')
plt.show()

recovered_woah = flow(noisy_woah, average_weights, theta=theta, steps=steps)

plt.imshow(recovered_woah, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Beautifully noisy. Can barely see anything in it. But now if we start with that, and apply the weights, it should recover the spurious pattern!
Can you write Python code for it?


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Cool. So now we make some weights for this image...
Can you write Python code for it?

flattened_woah = woah.flatten()

flatlen = len(flattened_woah)

woah_weights = np.outer(flattened_woah,flattened_woah) - np.identity(len(flattened_woah))


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
I could force those 0 values to -1 or 1, but that biases the pattern towards deal and neil, respectively (at least, testing suggested this -- I think because Aang has more black pixels and Deal has more white pixels). So, I'll leave them in. I could probably solve this by randomly setting 0's to 1 or -1, but naw.
Can you write Python code for it?

#deal_with_neil[deal_with_neil == 0] = -1
#np.unique(deal_with_neil)

plt.imshow(deal_with_neil, cmap='Greys', interpolation='nearest')
plt.show()

recovered_deal_with_neil = flow(deal_with_neil, average_weights, theta = theta, steps = steps)

plt.imshow(recovered_deal_with_neil, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Let's load in a meme. I'm partial to 'Deal with it'.
Can you write Python code for it?

# deal = imread('small-deal-with-it-with-text.jpg', mode="L")
deal = imageio.imread('small-deal-with-it-with-text.jpg')
print(deal.shape)
deal = deal.astype(int)

np.unique(deal)


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Now make weights. For now, we'll use the Hebbian learning rule, whereby two units have a positive weight (+1) if their activation is the same, and a negative weight (-1) if their activations are differnet. We also stipulate that a neuron have no weight with itself.
Can you write Python code for it?

flattened_deal = deal.flatten()
flattened_deal.shape


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Now start with a noisy version of the image. We'll just flip a certain number of random pixels on each row of the image.
Can you write Python code for it?

def noisify(pattern, numb_flipped=30):

    noisy_pattern = pattern.copy()

    for idx, row in enumerate(noisy_pattern):
        choices = np.random.choice(range(len(row)), numb_flipped)
        noisy_pattern[idx,choices] = -noisy_pattern[idx,choices]

    return noisy_pattern

noisy_deal = noisify(pattern=deal)

plt.imshow(noisy_deal, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Now we can start with that, and use the weights to update it. We'll update the units asynchronously (one at a time).
While we update the units, let's keep track of the energy in the network:

Updating the units' activations causes the network to move toward a local minimum in that function. Put more informally, updating the units puts the network a more 'relaxed' state.
Efficient numpy matrix math for computing the energy taken from here.
Can you write Python code for it?

def flow(pattern, weights, theta=0, steps = 50000):

    pattern_flat = pattern.flatten()

    if (type(theta) == float) or (type(theta) == int):
        thetas = np.zeros(len(pattern_flat)) + theta

    for step in range(steps):
        unit = np.random.randint(low=0, high=(len(pattern_flat)-1))
        unit_weights = weights[unit,:]
        net_input = np.dot(unit_weights,pattern_flat)
        pattern_flat[unit] = 1 if (net_input > thetas[unit]) else -1

        if (step % 10000) == 0:
            energy = -0.5*np.dot(np.dot(pattern_flat.T,weights),pattern_flat) + np.dot(thetas,pattern_flat)
            print("Energy at step {:05d} is now {}".format(step,energy))

    evolved_pattern = np.reshape(a=pattern_flat, newshape=(pattern.shape[0],pattern.shape[1]))

    return evolved_pattern

steps = 50000
theta = 0

noisy_deal_evolved = flow(noisy_deal, deal_weights, theta = theta, steps = steps)


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Now, let's make a noisy Aang, and have the network try to recover the clean, pristine Aang.
Can you write Python code for it?

noisy_woah = noisify(pattern=woah)

plt.imshow(noisy_woah, cmap='Greys', interpolation='nearest')
plt.show()

recovered_woah = flow(noisy_woah, average_weights, theta = theta, steps = steps)

plt.imshow(recovered_woah, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Okay, now let's make a spurious pattern. Any linear combination will do.
Can you write Python code for it?

spurious_meme = shrek + deal + woah
np.unique(spurious_meme)

spurious_meme[spurious_meme > 0] = 1
spurious_meme[spurious_meme < 0] = -1

plt.imshow(spurious_meme, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Sweet. So now we can try something like feeding it a pattern that is halfway between the two patterns -- it should eventually settle into one of them! Who has greater meme strength!??!
Can you write Python code for it?


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
This next cell can take a little while if the image is large...
Can you write Python code for it?

flatlen = len(flattened_deal)

deal_weights = np.outer(flattened_deal,flattened_deal) - np.identity(len(flattened_deal))
deal_weights[:5,:5]


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
To convert this to a 1 bit image, I convert everything darker than some threshold to black (1), and everything else to white (-1). Experimenting a bit with the particular image of the 'deal with it meme' that I have, a threshold of 80 seemed to work reasonably. The resulting image is still a bit rough around the edges, but it's recognizable.
Can you write Python code for it?

bvw_threshold = 80

deal[deal <= bvw_threshold] = -1
deal[deal >  bvw_threshold] = 1
deal = -deal
deal

np.unique(deal)

plt.imshow(deal, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Hopfield Network of memes.ipynb".
The first task is:
Voila.
Training the network on a second pattern
The cooler thing about the Hopfield networks is that they can encode multiple patterns (to a limit depending on the training regimen, and the number of units). So let's try another maymay.
I got the next meme from here, and then tweaked its levels in Mac's preview so that it'd translate nicely to a 1 bit (black or white) image.
Can you write Python code for it?

# woah = imread('woah.png', mode="L")
# woah = imageio.imread('woah.png')
woah = imageio.imread('aang.jpg')[:,:,0]
woah = woah.astype(int)
woah[woah <  100] = -1
woah[woah >= 100] = 1

plt.imshow(woah, cmap='Greys', interpolation='nearest')
plt.show()


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
As you can see we go from the raw pixels of the images to increasingly abstract and compact representations. The representations downstream start highlighting what the network pays attention to, and they show fewer and fewer features being "activated"; most are set to zero. This is called "sparsity." Representation sparsity is a key feature of deep learning.
These representations carry increasingly less information about the original pixels of the image, but increasingly refined information about the class of the image. You can think of a convnet (or a deep network in general) as an information distillation pipeline.
Clean Up
Before running the next exercise, run the following cell to terminate the kernel and free memory resources:
Can you write Python code for it?

import os, signal
os.kill(os.getpid(), signal.SIGKILL)


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
Let's find out the total number of horse and human images in the directories:
Can you write Python code for it?

print('total training horse images:', len(os.listdir(train_horse_dir)))
print('total training human images:', len(os.listdir(train_human_dir)))


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
Now let's take a look at a few pictures to get a better sense of what they look like. First, configure the matplot parameters:
Can you write Python code for it?

%matplotlib inline

import matplotlib.pyplot as plt
import matplotlib.image as mpimg

# Parameters for our graph; we'll output images in a 4x4 configuration
nrows = 4
ncols = 4

# Index for iterating over images
pic_index = 0


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
Now, display a batch of 8 horse and 8 human pictures. You can rerun the cell to see a fresh batch each time:
Can you write Python code for it?

# Set up matplotlib fig, and size it to fit 4x4 pics
fig = plt.gcf()
fig.set_size_inches(ncols * 4, nrows * 4)

pic_index += 8
next_horse_pix = [os.path.join(train_horse_dir, fname)
                for fname in train_horse_names[pic_index-8:pic_index]]
next_human_pix = [os.path.join(train_human_dir, fname)
                for fname in train_human_names[pic_index-8:pic_index]]

for i, img_path in enumerate(next_horse_pix+next_human_pix):
  # Set up subplot; subplot indices start at 1
  sp = plt.subplot(nrows, ncols, i + 1)
  sp.axis('Off') # Don't show axes (or gridlines)

  img = mpimg.imread(img_path)
  plt.imshow(img)

plt.show()


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
Now, let's see what the filenames look like in the horses and humans training directories:
Can you write Python code for it?

train_horse_names = os.listdir(train_horse_dir)
print(train_horse_names[:10])

train_human_names = os.listdir(train_human_dir)
print(train_human_names[:10])


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
The "output shape" column shows how the size of your feature map evolves in each successive layer. The convolution layers reduce the size of the feature maps by a bit due to padding, and each pooling layer halves the dimensions.
Next, we'll configure the specifications for model training. We will train our model with the binary_crossentropy loss, because it's a binary classification problem and our final activation is a sigmoid. (For a refresher on loss metrics, see the Machine Learning Crash Course.) We will use the rmsprop optimizer with a learning rate of 0.001. During training, we will want to monitor classification accuracy.
NOTE: In this case, using the RMSprop optimization algorithm is preferable to stochastic gradient descent (SGD), because RMSprop automates learning-rate tuning for us. (Other optimizers, such as Adam and Adagrad, also automatically adapt the learning rate during training, and would work equally well here.)
Can you write Python code for it?

from tensorflow.keras.optimizers import RMSprop

model.compile(loss='binary_crossentropy',
              optimizer=RMSprop(lr=0.001),
              metrics=['acc'])


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
The contents of the .zip are extracted to the base directory /tmp/horse-or-human, which in turn each contain horses and humans subdirectories.
In short: The training set is the data that is used to tell the neural network model that 'this is what a horse looks like', 'this is what a human looks like' etc.
One thing to pay attention to in this sample: We do not explicitly label the images as horses or humans. If you remember with the handwriting example earlier, we had labelled 'this is a 1', 'this is a 7' etc.  Later you'll see something called an ImageGenerator being used -- and this is coded to read images from subdirectories, and automatically label them from the name of that subdirectory. So, for example, you will have a 'training' directory containing a 'horses' directory and a 'humans' one. ImageGenerator will label the images appropriately for you, reducing a coding step.
Let's define each of these directories:
Can you write Python code for it?

# Directory with our training horse pictures
train_horse_dir = os.path.join('/tmp/horse-or-human/horses')

# Directory with our training human pictures
train_human_dir = os.path.join('/tmp/horse-or-human/humans')


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
The following python code will use the OS library to use Operating System libraries, giving you access to the file system, and the zipfile library allowing you to unzip the data.
Can you write Python code for it?

import os
import zipfile

local_zip = '/tmp/horse-or-human.zip'
zip_ref = zipfile.ZipFile(local_zip, 'r')
zip_ref.extractall('/tmp/horse-or-human')
zip_ref.close()


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
The model.summary() method call prints a summary of the NN
Can you write Python code for it?


I am working on the file "Horse_or_Human_NoValidation.ipynb".
The first task is:
We then add convolutional layers as in the previous example, and flatten the final result to feed into the densely connected layers.
Finally we add the densely connected layers.
Note that because we are facing a two-class classification problem, i.e. a binary classification problem, we will end our network with a sigmoid activation, so that the output of our network will be a single scalar between 0 and 1, encoding the probability that the current image is class 1 (as opposed to class 0).
Can you write Python code for it?

model = tf.keras.models.Sequential([
    # Note the input shape is the desired size of the image 300x300 with 3 bytes color
    # This is the first convolution
    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(300, 300, 3)),
    tf.keras.layers.MaxPooling2D(2, 2),
    # The second convolution
    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The third convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The fourth convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The fifth convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # Flatten the results to feed into a DNN
    tf.keras.layers.Flatten(),
    # 512 neuron hidden layer
    tf.keras.layers.Dense(512, activation='relu'),
    # Only 1 output neuron. It will contain a value from 0-1 where 0 for 1 class ('horses') and 1 for the other ('humans')
    tf.keras.layers.Dense(1, activation='sigmoid')
])


I am working on the file "Hospital_readmit.ipynb".
The first task is:
Preliminary Analysis
Can you write Python code for it?

# deal with missing and inconvenient portions of data
clean_hospital_read_df = hospital_read_df[hospital_read_df['Number of Discharges'] != 'Not Available']
clean_hospital_read_df.loc[:,'Number of Discharges'] = clean_hospital_read_df['Number of Discharges'].astype(int)
clean_hospital_read_df = clean_hospital_read_df.sort_values('Number of Discharges')

# generate a scatterplot for number of discharges vs. excess rate of readmissions
# lists work better with matplotlib scatterplot function
x = [a for a in clean_hospital_read_df['Number of Discharges'][81:-3]]
y = list(clean_hospital_read_df['Excess Readmission Ratio'][81:-3])

fig, ax = plt.subplots(figsize=(8,5))
ax.scatter(x, y,alpha=0.2)

#ax.plot(x,y)

ax.fill_between([0,350], 1.15, 2, facecolor='red', alpha = .15, interpolate=True)
ax.fill_between([800,2500], .5, .95, facecolor='green', alpha = .15, interpolate=True)

ax.set_xlim([0, max(x)])
ax.set_xlabel('Number of discharges', fontsize=12)
ax.set_ylabel('Excess rate of readmissions', fontsize=12)
ax.set_title('Scatterplot of number of discharges vs. excess rate of readmissions', fontsize=14)

ax.grid(True)
fig.tight_layout()


I am working on the file "Hospital_readmit.ipynb".
The first task is:
Preliminary Report
Read the following results/report. While you are reading it, think about if the conclusions are correct, incorrect, misleading or unfounded. Think about what you would change or what additional analyses you would perform.
A. Initial observations based on the plot above
+ Overall, rate of readmissions is trending down with increasing number of discharges
+ With lower number of discharges, there is a greater incidence of excess rate of readmissions (area shaded red)
+ With higher number of discharges, there is a greater incidence of lower rates of readmissions (area shaded green)
B. Statistics
+ In hospitals/facilities with number of discharges < 100, mean excess readmission rate is 1.023 and 63% have excess readmission rate greater than 1
+ In hospitals/facilities with number of discharges > 1000, mean excess readmission rate is 0.978 and 44% have excess readmission rate greater than 1
C. Conclusions
+ There is a significant correlation between hospital capacity (number of discharges) and readmission rates.
+ Smaller hospitals/facilities may be lacking necessary resources to ensure quality care and prevent complications that lead to readmissions.
D. Regulatory policy recommendations
+ Hospitals/facilties with small capacity (< 300) should be required to demonstrate upgraded resource allocation for quality care to continue operation.
+ Directives and incentives should be provided for consolidation of hospitals and facilities to have a smaller number of them with higher capacity and number of discharges.
Exercise
Include your work on the following in this notebook and submit to your Github account.
A. Do you agree with the above analysis and recommendations? Why or why not?
B. Provide support for your arguments and your own recommendations with a statistically sound analysis:

Setup an appropriate hypothesis test.
Compute and report the observed significance value (or p-value).
Report statistical significance for $\alpha$ = .01.
Discuss statistical significance and practical...
Can you write Python code for it?

hospital_cap_small= clean_hospital_read_df[clean_hospital_read_df['Number of Discharges'].astype(int)<100]
hospital_cap_small = hospital_cap_small[hospital_cap_small['Number of Discharges'].astype(int) != 0]
hospital_small=sorted(hospital_cap_small['Excess Readmission Ratio'])

hospital_cap_big= clean_hospital_read_df[clean_hospital_read_df['Number of Discharges'].astype(int)>1000]
hospital_cap_big = hospital_cap_big[hospital_cap_big['Number of Discharges'].astype(int) != 0]
hospital_big=sorted(hospital_cap_big['Excess Readmission Ratio'])

from scipy import stats
statistic = stats.f_oneway(hospital_small,hospital_big)
statistic


I am working on the file "HouMath/Face_classification_HouMath_XGB_01.ipynb".
The first task is:
Data Preparation and Model Selection
Now we are ready to test the XGB approach, along the way confusion matrix and f1_score are imported as metric for classification, as well as GridSearchCV, which is an excellent tool for parameter optimization.
Can you write Python code for it?

import xgboost as xgb
import numpy as np
from sklearn.metrics import confusion_matrix, f1_score
from classification_utilities import display_cm, display_adj_cm
from sklearn.model_selection import GridSearchCV

X_train = training_data.drop(['Facies', 'Well Name','Formation','Depth'], axis = 1 )
Y_train = training_data['Facies' ] - 1
dtrain = xgb.DMatrix(X_train, Y_train)


I am working on the file "HouMath/Face_classification_HouMath_XGB_01.ipynb".
The first task is:
The accuracy function and accuracy_adjacent function are defined in teh following to quatify the prediction correctness.
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))


I am working on the file "HouMath/Face_classification_HouMath_XGB_01.ipynb".
The first task is:
The typical range for learning rate is around 0.01~0.2, so we vary ther learning rate a bit and at the same time, scan over the number of boosted trees to fit. This will take a little bit of time to finish.
Can you write Python code for it?

print("Parameter optimization")
grid_search1 = GridSearchCV(xgb1,{'learning_rate':[0.05,0.01,0.1,0.2] , 'n_estimators':[200,400,600,800]},
                                   scoring='accuracy' , n_jobs = 4)
grid_search1.fit(X_train,Y_train)
print("Best Set of Parameters")
grid_search1.grid_scores_, grid_search1.best_params_, grid_search1.best_score_


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Data Preparation and Model Selection
Now we are ready to test the XGB approach, and will use confusion matrix and f1_score, which were imported, as metric for classification, as well as GridSearchCV, which is an excellent tool for parameter optimization.
Can you write Python code for it?

X_train = training_data.drop(['Facies', 'Well Name','Formation','Depth'], axis = 1 )
Y_train = training_data['Facies' ] - 1
dtrain = xgb.DMatrix(X_train, Y_train)

train = X_train.copy()

train['Facies']=Y_train

train.head()


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Next we use our tuned final model to do cross validation on the training data set. One of the wells will be used as test data and the rest will be the training data. Each iteration, a different well is chosen.
Can you write Python code for it?

# Load data
filename = './facies_vectors.csv'
data = pd.read_csv(filename)

# Change to category data type
data['Well Name'] = data['Well Name'].astype('category')
data['Formation'] = data['Formation'].astype('category')

# Leave one well out for cross validation
well_names = data['Well Name'].unique()
f1=[]
for i in range(len(well_names)):

    # Split data for training and testing
    X_train = data.drop(['Facies', 'Formation','Depth'], axis = 1 )
    Y_train = data['Facies' ] - 1

    train_X = X_train[X_train['Well Name'] != well_names[i] ]
    train_Y = Y_train[X_train['Well Name'] != well_names[i] ]
    test_X  = X_train[X_train['Well Name'] == well_names[i] ]
    test_Y  = Y_train[X_train['Well Name'] == well_names[i] ]

    train_X = train_X.drop(['Well Name'], axis = 1 )
    test_X = test_X.drop(['Well Name'], axis = 1 )

    # Final recommended model based on the extensive parameters search
    model_final = XGBClassifier(base_score=0.5, colsample_bylevel=1,
       colsample_bytree=0.8, gamma=0.2,
       learning_rate=0.01, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=432, nthread=4,
       objective='multi:softmax', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=1,
       subsample=0.6)
    # Train the model based on training data
    model_final.fit( train_X , train_Y , eval_metric = 'merror' )

    # Predict on the test set
    predictions = model_final.predict(test_X)

    # Print report
    print ("\n------------------------------------------------------")
    print ("Validation on the leaving out well " + well_names[i])
    conf = confusion_matrix( test_Y, predictions, labels = np.arange(9) )
    print ("\nModel Report")
    print ("-Accuracy: %.6f" % ( accuracy(conf) ))
    print ("-Adjacent Accuracy: %.6f" % ( accuracy_adjacent(conf, adjacent_facies) ))
    print ("-F1 Score: %.6f" % ( f1_score ( test_Y , predictions , labels = np.arange(9), average = 'weighted' ) ))
    f1.append(f1_score ( test_Y , predictions...


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Step 2: Tune max_depth and min_child_weight
Can you write Python code for it?

param_test1={
    'max_depth':range(3,10,2),
    'min_child_weight':range(1,6,2)
}

gs1 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=5,
       min_child_weight=1, n_estimators=290, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test1,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs1.fit(train[features],train[target])
gs1.grid_scores_, gs1.best_params_,gs1.best_score_

param_test2={
    'max_depth':[8,9,10],
    'min_child_weight':[1,2]
}

gs2 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=5,
       min_child_weight=1, n_estimators=290, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test2,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs2.fit(train[features],train[target])
gs2.grid_scores_, gs2.best_params_,gs2.best_score_

gs2.best_estimator_


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Step 3: Tune gamma
Can you write Python code for it?

param_test3={
   'gamma':[i/10.0 for i in range(0,5)]
}

gs3 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=290, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test3,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs3.fit(train[features],train[target])
gs3.grid_scores_, gs3.best_params_,gs3.best_score_

xgb2 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.8,
    colsample_bytree=0.8,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb2,train,features)

xgb2


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Step 4:Tune subsample and colsample_bytree
Can you write Python code for it?

param_test4={
    'subsample':[i/10.0 for i in range(6,10)],
    'colsample_bytree':[i/10.0 for i in range(6,10)]
}

gs4 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test4,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs4.fit(train[features],train[target])
gs4.grid_scores_, gs4.best_params_,gs4.best_score_

param_test4b={
    'subsample':[i/10.0 for i in range(5,7)],
}

gs4b = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test4b,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs4b.fit(train[features],train[target])
gs4b.grid_scores_, gs4b.best_params_,gs4b.best_score_


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Step 5: Tuning Regularization Parameters
Can you write Python code for it?

param_test5={
    'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]
}

gs5 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.6),param_grid=param_test5,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs5.fit(train[features],train[target])
gs5.grid_scores_, gs5.best_params_,gs5.best_score_

param_test6={
    'reg_alpha':[0, 0.001, 0.005, 0.01, 0.05]
}

gs6 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.6),param_grid=param_test6,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs6.fit(train[features],train[target])
gs6.grid_scores_, gs6.best_params_,gs6.best_score_

xgb3 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.05,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb3,train,features)

xgb3


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
Step 6: Reducing Learning Rate
Can you write Python code for it?

xgb4 = XGBClassifier(
    learning_rate = 0.01,
    n_estimators=5000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.05,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb4,train,features)

xgb4


I am working on the file "HouMath/Face_classification_HouMath_XGB_02.ipynb".
The first task is:
The accuracy function and accuracy_adjacent function are defined in the following to quatify the prediction correctness.
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

target='Facies'


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Before processing further, we define a functin which will help us create XGBoost models and perform cross-validation.
Can you write Python code for it?

def modelfit(alg, dtrain, features, useTrainCV=True,
             cv_fold=10,early_stopping_rounds = 50):
    if useTrainCV:
        xgb_param = alg.get_xgb_params()
        xgb_param['num_class']=9
        xgtrain = xgb.DMatrix(train[features].values,label = train[target].values)
        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=
                          alg.get_params()['n_estimators'],nfold=cv_fold,
                          metrics='merror',early_stopping_rounds = early_stopping_rounds)
        alg.set_params(n_estimators=cvresult.shape[0])

    #Fit the algorithm on the data
    alg.fit(dtrain[features], dtrain[target],eval_metric='merror')

    #Predict training set:
    dtrain_prediction = alg.predict(dtrain[features])
    dtrain_predprob = alg.predict_proba(dtrain[features])[:,1]

    #Pring model report
    print ("\nModel Report")
    print ("Accuracy : %.4g" % accuracy_score(dtrain[target],
                                            dtrain_prediction))
    print ("F1 score (Train) : %f" % f1_score(dtrain[target],
                                            dtrain_prediction,average='weighted'))
    feat_imp = pd.Series(alg.booster().get_fscore()).sort_values(ascending=False)
    feat_imp.plot(kind='bar',title='Feature Importances')
    plt.ylabel('Feature Importance Score')

features =[x for x in X_train.columns]
features


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Data Preparation and Model Selection
Now we are ready to test the XGB approach, and will use confusion matrix and f1_score, which were imported, as metric for classification, as well as GridSearchCV, which is an excellent tool for parameter optimization.
Can you write Python code for it?

import xgboost as xgb
X_train = training_data.drop(['Facies', 'Well Name','Formation','Depth'], axis = 1 )
Y_train = training_data['Facies' ] - 1
dtrain = xgb.DMatrix(X_train, Y_train)

train = X_train.copy()

train['Facies']=Y_train

train.head()


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
General Approach for Parameter Tuning
We are going to preform the steps as follows:
1.Choose a relatively high learning rate, e.g., 0.1. Usually somewhere between 0.05 and 0.3 should work for different problems.
2.Determine the optimum number of tress for this learning rate.XGBoost has a very usefull function called as "cv" which performs cross-validation at each boosting iteration and thus returns the optimum number of tress required.
3.Tune tree-based parameters(max_depth, min_child_weight, gamma, subsample, colsample_bytree) for decided learning rate and number of trees.
4.Tune regularization parameters(lambda, alpha) for xgboost which can help reduce model complexity and enhance performance.
5.Lower the learning rate and decide the optimal parameters.
Step 1:Fix learning rate and number of estimators for tuning tree-based parameters
In order to decide on boosting parameters, we need to set some initial values of other parameters. Lets take the following values:
1.max_depth = 5
2.min_child_weight = 1
3.gamma = 0
4.subsample, colsample_bytree = 0.8 : This is a commonly used used start value.
5.scale_pos_weight = 1
Please note that all the above are just initial estimates and will be tuned later. Lets take the default learning rate of 0.1 here and check the optimum number of trees using cv function of xgboost. The function defined above will do it for us.
Can you write Python code for it?

from xgboost import XGBClassifier
xgb1 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=5,
    min_child_weight=1,
    gamma = 0,
    subsample=0.8,
    colsample_bytree=0.8,
    objective='multi:softmax',
    nthread =4,
    seed = 123,
)

modelfit(xgb1, train, features)

xgb1


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Model from all data set
Can you write Python code for it?

# Load data
filename = './facies_vectors.csv'
data = pd.read_csv(filename)

# Change to category data type
data['Well Name'] = data['Well Name'].astype('category')
data['Formation'] = data['Formation'].astype('category')

# Split data for training and testing
X_train_all = data.drop(['Facies', 'Formation','Depth'], axis = 1 )
Y_train_all = data['Facies' ] - 1

X_train_all = X_train_all.drop(['Well Name'], axis = 1)

# Final recommended model based on the extensive parameters search
model_final = XGBClassifier(base_score=0.5, colsample_bylevel=1,
       colsample_bytree=0.8, gamma=0.2,
       learning_rate=0.01, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=432, nthread=4,
       objective='multi:softmax', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=1,
       subsample=0.6)

# Train the model based on training data
model_final.fit(X_train_all , Y_train_all , eval_metric = 'merror' )

# Leave one well out for cross validation
well_names = data['Well Name'].unique()
f1=[]

for i in range(len(well_names)):

    X_train = data.drop(['Facies', 'Formation','Depth'], axis = 1 )
    Y_train = data['Facies' ] - 1

    train_X = X_train[X_train['Well Name'] != well_names[i] ]
    train_Y = Y_train[X_train['Well Name'] != well_names[i] ]
    test_X  = X_train[X_train['Well Name'] == well_names[i] ]
    test_Y  = Y_train[X_train['Well Name'] == well_names[i] ]

    train_X = train_X.drop(['Well Name'], axis = 1 )
    test_X = test_X.drop(['Well Name'], axis = 1 )
    #print(test_Y)
    predictions = model_final.predict(test_X)

    # Print report
    print ("\n------------------------------------------------------")
    print ("Validation on the leaving out well " + well_names[i])
    conf = confusion_matrix( test_Y, predictions, labels = np.arange(9) )
    print ("\nModel Report")
    print ("-Accuracy: %.6f" % ( accuracy(conf) ))
    print ("-Adjacent Accuracy: %.6f" % ( accuracy_adjacent(conf, adjacent_facies) ))
    print ("-F1 Score:...


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Step 2: Tune max_depth and min_child_weight
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
param_test1={
    'max_depth':range(3,10,2),
    'min_child_weight':range(1,6,2)
}

gs1 = GridSearchCV(xgb1,param_grid=param_test1,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs1.fit(train[features],train[target])
gs1.grid_scores_, gs1.best_params_,gs1.best_score_

param_test2={
    'max_depth':[8,9,10],
    'min_child_weight':[1,2]
}

gs2 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=5,
       min_child_weight=1, n_estimators=290, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test2,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs2.fit(train[features],train[target])
gs2.grid_scores_, gs2.best_params_,gs2.best_score_

gs2.best_estimator_


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Step 3: Tune gamma
Can you write Python code for it?

param_test3={
   'gamma':[i/10.0 for i in range(0,5)]
}

gs3 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=370, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.8),param_grid=param_test3,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs3.fit(train[features],train[target])
gs3.grid_scores_, gs3.best_params_,gs3.best_score_

xgb2 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.8,
    colsample_bytree=0.8,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb2,train,features)

xgb2


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Step 5: Tuning Regularization Parameters
Can you write Python code for it?

param_test5={
    'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]
}

gs5 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.6),param_grid=param_test5,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs5.fit(train[features],train[target])
gs5.grid_scores_, gs5.best_params_,gs5.best_score_

param_test6={
    'reg_alpha':[0, 0.001, 0.005, 0.01, 0.05]
}

gs6 = GridSearchCV(XGBClassifier(colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, n_estimators=236, nthread=4,
       objective='multi:softprob', reg_alpha=0, reg_lambda=1,
       scale_pos_weight=1, seed=123,subsample=0.6),param_grid=param_test6,
                   scoring='accuracy', n_jobs=4,iid=False, cv=5)
gs6.fit(train[features],train[target])
gs6.grid_scores_, gs6.best_params_,gs6.best_score_

xgb3 = XGBClassifier(
    learning_rate = 0.1,
    n_estimators=1000,
    max_depth=9,
    min_child_weight=1,
    gamma = 0.2,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.05,
    objective='multi:softmax',
    nthread =4,
    scale_pos_weight=1,
    seed = seed,
)
modelfit(xgb3,train,features)

xgb3

model = XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=0.8,
       gamma=0.2, learning_rate=0.1, max_delta_step=0, max_depth=9,
       min_child_weight=1, missing=None, n_estimators=122, nthread=4,
       objective='multi:softprob', reg_alpha=0.05, reg_lambda=1,
       scale_pos_weight=1, seed=123, silent=True, subsample=0.6)
model.fit(X_train, Y_train)
xgb.plot_importance(model)


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Step 6: Reducing Learning Rate
Can you write Python code for it?


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
The accuracy function and accuracy_adjacent function are defined in the following to quatify the prediction correctness.
Can you write Python code for it?


I am working on the file "HouMath/Face_classification_HouMath_XGB_03.ipynb".
The first task is:
Use final model to predict the given test data set
Can you write Python code for it?

# Load test data
test_data = pd.read_csv('validation_data_nofacies.csv')
test_data['Well Name'] = test_data['Well Name'].astype('category')
X_test = test_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
# Predict facies of unclassified data
Y_predicted = model_final.predict(X_test)
test_data['Facies'] = Y_predicted + 1
# Store the prediction
test_data.to_csv('Prediction3.csv')

test_data


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Before processing further, we define a functin which will help us create XGBoost models and perform cross-validation.
Can you write Python code for it?

skf = StratifiedKFold(n_splits=5)

cv = skf.split(X_train, Y_train)

def modelfit(alg, Xtrain, Ytrain, useTrainCV=True, cv_fold=skf):

    #Fit the algorithm on the data
    alg.fit(Xtrain, Ytrain,eval_metric='merror')

    #Predict training set:
    dtrain_prediction = alg.predict(Xtrain)
    #dtrain_predprob = alg.predict_proba(Xtrain)[:,1]

    #Pring model report
    print ("\nModel Report")
    print ("Accuracy : %.4g" % accuracy_score(Ytrain,dtrain_prediction))
    print ("F1 score (Train) : %f" % f1_score(Ytrain,dtrain_prediction,average='micro'))
    #Perform cross-validation:
    if useTrainCV:
        cv_score = cross_val_score(alg, Xtrain, Ytrain, cv=cv_fold, scoring='f1_micro')
        print ("CV Score : Mean - %.7g | Std - %.7g | Min - %.7g | Max - %.7g" %
        (np.mean(cv_score), np.std(cv_score), np.min(cv_score), np.max(cv_score)))

    #Pring Feature Importance
    feat_imp = pd.Series(alg.booster().get_fscore()).sort_values(ascending=False)
    feat_imp.plot(kind='bar',title='Feature Importances')
    plt.ylabel('Feature Importance Score')


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Check distribution of classes in each well
Can you write Python code for it?

wells = training_data['Well Name'].unique()

plt.figure(figsize=(15,9))
for index, w in enumerate(wells):
    ax = plt.subplot(2,5,index+1)

    facies_counts = pd.Series(np.zeros(9), index=range(1,10))
    facies_counts = facies_counts.add(training_data[training_data['Well Name']==w]['Facies'].value_counts().sort_index())
    #facies_counts.replace(np.nan,0)
    facies_counts.index = facies_labels
    facies_counts.plot(kind='bar',color=facies_colors,title=w)
    ax.set_ylim(0,160)


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Data Preparation and Model Selection
Now we are ready to test the XGB approach, and will use confusion matrix and f1_score, which were imported, as metric for classification, as well as GridSearchCV, which is an excellent tool for parameter optimization.
Can you write Python code for it?

X_train = training_data.drop(['Facies', 'Well Name','Formation','Depth'], axis = 1 )
Y_train = training_data['Facies' ] - 1
dtrain = xgb.DMatrix(X_train, Y_train)

features = ['GR','ILD_log10','DeltaPHI','PHIND','PE','NM_M','RELPOS']


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
General Approach for Parameter Tuning
We are going to preform the steps as follows:
1.Choose a relatively high learning rate, e.g., 0.1. Usually somewhere between 0.05 and 0.3 should work for different problems.
2.Determine the optimum number of tress for this learning rate.XGBoost has a very usefull function called as "cv" which performs cross-validation at each boosting iteration and thus returns the optimum number of tress required.
3.Tune tree-based parameters(max_depth, min_child_weight, gamma, subsample, colsample_bytree) for decided learning rate and number of trees.
4.Tune regularization parameters(lambda, alpha) for xgboost which can help reduce model complexity and enhance performance.
5.Lower the learning rate and decide the optimal parameters.
Step 1:Fix learning rate and number of estimators for tuning tree-based parameters
In order to decide on boosting parameters, we need to set some initial values of other parameters. Lets take the following values:
1.max_depth = 5
2.min_child_weight = 1
3.gamma = 0
4.subsample, colsample_bytree = 0.8 : This is a commonly used used start value.
5.scale_pos_weight = 1
Please note that all the above are just initial estimates and will be tuned later. Lets take the default learning rate of 0.1 here and check the optimum number of trees using cv function of xgboost. The function defined above will do it for us.
Can you write Python code for it?

xgb1= XGBClassifier(
    learning_rate=0.05,
    objective = 'multi:softmax',
    nthread = 4,
    seed = seed
)

xgb1

modelfit(xgb1, X_train, Y_train)


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Set columns 'Well Name' and 'Formation' to be category
Can you write Python code for it?

training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data.info()

training_data.describe()


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Step 2: Tune max_depth and min_child_weight
Can you write Python code for it?

param_test1={
    'n_estimators':range(20, 100, 10)
}

gs1 = GridSearchCV(xgb1,param_grid=param_test1,
                   scoring='accuracy', n_jobs=4,iid=False, cv=skf)
gs1.fit(X_train, Y_train)
gs1.grid_scores_, gs1.best_params_,gs1.best_score_

gs1.best_estimator_

param_test2={
    'max_depth':range(5,16,2),
    'min_child_weight':range(1,15,2)
}

gs2 = GridSearchCV(gs1.best_estimator_,param_grid=param_test2,
                   scoring='accuracy', n_jobs=4,iid=False, cv=skf)
gs2.fit(X_train, Y_train)
gs2.grid_scores_, gs2.best_params_,gs2.best_score_

gs2.best_estimator_

modelfit(gs2.best_estimator_, X_train, Y_train)


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Step 3: Tune gamma
Can you write Python code for it?

param_test3={
    'gamma':[0,.05,.1,.15,.2,.3,.4],
    'subsample':[0.6,.7,.75,.8,.85,.9],
    'colsample_bytree':[i/10.0 for i in range(4,10)]
}

gs3 = GridSearchCV(gs2.best_estimator_,param_grid=param_test3,
                   scoring='accuracy', n_jobs=4,iid=False, cv=skf)
gs3.fit(X_train, Y_train)
gs3.grid_scores_, gs3.best_params_,gs3.best_score_

gs3.best_estimator_

modelfit(gs3.best_estimator_,X_train,Y_train)


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Step 5: Tuning Regularization Parameters
Can you write Python code for it?

param_test4={
    'reg_alpha':[0, 1e-5, 1e-2, 0.1, 0.2],
    'reg_lambda':[0, .25,.5,.75,.1]
}

gs4 = GridSearchCV(gs3.best_estimator_,param_grid=param_test4,
                   scoring='accuracy', n_jobs=4,iid=False, cv=skf)
gs4.fit(X_train, Y_train)
gs4.grid_scores_, gs4.best_params_,gs4.best_score_

modelfit(gs4.best_estimator_,X_train, Y_train)

gs4.best_estimator_

param_test5={
    'reg_alpha':[.15,0.2,.25,.3,.4],
}

gs5 = GridSearchCV(gs4.best_estimator_,param_grid=param_test5,
                   scoring='accuracy', n_jobs=4,iid=False, cv=skf)
gs5.fit(X_train, Y_train)
gs5.grid_scores_, gs5.best_params_,gs5.best_score_

modelfit(gs5.best_estimator_, X_train, Y_train)

gs5.best_estimator_


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Step 6: Reducing Learning Rate
Can you write Python code for it?

xgb4 = XGBClassifier(
    learning_rate = 0.025,
    n_estimators=120,
    max_depth=7,
    min_child_weight=7,
    gamma = 0.05,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.2,
    reg_lambda =0.75,
    objective='multi:softmax',
    nthread =4,
    seed = seed,
)
modelfit(xgb4,X_train, Y_train)

xgb5 = XGBClassifier(
    learning_rate = 0.00625,
    n_estimators=480,
    max_depth=7,
    min_child_weight=7,
    gamma = 0.05,
    subsample=0.6,
    colsample_bytree=0.8,
    reg_alpha=0.2,
    reg_lambda =0.75,
    objective='multi:softmax',
    nthread =4,
    seed = seed,
)
modelfit(xgb5,X_train, Y_train)


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
Use final model to predict the given test data set
Can you write Python code for it?

# Load test data
test_data = pd.read_csv('validation_data_nofacies.csv')
test_data['Well Name'] = test_data['Well Name'].astype('category')
X_test = test_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
# Predict facies of unclassified data
Y_predicted = model_final.predict(X_test)
test_data['Facies'] = Y_predicted + 1
# Store the prediction
test_data.to_csv('Prediction4.csv')

test_data[test_data['Well Name']=='STUART'].head()

test_data[test_data['Well Name']=='CRAWFORD'].head()

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
...


I am working on the file "HouMath/Face_classification_HouMath_XGB_04.ipynb".
The first task is:
We can see that classes are very imbalanced in each well
Can you write Python code for it?

plt.figure(figsize=(5,5))
sns.heatmap(training_data.corr(), vmax=1.0, square=True)


I am working on the file "HourofCode2015.ipynb".
The first task is:
Floor numbering is the numbering scheme used for a building's floors. There are two major schemes in use across the world. In one system, used in the majority of Europe, the ground floor is the floor on the ground and often has no number or is assigned the number zero. Therefore, the next floor up is assigned the number 1 and is the first floor.
The other system, used primarily in the United States and Canada, counts the bottom floor as number 1 or first floor.
https://en.wikipedia.org/wiki/Storey
Can you write Python code for it?

from functools import partial
# https://docs.python.org/2/library/functools.html
from random import choice, randint

choice('yes no maybe'.split()) # split is a method

for i in range(10):
    print("Call me " + choice('yes no maybe'.split()))

randint(1, 6)

# If you need dice, try this:

roll = partial(randint, 1, 20)

roll()
# how would you make 20 sided dice?

# Create a list of numbers
vals = [3, -8, 2, 7, 6, 2, 5, 12, 4, 9]

#Find the even numbers
evens = []
for v in vals:
    if v%2 == 0:
        evens.append(v)
#How is this working?

evens

squares = []
for v in vals:
    squares.append(v*v)

squares

bigsquares = []
for v in vals:
    s = v*v
    if s > 10:
        bigsquares.append(s)

bigsquares


I am working on the file "HourofCode2015.ipynb".
The first task is:
Some Basic Things
Leveraging a tutorial by David Beazley, Ian Stokes-Rees and Continuum Analytics
http://localhost:8888/notebooks/Dropbox/Python/Harvard%20SEAS%20Tutorial/python-mastery-isr19-master/1-PythonReview.ipynb
and other resources
Can you write Python code for it?

# Comments
# ls list of the files in this folder. See below.
This line will make an error because this line is not python code and this is a code cell.
# Leveraging
#http://localhost:8888/notebooks/Dropbox/Python/Harvard%20SEAS%20Tutorial/python-mastery-isr19-master/1-PythonReview.ipynb

ls # NOT PYTHON! command line

pwd # ALSO NOT PYTHON! Shows what folder you are in.

# math
1+2

4000*3

import math
math.sqrt(2)

2 ** (0.5)

637*532.6

from __future__ import division
1/2

(8+5)*4

# Create a variable
name = 'Perry Grossman'

# Print the variable
name

name[6]


I am working on the file "House Investment Return-SF.ipynb".
The first task is:
Checks out.
Calculating the cost of buying is obviously more involved. Let's start with some basic mortgage definitions:
Can you write Python code for it?

class Mortgage:
    def __init__(self, down, total, payments, rate):
        self.down, self.total, self.payments, self.rate = down, total, payments, rate

    def monthly_payment(self):
        r = self.rate
        P = self.total - self.down
        N = self.payments
        # From https://en.wikipedia.org/wiki/Mortgage_calculator#Monthly_payment_formula
        return (r * P)  / (1 - (1 + r) ** -N)

    def remaining(self, t):
        balance = self.total - self.down
        for _ in xrange(0, t):
            interest = balance * self.rate
            balance -= (self.monthly_payment() - interest)
        return max(0, balance)


I am working on the file "House Investment Return-SF.ipynb".
The first task is:
Let's assume:

Closing costs are 1% on buying, and 6.5% on sale. Costs are dramatically higher on sale because the sale transaction is typically structured so that agent commissions are "paid" by the seller out of the price of the home.
Maintenance is 1% per year.
The home does not appreciate in value whatsoever during the term of the investment (which is arguably pessimistic).
Can you write Python code for it?

def buy(months, r):
    return buy_in_general(mortgage, 0.01, 0.065, 0.01, yearly_taxes, months, 1.0, r)


I am working on the file "House Investment Return-SF.ipynb".
The first task is:
Note that there are cases where there is no IRR. Your investment loses money, even assuming your upfront dollars are valued equally to later hypothetical dollars. It cannot grow your wealth.
There will be many cases where this happens with our house investment.
We can model this "investment" by subtracting the rent we're not paying from the other costs that we now must pay. Now, (drumroll please), let's look at the IRR for our hypothetical house, depending on how long we stay in it:
Can you write Python code for it?

for year in xrange(1, 16):
    def dcf(r_i): return buy(12 * year, r_i / 12) - rent(12 * year, r_i / 12)
    print year, irr(dcf)


I am working on the file "House Investment Return-SF.ipynb".
The first task is:
Now let's do some basic sanity checking:
Can you write Python code for it?

mortgage.monthly_payment()

mortgage.remaining(12 * 30)

mortgage.remaining(0)


I am working on the file "House Investment Return-SF.ipynb".
The first task is:
So, after 10 years, we will pay:
Can you write Python code for it?


I am working on the file "House Investment Return-SF.ipynb".
The first task is:
So, renting becomes more expensive quickly, if we assume our money has no time value. That's not a good assumption, so let's delve a little deeper.
IRR
IRR is a common concept when evaluating investments. The IRR essentially states the smallest time value your money needs to have for you to lose money on your investment. In other words, you want to know the rate that causes your discounted cash flow to become negative.
Can you write Python code for it?

def irr(dcf):
    if dcf(0) < 0: return
    for ix in xrange(1000):
        r_t = float(ix) / 1000
        if dcf(r_t) < 0:
            return r_t


I am working on the file "House Investment Return.ipynb".
The first task is:
All looking good.
Now we can model the cost of buying. We need to pay:

money down for the mortgage
closing costs on purchase, including agent fees, inspection, etc
closing costs on sale
monthly maintenance, monthly mortgage
taxes (there are actually some important tax deductions you can claim against mortgage interest and PMI, but we're going to ignore those for simplicity)
Can you write Python code for it?

def buy_in_general(r, months, mortgage, buy_closing_costs, sale_closing_costs, maintenance, taxes, appreciation):
    cost_to_close = mortgage.down + (mortgage.total * buy_closing_costs)

    mortgage_cost = sum(npv(mortgage.monthly_payment(), r, t) for t in xrange(0, min(months, mortgage.payments)))
    maintenance = sum(npv(maintenance * mortgage.total / 12, r, t) for t in xrange(0, months))

    taxes = sum(npv(taxes, r, year * 12) for year in xrange(months / 12))

    owed_bank = mortgage.remaining(months)
    from_buyer = mortgage.total * appreciation
    cash_at_sale = from_buyer - owed_bank - (from_buyer * sale_closing_costs)

    return -cost_to_close - mortgage_cost - maintenance - taxes + npv(cash_at_sale, r, months)


I am working on the file "House Investment Return.ipynb".
The first task is:
Let's compare that to renting:
Can you write Python code for it?

[rent(0, y * 12) for y in xrange(1, 11)]


I am working on the file "House Investment Return.ipynb".
The first task is:
Let's look at how our equity changes in the first years:
Can you write Python code for it?

for year in (1, 2, 5, 10, 25, 50):
    print year, list(m.total - m.remaining(year * 12) for _, m in mortgages(year * 12, appreciation / 12))[:5]


I am working on the file "House Investment Return.ipynb".
The first task is:
MATH TIME
First, the fundamental law of finance. Net present value of any sum of money:
Can you write Python code for it?

def npv(future_value, discount_rate, t):
    return future_value / (1 + discount_rate) ** t


I am working on the file "House Investment Return.ipynb".
The first task is:
Not bad. If we're willing to put up with some pretty high catch-up payments at the end of our 30-year term, we easily beat the stock market. Even if we switch houses every five years.
If we're planning on staying put for a very long time, a house looks like it could be a fantastic investment.
Another thing to keep in mind: with this time horizon, the final sale price of the house itself is practically inconsequential. Here's the present value of the cash we'll get on sale, discounted at 10% per year:
Can you write Python code for it?


I am working on the file "House Investment Return.ipynb".
The first task is:
Not bad. We're well above the paltry 1.3% current risk free rate in year three. By year four, our return rises above the 10% average annual returns for the S&P 500.
There's obviously a couple caveats. First, all of this analysis is predicated on the house being a principle residence. There are many benefits, especially regarding capital gains, that flow from the government in this case. Investment properties are a whole 'nother ball of wax.
Finally, let's keep in mind that this investment scenario has one big caveat: we disappear into thin air after we sell the house, and never spend money on housing again. We'll return to that assumption later.
Bring on the Bears
Of course, market conditions can change drastically and quickly. What does our investment look like in a super-pessimistic case, where our house loses half its value?
Can you write Python code for it?

def buy_ohno(r, months):
    return buy(r, months, appreciation=0.5)

for year in xrange(1, 21):
    def dcf(r_i): return buy_ohno(r_i / 12, 12 * year) - rent(r_i / 12, 12 * year)
    print year, irr(dcf)


I am working on the file "House Investment Return.ipynb".
The first task is:
Note that there are cases where there is no IRR. Your investment loses money, even assuming your upfront dollars are valued equally to later hypothetical dollars. It cannot grow your wealth.
There will be cases where this happens with our house investment.
We can model this "investment" by subtracting the rent we're not paying from the other costs that we now must pay. Now, (drumroll please), let's look at the IRR for our hypothetical house, depending on how long we stay in it:
Can you write Python code for it?

for year in xrange(1, 16):
    def dcf(r_i): return buy(r_i / 12, 12 * year) - rent(r_i / 12, 12 * year)
    print year, irr(dcf)


I am working on the file "House Investment Return.ipynb".
The first task is:
Now let's do some basic sanity checking:
Can you write Python code for it?


I am working on the file "House Investment Return.ipynb".
The first task is:
So, after 10 years, we will pay:
Can you write Python code for it?


I am working on the file "House Investment Return.ipynb".
The first task is:
So, renting becomes more expensive quickly, if we assume our money has no time value. That's not a good assumption, so let's delve a little deeper.
IRR
IRR is a common concept when evaluating investments. The IRR essentially states the smallest time value your money needs to have for you to lose money on your investment. In other words, you want to know the rate that causes your discounted cash flow to become negative.
Can you write Python code for it?


I am working on the file "House Investment Return.ipynb".
The first task is:
Sounds about right.
Calculating the cost of buying is obviously more involved. Let's start with some basic mortgage definitions:
Can you write Python code for it?


I am working on the file "House Investment Return.ipynb".
The first task is:
This is a tiny fraction of the total NPV of our housing costs:
Can you write Python code for it?


I am working on the file "House Investment Return.ipynb".
The first task is:
What? Our rate of return gets worse the longer we wait to sell?
Because the house appreciates less quickly than our return rate, it makes sense to sell it ASAP. We're forgetting that we actually have to live somewhere if we don't own a house.  Let's do a more realistic comparison: whenever we sell our house, we'll buy another similar house somewhere else. So we'll be taking out a series of mortgages with the following guidelines:

We want to retire someday, so we structure our payments to zero the mortgage after thirty years no matter what.
We plow all of our equity from before into the new mortgage, minus closing costs.
If we don't have enough equity to cover closing costs, we assume we can pony up the difference.
Can you write Python code for it?

total_months = 50 * 12
mortgage_months = 30 * 12
appreciation = 0.02

def mortgages(swap_months, appreciation):
    prior_mortgage = None
    swaps = [m for m in xrange(swap_months, total_months, swap_months)]

    for month in [0] + swaps:
        new_term = mortgage_months - month
        if new_term > 0:
            new_price = price * (1 + appreciation) ** month
            new_down = new_price * 0.20
            if not prior_mortgage:
                equity = down
            else:
                equity = new_price - prior_mortgage.remaining(swap_months) - 0.075 * new_price

            prior_mortgage = Mortgage(max(equity, new_down), new_price, new_term, apr / 12)
            yield month, prior_mortgage


I am working on the file "House Investment Return.ipynb".
The first task is:
Wow, those one and two year scenarios look pretty grisly. We have to really pay to catch our equity back up. Even the final payments for five-year switching don't look too great, but they're still manageable.
The math is basically reinforcing common sense. It makes very little finanacial sense to constantly house swap. Let's look at the rate of return for the sensible swapping intervals:
Can you write Python code for it?

for year in (5, 10, 25, 50):
    swap_months = year * 12

    def buy_and_appreciate(r, term, mortgage):
        return buy(r, term, mortgage=mortgage, appreciation=1.0)

    def dcf_swapping_houses(r):
        mortgage_list = list(mortgages(swap_months, appreciation / 12))
        buy_costs = [npv(buy_and_appreciate(r / 12, swap_months, m), r / 12, month)
                     for month, m in mortgage_list[:-1]]

        final_month, final_mortgage = mortgage_list[-1]
        mortgage_remaining = 30 * 12 - swap_months
        months_to_final = final_month
        months_remaining = total_months - months_to_final
        final_cost = npv(buy_and_appreciate(r / 12, months_remaining, final_mortgage), r / 12, months_to_final)

        return sum(buy_costs) + final_cost - rent(r / 12, total_months)

    print year, irr(dcf_swapping_houses)


I am working on the file "House Investment Return.ipynb".
The first task is:
Yikes, we're barely building any equity at all when we only switch every 1-2 years. We're probably ponying up a ton of closing costs too. Now let's look at what our final monthly payments look like:
Can you write Python code for it?

for year in (1, 2, 5, 10, 25, 50):
    print year, list(m.monthly_payment() for _, m in mortgages(year * 12, appreciation / 12))[-5:]


I am working on the file "House Investment Return.ipynb".
The first task is:
Zillow has also helpfully calculated the 30-year fixed mortgage terms:
Can you write Python code for it?


I am working on the file "House price prediction with Pandas.ipynb".
The first task is:
Basic commands for data analysis
Can you write Python code for it?

len(test_data)

len(train_data)

sales.describe()

sales['bedrooms'].unique()

sales.count()

sales['renovated'] = sales['yr_renovated'] > 0

sales.head()

sales.plot.scatter(x='sqft_living', y='price')

sales.index

sales.columns

sales.values

sales.index

bed_count = sales.bedrooms.value_counts()

type(bed_count)

bed_count

bed_count.sort_index().plot()

sales.ix[10]

regr_one_feature = linear_model.LinearRegression()

training_data_features = train_data["sqft_living"].values

training_data_features.shape

training_data_features = np.array([train_data.sqft_living]).T

training_data_features.shape

training_data_targets = train_data["price"].values

training_data_targets.shape

training_data_targets = np.array([train_data.price]).T

training_data_targets.shape

regr_one_feature.fit(training_data_features, training_data_targets)

regr_one_feature.coef_

# Make predictions using the model and the data we set aside
test_data_features = np.array([test_data.sqft_living]).T
test_data_targets = np.array([test_data.price]).T

# Let's look at the variance (1 is a perfect prediction)
regr_one_feature.score(test_data_features, test_data_targets)

import math
np.mean((regr_one_feature.predict(test_data_features) - test_data_targets) ** 2)

plt.scatter(test_data_features, test_data_targets, color='blue')
plt.plot(test_data_features, regr_one_feature.predict(test_data_features), color='red', linewidth=3)


I am working on the file "House price prediction with Pandas.ipynb".
The first task is:
Metrics
Can you write Python code for it?

from sklearn.metrics import  mean_squared_error # Same as the computation above the plot
mean_squared_error(test_data_targets, regr_one_feature.predict(test_data_features))

regr_one_feature.coef_

regr_one_feature.intercept_