I am working on the file ".ipynb_checkpoints/MNIST_advanced-checkpoint.ipynb".
The first task is:
Test the Sparse Autoencoder with a simple Neural Net (not for presentation)
Can you write Python code for it?

sparse_ae_test_model = Sequential()
# single_ae_test_model.add(Dense(16, activation='relu', input_shape=(36,)))
# single_ae_test_model.add(Dropout(0.2))
sparse_ae_test_model.add(Dense(num_classes, activation='softmax', input_shape=(encoding_dim,)))

sparse_ae_test_model.compile(loss='categorical_crossentropy',
                             optimizer=RMSprop(),
                             metrics=['accuracy'])

# sparse_ae_test_model = keras.models.load_model('models/sparse_ae_test_model.h5')
sparse_ae_test_model.fit(sparse_encoder.predict(x_train), y_train,
                         batch_size=batch_size,
                         epochs=40,
                         verbose=1,
                         validation_data=(sparse_encoder.predict(x_val), y_val))

sparse_ae_test_model.save('models/sparse_ae_test_model.h5')

score = sparse_ae_test_model.evaluate(sparse_encoder.predict(x_val), y_val, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file ".ipynb_checkpoints/MNIST_advanced-checkpoint.ipynb".
The first task is:
The difference in accuracy is huge. The autoencoder architecture is highly recommendable for denoising signals.
Train with sparsity constraint on hidden layer (not for presentation)
So what does all this mean for sparsity? How does the rectifier activation function allow for sparsity in the hidden units? If the hidden units are exposed to a range of input values it makes sense that the rectifier activation function should lead to more ‘real zeros’ as we sweep across possible inputs. In other words, less neurons in our network would activate because of the limitations imposed by the rectifier activation function. https://www.quora.com/What-does-it-mean-that-activation-functions-like-ReLUs-in-NNs-induce-sparsity-in-the-hidden-units
Can you write Python code for it?

input_img = Input(shape=(input_dim,))
# add a Dense layer with a L1 activity regularizer
encoded = Dense(encoding_dim, activation='relu',
                activity_regularizer=regularizers.l1(10e-5))(input_img)
decoded = Dense(input_dim, activation='sigmoid')(encoded)

sparse_autoencoder = Model(input_img, decoded)

sparse_encoder = Model(input_img, encoded)

encoded_input = Input(shape=(encoding_dim,))
# retrieve the last layer of the autoencoder model
decoder_layer = sparse_autoencoder.layers[-1]
# create the decoder model
sparse_decoder = Model(encoded_input, decoder_layer(encoded_input))


I am working on the file ".ipynb_checkpoints/MNIST_advanced-checkpoint.ipynb".
The first task is:
The result is slightly better than for single autoencoder.
Can you write Python code for it?

# plot_model(stacked_autoencoder, to_file='images/stacked_autoencoderTD.png', show_shapes=True, show_layer_names=True, rankdir='LR')

Image("images/stacked_autoencoderTD.png")

decoded_imgs = stacked_autoencoder.predict(x_val)

n = 10  # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(x_val[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file ".ipynb_checkpoints/MNIST_advanced-checkpoint.ipynb".
The first task is:
Train Neural Net to recognize MNIST digits
Can you write Python code for it?

# constants
hidden1_dim = 512
hidden2_dim = 512


I am working on the file ".ipynb_checkpoints/MNIST_advanced-checkpoint.ipynb".
The first task is:
Train/load model
Can you write Python code for it?

# single_autoencoder = keras.models.load_model('models/single_autoencoder.h5')
single_autoencoder.fit(x_train, x_train,
                       epochs=50,
                       batch_size=batch_size,
                       shuffle=False,
                       verbose=0,
                       validation_split=0.1,
                       callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])

# single_autoencoder.save('models/single_autoencoder.h5')

score = single_autoencoder.evaluate(x_val, x_val, verbose=0)
print(score)

# plot_model(single_autoencoder, to_file='images/single_autoencoder.png', show_shapes=True, show_layer_names=True, rankdir='LR')

Image("images/single_autoencoder.png")


I am working on the file ".ipynb_checkpoints/Magic_notebook-checkpoint.ipynb".
The first task is:
Other display utilities
Can you write Python code for it?

from IPython.display import YouTubeVideo, display
YouTubeVideo("yH-7XWLm_Po")
# vid = YouTubeVideo("yH-7XWLm_Po")
# display(vid)

# diplay in interactive way embedded in jupyer notebook
%matplotlib notebook

import pandas as pd
df = pd.DataFrame({
        'Letter': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
        'X': [4, 3, 5, 2, 1, 7, 7, 5, 9],
        'Y': [0, 4, 3, 6, 7, 10, 11, 9, 13],
        'Z': [1, 2, 3, 1, 2, 3, 1, 2, 3]
    })

df.plot()


I am working on the file ".ipynb_checkpoints/Magic_notebook-checkpoint.ipynb".
The first task is:
Scala on jupyter notebook

list of jupyter kernel: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
get scala: https://www.scala-lang.org/download/
get spark: http://spark.apache.org/downloads.html
Can you write Python code for it?

# list of enabled kernel in your jupyter
!ipython kernelspec list

!pip install toree

# you need the env variable SPARK_HOME in your .bashrc
!jupyter toree install --interpreters=Scala,PySpark,SparkR,SQL --user --spark_home=$SPARK_HOME
# you can now create a new jupyter notebook with apache torere and use scala there


I am working on the file ".ipynb_checkpoints/Magic_notebook-checkpoint.ipynb".
The first task is:
To use an nbextension, you'll also need to enable it, which tells the notebook interface to load it. To do this, you can use a Jupyter subcommand:
- jupyter nbextension enable <nbextension require path>
for example,
- jupyter nbextension enable codefolding/main
To disable the extension again, use
- jupyter nbextension disable <nbextension require path>
Alternatively, and more conveniently, you can use the jupyter_nbextensions_configurator server extension, which is installed as a dependency of this repo, and can be used to enable and disable the individual nbextensions, as well as configure their options. You can then open the nbextensions tab on the tree (dashboard/file browser) notebook page to configure nbextensions. You will have access there to a dashboard where extensions can be enabled/disabled via checkboxes. Additionally a short documentation for each extension is displayed, and configuration options are presented.
Can you write Python code for it?

## run the line below, restart jupyter (close and restart) and go to localhost:8888 or edit -> nbextensions
## from the nbextension you can enable/disable some notebook extensions:
# NICE ONE ARE: code prettify, spell-checker
# check the nbextensions utility to understant what the extention does
# ATTENTION: probably when you activate a new extension you need to refresh ore reload the jupyter notebook
# ATTENTION: (so activate the extension before you start everything)
!jupyter contrib nbextension install --user
!jupyter nbextensions_configurator enable --user


I am working on the file ".ipynb_checkpoints/Magic_notebook-checkpoint.ipynb".
The first task is:
change jupyter-theme

https://github.com/dunovank/jupyter-themes
Can you write Python code for it?

!pip install --upgrade jupyterthemes

!jt -l  # list all theme you can access

!jt -t oceans16 # set a theme

!jt -t grade3

!jt -r # restore previous theme


I am working on the file ".ipynb_checkpoints/Magic_notebook-checkpoint.ipynb".
The first task is:
latex formula
$$ P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)} $$
Can you write Python code for it?

%%bash
## use %%bash, %%python3, %%python2, %%perl, %%HTML to run code of that languages (%R requires the cells below)
for i in {1..5}
do
   echo "i is $i"
done

%%python2
print "ciao"

# get R work on your environment (type on terminal the line below)
!conda install -c r r-essentials

# to make R run in the same jupyter notebook of python
!conda install rpy2

%load_ext rpy2.ipython

%R require(ggplot2)

import pandas as pd
df = pd.DataFrame({
        'Letter': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
        'X': [4, 3, 5, 2, 1, 7, 7, 5, 9],
        'Y': [0, 4, 3, 6, 7, 10, 11, 9, 13],
        'Z': [1, 2, 3, 1, 2, 3, 1, 2, 3]
    })

%%R -i df
ggplot(data = df) + geom_point(aes(x = X, y= Y, color = Letter, size = Z))

# jupyter extensions: https://github.com/ipython-contrib/jupyter_contrib_nbextensions
# example are jupyter spell-checker and code-formatter
!pip install https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tarball/master
!pip install jupyter_nbextensions_configurator


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-checkpoint.ipynb".
The first task is:
Let us define the Poisson matrix also as a Linear Operator
Can you write Python code for it?

def Laplace(nx,ny):
  '''
  Action of the Laplace matrix on a vector v
  '''
  def mv(v):
    u =np.zeros([nx+2,ny+2])
    ut=np.zeros([nx,ny])
    u[1:nx+1,1:ny+1]=v.reshape([nx,ny])

    dx=1.0/nx; dy=1.0/ny

    Ax=1.0/dx**2;Ay=1.0/dy**2

    #BCs. Homogenous Dirichlet
    u[ 0,:] = -u[ 1,:]
    u[-1,:] = -u[-2,:]
    u[:, 0] = -u[:, 1]
    u[:,-1] = -u[:,-2]

    for i in range(1,nx+1):
      for j in range(1,ny+1):
        ut[i-1,j-1]=(Ax*(u[i+1,j]+u[i-1,j])+Ay*(u[i,j+1]+u[i,j-1]) - 2.0*(Ax+Ay)*u[i,j])
    return ut.reshape(v.shape)
  return mv


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-checkpoint.ipynb".
The first task is:
Lets call the FMG solver for the same problem
Can you write Python code for it?

print('mgd2d.py FMG solver:')
print('NX:',NX,', NY:',NY,', levels: ',nlevels)

u,res=FMG(NX,NY,nlevels,f,nv=1)
rtol=np.max(np.max(np.abs(res)))

print(' FMG L_inf(res.)= ',rtol)
error=uann[1:NX+1,1:NY+1]-u[1:NX+1,1:NY+1]
print('L_inf (true error): ',np.max(np.max(np.abs(error))))


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-checkpoint.ipynb".
The first task is:
Lets look at what happens with and without the preconditioner.
Can you write Python code for it?

A = LinearOperator((NX*NY,NX*NY), matvec=Laplace(NX,NY))
#Exact solution and RHS
uex=np.random.rand(NX*NY,1)
b=A*uex

#Multigrid Preconditioner
M=MGVP(NX,NY,nlevels)

u,info,iters=solve_sparse(bicgstab,A,b,tol=1e-10,maxiter=500)
print('without preconditioning. status:',info,', Iters: ',iters)
error=uex.reshape([NX,NY])-u.reshape([NX,NY])
print('error :',np.max(np.max(np.abs(error))))

u,info,iters=solve_sparse(bicgstab,A,b,tol=1e-10,maxiter=500,M=M)
print('With preconditioning. status:',info,', Iters: ',iters)

error=uex.reshape([NX,NY])-u.reshape([NX,NY])
print('error :',np.max(np.max(np.abs(error))))


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-checkpoint.ipynb".
The first task is:
Now we can call the solver
Can you write Python code for it?

print('mgd2d.py solver:')
print('NX:',NX,', NY:',NY,', tol:',tol,'levels: ',nlevels)
for it in range(1,max_cycles+1):
  u,res=V_cycle(NX,NY,nlevels,u,f)
  rtol=np.max(np.max(np.abs(res)))
  if(rtol<tol):
    break
  error=uann[1:NX+1,1:NY+1]-u[1:NX+1,1:NY+1]
  print('  cycle: ',it,', L_inf(res.)= ',rtol,',L_inf(true error): ',np.max(np.max(np.abs(error))))

error=uann[1:NX+1,1:NY+1]-u[1:NX+1,1:NY+1]
print('L_inf (true error): ',np.max(np.max(np.abs(error))))


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-checkpoint.ipynb".
The first task is:
The nested function is required because "matvec" in LinearOperator takes only one argument-- the vector. But we require the grid details and boundary condition information to create the Poisson matrix. Now will use these to solve a problem. Unlike earlier where we used an analytical solution and RHS, we will start with a random vector which will be our exact solution, and multiply it with the Poisson matrix to get the Rhs vector for the problem. There is no analytical equation associated with the matrix equation.
The scipy sparse solve routines do not return the number of iterations performed. We can use this wrapper to get the number of iterations
Can you write Python code for it?

def solve_sparse(solver,A, b,tol=1e-10,maxiter=500,M=None):
      num_iters = 0
      def callback(xk):
         nonlocal num_iters
         num_iters+=1
      x,status=solver(A, b,tol=tol,maxiter=maxiter,callback=callback,M=M)
      return x,status,num_iters


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-checkpoint.ipynb".
The first task is:
True error is the difference of the approximation with the analytical solution. It is largely the discretization error. This what would be present when we solve the discrete equation with a direct/exact method like gaussian elimination. We see that true error stops reducing at the 4th cycle. The approximation is not getting any better after this point. So we can stop after 4 cycles. But, in general we dont know the true error. In practice we use the norm of the (relative) residual as a stopping criterion. As the cycles progress the floating point round-off error limit is reached and the residual also stops decreasing.
This was the multigrid V cycle. We can use this as preconditioner to a Krylov solver. But before we get to that let's complete the multigrid introduction by looking at the Full Multi-Grid algorithm. You can skip this section safely.
Full Multi-Grid
We started with a zero initial guess for the V-cycle. Presumably, if we had a better initial guess we would get better results.  So we solve a coarse problem exactly and interpolate it onto the fine grid and use that as the initial guess for the V-cycle. The result of doing this recursively is the Full Multi-Grid(FMG) Algorithm. Unlike the V-cycle which was an iterative procedure, FMG is a direct solver. There is no successive improvement of the approximation. It straight away gives us an approximation that is within the discretization error. The FMG algorithm is given below.
Can you write Python code for it?

def FMG(nx,ny,num_levels,f,nv=1,level=1):

  if(level==num_levels):#bottom solve
    u=np.zeros([nx+2,ny+2])
    u,res=GSrelax(nx,ny,u,f,iters=50)
    return u,res

  #Step 1: Restrict the rhs to a coarse grid
  f_c=restrict(nx//2,ny//2,f)

  #Step 2: Solve the coarse grid problem using FMG
  u_c,_=FMG(nx//2,ny//2,num_levels,f_c,nv,level+1)

  #Step 3: Interpolate u_c to the fine grid
  u=prolong(nx//2,ny//2,u_c)

  #step 4: Execute 'nv' V-cycles
  for _ in range(nv):
    u,res=V_cycle(nx,ny,num_levels-level,u,f)
  return u,res


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
1.2 Interpolation Operator
This operator takes values on a coarse grid and transfers them onto a fine grid. It is also called prolongation. The function below uses bilinear interpolation for this purpose. 'v' is on a coarse grid and we want to interpolate it on a fine grid and store it in v_f.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
1.3 Restriction
This is exactly the opposite of the interpolation. It takes values from the find grid and transfers them onto the coarse grid. It is kind of an averaging process. This is fundamentally different from interpolation. Each coarse grid point is surrounded by four fine grid points. So quite simply we take the value of the coarse point to be the average of 4 fine points. Here 'v' is the fine grid quantity and 'v_c' is the coarse grid quantity
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
Let us define the Poisson matrix also as a LinearOperator
Can you write Python code for it?

def Laplace(nx,ny):
  '''
  Action of the Laplace matrix on a vector v
  '''
  def mv(v):
    u =np.zeros([nx+2,ny+2])

    u[1:nx+1,1:ny+1]=v.reshape([nx,ny])
    dx=1.0/nx; dy=1.0/ny
    Ax=1.0/dx**2; Ay=1.0/dy**2

    #BCs. Needs to be generalized!
    u[ 0,:] = -u[ 1,:]
    u[-1,:] = -u[-2,:]
    u[:, 0] = -u[:, 1]
    u[:,-1] = -u[:,-2]

    ut = (Ax*(u[2:nx+2,1:ny+1]+u[0:nx,1:ny+1])
        + Ay*(u[1:nx+1,2:ny+2]+u[1:nx+1,0:ny])
        - 2.0*(Ax+Ay)*u[1:nx+1,1:ny+1])
    return ut.reshape(v.shape)
  A = LinearOperator((nx*ny,nx*ny), matvec=mv)
  return A


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
Let us set up the problem, discretization and solver details. The number of divisions along each dimension is given as a power of two function of the number of levels. In principle this is not required, but having it makes the inter-grid transfers easy.
The coarsest problem is going to have a 2-by-2 grid.
Can you write Python code for it?

#input
max_cycles = 30
nlevels    = 6
NX         = 2*2**(nlevels-1)
NY         = 2*2**(nlevels-1)
tol        = 1e-15

#the grid has one layer of ghost cellss
uann=np.zeros([NX+2,NY+2])#analytical solution
u   =np.zeros([NX+2,NY+2])#approximation
f   =np.zeros([NX+2,NY+2])#RHS

#calcualte the RHS and exact solution
DX=1.0/NX
DY=1.0/NY

xc=np.linspace(0.5*DX,1-0.5*DX,NX)
yc=np.linspace(0.5*DY,1-0.5*DY,NY)
XX,YY=np.meshgrid(xc,yc,indexing='ij')

uann[1:NX+1,1:NY+1]=Uann(XX,YY)
f[1:NX+1,1:NY+1]   =source(XX,YY)


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
Lets call the FMG solver for the same problem
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
Lets look at what happens with and without the preconditioner.
Can you write Python code for it?

A = Laplace(NX,NY)
#Exact solution and RHS
uex=np.random.rand(NX*NY,1)
b=A*uex

#Multigrid Preconditioner
M=MGVP(NX,NY,nlevels)

u,info,iters=solve_sparse(bicgstab,A,b,tol=1e-10,maxiter=500)
print('Without preconditioning. status:',info,', Iters: ',iters)
error=uex-u
print('error :',np.max(np.abs(error)))

u,info,iters=solve_sparse(bicgstab,A,b,tol=1e-10,maxiter=500,M=M)
print('With preconditioning. status:',info,', Iters: ',iters)
error=uex-u
print('error :',np.max(np.abs(error)))


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
Thats it! Now we can see it in action. We can use a problem with a known solution to test our code. The following functions set up a rhs for a problem with homogenous dirichlet BC on the unit square.
Can you write Python code for it?

#analytical solution
def Uann(x,y):
   return (x**3-x)*(y**3-y)
#RHS corresponding to above
def source(x,y):
  return 6*x*y*(x**2+ y**2 - 2)


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
The nested function is required because "matvec" in LinearOperator takes only one argument-- the vector. But we require the grid details and boundary condition information to create the Poisson matrix. Now will use these to solve a problem. Unlike earlier where we used an analytical solution and RHS, we will start with a random vector which will be our exact solution, and multiply it with the Poisson matrix to get the Rhs vector for the problem. There is no analytical equation associated with the matrix equation.
The scipy sparse solve routines do not return the number of iterations performed. We can use this wrapper to get the number of iterations
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
True error is the difference of the approximation with the analytical solution. It is largely the discretization error. This what would be present when we solve the discrete equation with a direct/exact method like gaussian elimination. We see that true error stops reducing at the 5th cycle. The approximation is not getting any better after this point. So we can stop after 5 cycles. But, in general we dont know the true error. In practice we use the norm of the (relative) residual as a stopping criterion. As the cycles progress the floating point round-off error limit is reached and the residual also stops decreasing.
This was the multigrid V cycle. We can use this as preconditioner to a Krylov solver. But before we get to that let's complete the multigrid introduction by looking at the Full Multi-Grid algorithm. You can skip this section safely.
1.6 Full Multi-Grid
We started with a zero initial guess for the V-cycle. Presumably, if we had a better initial guess we would get better results.  So we solve a coarse problem exactly and interpolate it onto the fine grid and use that as the initial guess for the V-cycle. The result of doing this recursively is the Full Multi-Grid(FMG) Algorithm. Unlike the V-cycle which was an iterative procedure, FMG is a direct solver. There is no successive improvement of the approximation. It straight away gives us an approximation that is within the discretization error. The FMG algorithm is given below.
Can you write Python code for it?

def FMG(nx,ny,num_levels,f,nv=1,level=1):

  if(level==num_levels):#bottom solve
    u=np.zeros([nx+2,ny+2])
    u,res=Jacrelax(nx,ny,u,f,iters=50)
    return u,res

  #Step 1: Restrict the rhs to a coarse grid
  f_c=restrict(nx//2,ny//2,f)

  #Step 2: Solve the coarse grid problem using FMG
  u_c,_=FMG(nx//2,ny//2,num_levels,f_c,nv,level+1)

  #Step 3: Interpolate u_c to the fine grid
  u=prolong(nx//2,ny//2,u_c)

  #step 4: Execute 'nv' V-cycles
  for _ in range(nv):
    u,res=V_cycle(nx,ny,num_levels-level,u,f)
  return u,res


I am working on the file ".ipynb_checkpoints/Making_a_Preconditioner-vectorized-checkpoint.ipynb".
The first task is:
Without the preconditioner ~150 iterations were needed, where as with the V-cycle preconditioner the solution was obtained in far fewer iterations. Let's try with CG:
Can you write Python code for it?

u,info,iters=solve_sparse(cg,A,b,tol=1e-10,maxiter=500)
print('Without preconditioning. status:',info,', Iters: ',iters)
error=uex-u
print('error :',np.max(np.abs(error)))

u,info,iters=solve_sparse(cg,A,b,tol=1e-10,maxiter=500,M=M)
print('With preconditioning. status:',info,', Iters: ',iters)
error=uex-u
print('error :',np.max(np.abs(error)))


I am working on the file ".ipynb_checkpoints/Masimo160127-Copy1-checkpoint.ipynb".
The first task is:
Average Every 10 Sec During ROP Exam for first 4 minutes
Can you write Python code for it?

#AVERAGE DURING ROP EXAM FOR FIRST FOUR MINUTES
def perdelta1(start, end, delta):
    r1 = []
    curr = start
    while curr < end:
        r1.append(curr)
        curr += delta
    return r1

df1PI = df_clean.PI[X:X+timedelta(minutes=4)]
df1O2 = df.SpO2[X:X+timedelta(minutes=4)]
df1PR = df.PR[X:X+timedelta(minutes=4)]
win1 = timedelta(seconds=10) #any unit of time & make the range

r1 = perdelta1(X, X+timedelta(minutes=4), win1)

#make the series to store
avg1PI = Series(index = r1, name = 'PI DurEx')
avg1O2 = Series(index = r1, name = 'SpO2 DurEx')
avg1PR = Series(index = r1, name = 'PR DurEX')
#average!
for i1 in r1:
    avg1PI[i1] = df1PI[i1:(i1+win1)].mean()
    avg1O2[i1] = df1O2[i1:(i1+win1)].mean()
    avg1PR[i1] = df1PR[i1:(i1+win1)].mean()

result1 = concat([avg1PI, avg1O2, avg1PR], axis=1, join='inner')
print result1


I am working on the file ".ipynb_checkpoints/Masimo160127-Copy1-checkpoint.ipynb".
The first task is:
Average Every 30 Mins Hour 3-4 After ROP Exam
Can you write Python code for it?

#AVERAGE EVERY 30 MINUTES THREE HOURS AFTER ROP EXAM

def perdelta4(start, end, delta):
    r4 = []
    curr = start
    while curr < end:
        r4.append(curr)
        curr += delta
    return r4

# datetime(year, month, day, hour, etc.)

df4PI = df_clean.PI[(Z+timedelta(hours=2)):(Z+timedelta(hours=3))]
df4O2 = df.SpO2[(Z+timedelta(hours=2)):(Z+timedelta(hours=3))]
df4PR = df.PR[(Z+timedelta(hours=2)):(Z+timedelta(hours=3))]
win4 = timedelta(minutes=30) #any unit of time, make the range

r4 = perdelta4((Z+timedelta(hours=2)), (Z+timedelta(hours=3)), win4)

#make the series to store
avg4PI = Series(index = r4, name = 'PI q30MinHr3')
avg4O2 = Series(index = r4, name = 'O2 q30MinHr3')
avg4PR = Series(index = r4, name = 'PR q30MinHr3')

#average!
for i4 in r4:
    avg4PI[i4] = df4PI[i4:(i4+win4)].mean()
    avg4O2[i4] = df4O2[i4:(i4+win4)].mean()
    avg4PR[i4] = df4PR[i4:(i4+win4)].mean()

result4 = concat([avg4PI, avg4O2, avg4PR], axis=1, join='inner')
print result4


I am working on the file ".ipynb_checkpoints/Masimo160127-Copy1-checkpoint.ipynb".
The first task is:
Average q 5 Min for 1 hour after 1st Eye Drops
Can you write Python code for it?

# Every 5 min Average from start of eye drops to start of exam

def perdeltadrop(start, end, delta):
    rdrop = []
    curr = start
    while curr < end:
        rdrop.append(curr)
        curr += delta
    return rdrop

dfdropPI = df_clean.PI[W:W+timedelta(hours=1)]
dfdropO2 = df.SpO2[W:W+timedelta(hours=1)]
dfdropPR = df.PR[W:W+timedelta(hours=1)]
windrop = timedelta(minutes=5)#make the range
rdrop = perdeltadrop(W, W+timedelta(hours=1), windrop)

avgdropPI = Series(index = rdrop, name = 'PI DurEyeD')
avgdropO2 = Series(index = rdrop, name = 'SpO2 DurEyeD')
avgdropPR = Series(index = rdrop, name = 'PR DurEyeD')

for i in rdrop:
    avgdropPI[i] = dfdropPI[i:(i+windrop)].mean()
    avgdropO2[i] = dfdropO2[i:(i+windrop)].mean()
    avgdropPR[i] = dfdropPR[i:(i+windrop)].mean()

resultdrops = concat([avgdropPI, avgdropO2, avgdropPR], axis=1, join='inner')
print resultdrops


I am working on the file ".ipynb_checkpoints/Masimo160127-Copy1-checkpoint.ipynb".
The first task is:
Baseline Averages
Can you write Python code for it?

avg0PI = df_clean.PI[Y:W].mean()
avg0O2 = df.SpO2[Y:W].mean()
avg0PR = df.PR[Y:W].mean()

print 'Baseline Averages\n', 'PI :\t',avg0PI, '\nSpO2 :\t',avg0O2,'\nPR :\t',avg0PR,
#df.std() for standard deviation


I am working on the file ".ipynb_checkpoints/Masimo160127-Copy1-checkpoint.ipynb".
The first task is:
Export to CSV
Can you write Python code for it?

import csv
class excel_tab(csv.excel):
        delimiter = '\t'
csv.register_dialect("excel_tab", excel_tab)

with open('ROP006_PO.csv', 'w') as f:    #CHANGE CSV FILE NAME
    writer = csv.writer(f, dialect=excel_tab)
    writer.writerow(['PI'])
    writer.writerow([avg0PI])
    for i in rdrop:
        writer.writerow([avgdropPI[i]]) #NEEDS BRACKETS TO MAKE IT SEQUENCE
    for i in r1:
        writer.writerow([avg1PI[i]])
    for i in r2:
        writer.writerow([avg2PI[i]])
    for i in r3:
        writer.writerow([avg3PI[i]])
    for i in r4:
        writer.writerow([avg4PI[i]])
    for i in r5:
        writer.writerow([avg5PI[i]])
    writer.writerow(['O2'])
    writer.writerow([avg0O2])
    for i in rdrop:
        writer.writerow([avgdropO2[i]])
    for i in r1:
        writer.writerow([avg1O2[i]])
    for i in r2:
        writer.writerow([avg2O2[i]])
    for i in r3:
        writer.writerow([avg3O2[i]])
    for i in r4:
        writer.writerow([avg4O2[i]])
    for i in r5:
        writer.writerow([avg5O2[i]])
    writer.writerow(['PR'])
    writer.writerow([avg0PR])
    for i in rdrop:
        writer.writerow([avgdropPR[i]])
    for i in r1:
        writer.writerow([avg1PR[i]])
    for i in r2:
        writer.writerow([avg2PR[i]])
    for i in r3:
        writer.writerow([avg3PR[i]])
    for i in r4:
        writer.writerow([avg4PR[i]])
    for i in r5:
        writer.writerow([avg5PR[i]])
    writer.writerow(['Data Recording Time Length'])
    writer.writerows(RTL)
    writer.writerow(['Pre Desat Counts for X Minutes'])
    writer.writerows(PreDesats)
    writer.writerow(['Post Dest Counts for X Minutes'])
    writer.writerows(PostDesats)


I am working on the file ".ipynb_checkpoints/Masimo160127-Copy1-checkpoint.ipynb".
The first task is:
Set Date and Time of ROP Exam and Eye Drops
Can you write Python code for it?

df_first = df.first_valid_index() #get the first number from index

Y = pd.to_datetime(df_first) #convert index to datetime
# Y = TIME DATA COLLECTION BEGAN / First data point on CSV

# SYNTAX:
# datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])

W = datetime(2015, 6, 17, 7, 10)+TC
# W = first eye drop dtarts
X = datetime(2015, 6, 17, 8, 36)+TC
# X = ROP Exam Started

Z = datetime(2015, 6, 17, 8, 39)+TC
# Z = ROP Exam Ended

df_last = df.last_valid_index() #get the last number from index

Q = pd.to_datetime(df_last)

# Q = TIME DATA COLLECTION ENDED / Last Data point on CSV


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Average Every 10 Sec During ROP Exam for first 4 minutes
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Average Every 15 Mins Hour 2-3 After ROP Exam
Can you write Python code for it?

#AVERAGE EVERY 15 MINUTES TWO HOURS AFTER ROP EXAM

def perdelta3(start, end, delta):
    r3 = []
    curr = start
    while curr < end:
        r3.append(curr)
        curr += delta
    return r3

# datetime(year, month, day, hour, etc.)

df3PI = df_clean.PI[(Z+timedelta(hours=1)):(Z+timedelta(hours=2))]
df3O2 = df.SpO2[(Z+timedelta(hours=1)):(Z+timedelta(hours=2))]
df3PR = df.PR[(Z+timedelta(hours=1)):(Z+timedelta(hours=2))]
win3 = timedelta(minutes=15) #any unit of time, make the range

r3 = perdelta3((Z+timedelta(hours=1)), (Z+timedelta(hours=2)), win3)

#make the series to store
avg3PI = Series(index = r3, name = 'PI q15MinHr2')
avg3O2 = Series(index = r3, name = 'O2 q15MinHr2')
avg3PR = Series(index = r3, name = 'PR q15MinHr2')

#average!
for i3 in r3:
    avg3PI[i3] = df3PI[i3:(i3+win3)].mean()
    avg3O2[i3] = df3O2[i3:(i3+win3)].mean()
    avg3PR[i3] = df3PR[i3:(i3+win3)].mean()

result3 = concat([avg3PI, avg3O2, avg3PR], axis=1, join='inner')
print result3


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Average Every 30 Mins Hour 3-4 After ROP Exam
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Average Every 5 Mins Hour 1-2 After ROP Exam
Can you write Python code for it?

#AVERAGE EVERY 5 MINUTES ONE HOUR AFTER ROP EXAM

def perdelta2(start, end, delta):
    r2 = []
    curr = start
    while curr < end:
        r2.append(curr)
        curr += delta
    return r2

# datetime(year, month, day, hour, etc.)

df2PI = df_clean.PI[Z:(Z+timedelta(hours=1))]
df2O2 = df.SpO2[Z:(Z+timedelta(hours=1))]
df2PR = df.PR[Z:(Z+timedelta(hours=1))]
win2 = timedelta(minutes=5) #any unit of time, make the range

r2 = perdelta2(Z, (Z+timedelta(hours=1)), win2) #define the average using function

#make the series to store
avg2PI = Series(index = r2, name = 'PI q5MinHr1')
avg2O2 = Series(index = r2, name = 'O2 q5MinHr1')
avg2PR = Series(index = r2, name = 'PR q5MinHr1')

#average!
for i2 in r2:
    avg2PI[i2] = df2PI[i2:(i2+win2)].mean()
    avg2O2[i2] = df2O2[i2:(i2+win2)].mean()
    avg2PR[i2] = df2PR[i2:(i2+win2)].mean()

result2 = concat([avg2PI, avg2O2, avg2PR], axis=1, join='inner')
print result2


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Average Every Hour 4-24 Hours Post ROP Exam
Can you write Python code for it?

#AVERAGE EVERY 60 MINUTES 4-24 HOURS AFTER ROP EXAM

def perdelta5(start, end, delta):
    r5 = []
    curr = start
    while curr < end:
        r5.append(curr)
        curr += delta
    return r5

# datetime(year, month, day, hour, etc.)

df5PI = df_clean.PI[(Z+timedelta(hours=3)):(Z+timedelta(hours=24))]
df5O2 = df.SpO2[(Z+timedelta(hours=3)):(Z+timedelta(hours=24))]
df5PR = df.PR[(Z+timedelta(hours=3)):(Z+timedelta(hours=24))]
win5 = timedelta(minutes=60) #any unit of time, make the range

r5 = perdelta5((Z+timedelta(hours=3)), (Z+timedelta(hours=24)), win5)

#make the series to store
avg5PI = Series(index = r5, name = 'PI q60MinHr4+')
avg5O2 = Series(index = r5, name = 'O2 q60MinHr4+')
avg5PR = Series(index = r5, name = 'PR q60MinHr4+')

#average!
for i5 in r5:
    avg5PI[i5] = df5PI[i5:(i5+win5)].mean()
    avg5O2[i5] = df5O2[i5:(i5+win5)].mean()
    avg5PR[i5] = df5PR[i5:(i5+win5)].mean()

result5 = concat([avg5PI, avg5O2, avg5PR], axis=1, join='inner')
print result5


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Baseline Averages
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Import File into Python
Change File Name!
Can you write Python code for it?

df = pd.read_csv('/Users/John/Dropbox/LLU/ROP/Pulse Ox/ROP018PO.csv',
            parse_dates={'timestamp': ['Date','Time']},
            index_col='timestamp',
            usecols=['Date', 'Time', 'SpO2', 'PR', 'PI', 'Exceptions'],
            na_values=['0'],
            converters={'Exceptions':  readD}
            )

#parse_dates tells the read_csv function to combine the date and time column
#into one timestamp column and parse it as a timestamp.
#    pandas is smart enough to know how to parse a date in various formats

#index_col sets the timestamp column to be the index.

#usecols tells the read_csv function to select only the subset of the columns.
#na_values is used to turn 0 into NaN

#converters: readD is the dict that means any string with 'C' with be NaN (for PI)

#dfclean = df[27:33][df[27:33].loc[:, ['SpO2', 'PR', 'PI', 'Exceptions']].apply(pd.notnull).all(1)]
#clean the dataframe to get rid of rows that have NaN for PI purposes
df_clean = df[df.loc[:, ['PI', 'Exceptions']].apply(pd.notnull).all(1)]

"""Pulse ox date/time is 1 mins and 32 seconds faster than phone. Have to correct for it."""

TC = timedelta(minutes=1, seconds=32)


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Mild, Moderate, and Severe Desaturation Events
Can you write Python code for it?

df_O2_pre = df[Y:W]

#Find count of these ranges
below = 0 # v <=80
middle = 0 #v >= 81 and v<=84
above = 0 #v >=85 and v<=89
ls = []

b_dict = {}
m_dict = {}
a_dict = {}

for i, v in df_O2_pre['SpO2'].iteritems():

    if v <= 80: #below block

        if not ls:
            ls.append(v)
        else:
            if ls[0] >= 81: #if the range before was not below 80

                if len(ls) >= 5: #if the range was greater than 10 seconds, set to 5 because data points are every 2

                    if ls[0] <= 84: #was it in the middle range?
                        m_dict[middle] = ls
                        middle += 1
                        ls = [v]
                    elif ls[0] >= 85 and ls[0] <=89: #was it in the above range?
                        a_dict[above] = ls
                        above += 1
                        ls = [v]

                else: #old list wasn't long enough to count
                    ls = [v]
            else: #if in the same range
                ls.append(v)

    elif v >= 81 and v<= 84: #middle block

        if not ls:
            ls.append(v)
        else:
            if ls[0] <= 80 or (ls[0]>=85 and ls[0]<= 89): #if not in the middle range
                if len(ls) >= 5: #if range was greater than 10 seconds

                    if ls[0] <= 80: #was it in the below range?
                        b_dict[below] = ls
                        below += 1
                        ls = [v]
                    elif ls[0] >= 85 and ls[0] <=89: #was it in the above range?
                        a_dict[above] = ls
                        above += 1
                        ls = [v]
                else: #old list wasn't long enough to count
                    ls = [v]

            else:
                ls.append(v)

    elif v >= 85 and v <=89: #above block

        if not ls:
            ls.append(v)
        else:
            if ls[0] <=84 : #if not in the above range

                if len(ls) >= 5: #if range was...


I am working on the file ".ipynb_checkpoints/Masimo160127-checkpoint.ipynb".
The first task is:
Set Date and Time of ROP Exam and Eye Drops
Can you write Python code for it?

df_first = df.first_valid_index() #get the first number from index

Y = pd.to_datetime(df_first) #convert index to datetime
# Y = TIME DATA COLLECTION BEGAN / First data point on CSV

# SYNTAX:
# datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])

W = datetime(2016, 1, 20, 7, 30)+TC
# W = first eye drop dtarts
X = datetime(2016, 1, 20, 8, 42)+TC
# X = ROP Exam Started
Z = datetime(2016, 1, 20, 8, 46)+TC
# Z = ROP Exam Ended

df_last = df.last_valid_index() #get the last number from index

Q = pd.to_datetime(df_last)

# Q = TIME DATA COLLECTION ENDED / Last Data point on CSV


I am working on the file ".ipynb_checkpoints/Matching Market v2-Copy1-checkpoint.ipynb".
The first task is:
Test DataFrame Appending
Can you write Python code for it?

# Test the Book
ledger = pd.DataFrame(columns = ("role","name","price","cleared"))
ledger=ledger.append({"role":"seller","name":"gas","price":24,"cleared":"in process"},ignore_index=True)
ledger=ledger.append({"role":"buyer","name":"gas","price":25,"cleared":"in process"},ignore_index=True)
#df.append({'foo':1, 'bar':2}, ignore_index=True)
rows=ledger.loc[ledger['role'] == 'seller']
print(rows['price'].tolist())

for index, row in ledger.iterrows():
    if (row['role'] == 'seller'):
        print("yes","index")
        ledger.ix[index,'cleared']='True'
        row['cleared']='True'
    else:
        print("No change")
print()
print(ledger)


I am working on the file ".ipynb_checkpoints/Matching Market v2-checkpoint.ipynb".
The first task is:
Example Market
In the following code example we use the buyer and supplier objects to create a market. At the market a single price is announced which causes as many units of goods to be swapped as possible. The buyers and sellers stop trading when it is no longer in their own interest to continue.
Can you write Python code for it?

# make a supplier and get the asks
supplier = Seller("Natural Gas")
supplier.set_quantity(100,0,10)

book = Book()
book.set_asks([supplier])

# make a buyer and get the bids
buyerNames = ('home', 'industry', 'cat')
buyerDictionary = {}
for name in buyerNames:
    buyerDictionary[name] = Buyer('%s' %name)

for obj in buyerDictionary.values():
    obj.set_quantity(100,0,10)

'''
# make a buyer and get the bids
buyer1 = Buyer("Home")
buyer1.set_quantity(100,0,10)

# make a buyer and get the bids
buyer2 = Buyer("Industry")
buyer2.set_quantity(100,0,10)
'''

book.set_bids([buy for buy in buyerDictionary.values()])
ledger = book.get_ledger()

gas_market = Market()
gas_market.add_seller(supplier)
gas_market.add_buyer(buyer1)
gas_market.add_buyer(buyer2)
gas_market.set_book()
asks = gas_market.get_asks()
#print(asks)

clearing = gas_market.get_clearing_price()
gas_market.annotate_ledger(clearing)
new_ledger = gas_market.get_ledger()

pd.DataFrame.head(new_ledger)


I am working on the file ".ipynb_checkpoints/Matching Market v2-checkpoint.ipynb".
The first task is:
Operations Research Formulation
The market can also be formulated as a very simple linear program or linear complementarity problem. It is clearer and easier to implement this market clearing mechanism with agents. One merit of the agent-based approach is that we don't need linear or linearizeable supply and demand function.
The auctioneer is effectively following a very simple linear program subject to constraints on units sold. The auctioneer is, in the primal model, maximizing the consumer utility received by customers, with respect to the price being paid, subject to a fixed supply curve. On the dual side the auctioneer is minimizing the cost of production for the supplier, with respect to quantity sold, subject to a fixed demand curve. It is the presumed neutrality of the auctioneer which justifies the honest statement of supply and demand.
An alternative formulation is a linear complementarity problem. Here the presence of an optimal space of trades ensures that there is a Pareto optimal front of possible trades. The perfect opposition of interests in dividing the consumer and producer surplus means that this is a zero sum game. Furthermore the solution to this zero-sum game maximizes societal welfare and is therefore the Hicks optimal solution.
Next Steps
A possible addition of this model would be to have a weekly varying demand of customers, for instance caused by the use of natural gas as a heating agent. This would require the bids and asks to be time varying, and for the market to be run over successive time periods. A second addition would be to create transport costs, or enable intermediate goods to be produced. This would need a more elaborate market operator.  Another possible addition would be to add a profit maximizing broker. This may require adding belief, fictitious play, or message passing.
The object-orientation of the models will probably need to be further rationalized. Right now the market requires very particular ordering of calls to function correctly.
Can you write Python code for it?

# To
# Create a dictionary for the properties of the agents

objectNames = ("foo", "bar", "cat", "mouse")
objectDictionary = {}
for name in objectNames:
    objectDictionary[name] = MyClass(property=foo,property2=bar)

for obj in objectDictionary.itervalues():
    obj.DoStuff(variable = foobar)


I am working on the file ".ipynb_checkpoints/Matching Market v2-checkpoint.ipynb".
The first task is:
Test DataFrame Appending
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Matching Market v8-checkpoint.ipynb".
The first task is:
Example Market
In the following code example we use the buyer and supplier objects to create a market. At the market a single price is announced which causes as many units of goods to be swapped as possible. The buyers and sellers stop trading when it is no longer in their own interest to continue.
Can you write Python code for it?

# Show some real consumption data, for more data see folder data analytics
#read montly consumption data of 2010 into a dataframe
df = pd.read_csv('2010cbstestrun.csv', header=0, index_col=0)
df = df.transpose()

#plot the 2010 monthly consumption data
df.plot();
df

# make initialization dictionary
init_buyer = {'elec':{'b':400, 'm' : 673}, 'indu':{'b':400, 'm':1171}, 'home':{'b': 603, 'm': 3615}}
init_seller = {'netherlands' : (2000, 0, 10), 'Russia' : (2000, 0, 10)}

# make a history book to record every timestep
hist_book = []

# set the starting time
period_null= datetime.date(2010,1,1)


I am working on the file ".ipynb_checkpoints/Matching Market v8-checkpoint.ipynb".
The first task is:
Observer
The observer holds the clock and collects data. In this setup it tells the market another tick has past and it is time to act. The market will instruct the other agents. The observer initializes the model, thereby making real objects out of the classes defined above.
Can you write Python code for it?

class Observer():
    def __init__(self, x, y, z):
        self.init_buyer = x
        self.init_seller = y
        self.maxrun = z
        self.hist_book = []
        self.buyer_dict = {}
        self.seller_dict = {}
        self.timetick = 0
        self.gas_market = ''
        self.reserve = []

    def set_buyer(self, buyer_info):
        for name in buyerinfo:
            self.buyer_dict[name] = Buyer('%s' % name)
            self.buyer_dict[name].base_demand = buyer_info[name]['b']
            self.buyer_dict[name].max_demand = buyer_info[name]['m']

    def set_seller(self, seller_info):
        for name in seller_info:
            self.seller_dict[name] = Seller('%s' % name)
            self.seller_dict[name].prod = seller_info[name][0]

    def get_reserve(self):
        reserve = []
        for name in self.seller_dict:
            reserve.append(self.seller_dict[name].reserve)
        return reserve

    def set_market(self):
        self.gas_market = Market()
        #add suplliers and buyers to this market
        for supplier in self.seller_dict.values():
            self.gas_market.add_seller(supplier)
        for buyer in self.buyer_dict.values():
            self.gas_market.add_buyer(buyer)
        self.gas_market.seller_dict = self.seller_dict
        self.gas_market.buyer_dict = self.buyer_dict

    def run_it(self):
        # Timing
        # time initialising
        startit_init = time.time()

        #initialise, setting up all the agents
        first_run = True
        if first_run:
            self.set_buyer(self.init_buyer)
            self.set_seller(self.init_seller)
            self.set_market()
            first_run=False

        # time init stop
        stopit_init = time.time() - startit_init
        print('%s : init' % stopit_init)

        for period in range(self.maxrun):
            # time the period
            startit_period = time.time()

            self.timetick += 1
           ...


I am working on the file ".ipynb_checkpoints/Matching Market v8-checkpoint.ipynb".
The first task is:
Operations Research Formulation
The market can also be formulated as a very simple linear program or linear complementarity problem. It is clearer and easier to implement this market clearing mechanism with agents. One merit of the agent-based approach is that we don't need linear or linearizable supply and demand function.
The auctioneer is effectively following a very simple linear program subject to constraints on units sold. The auctioneer is, in the primal model, maximizing the consumer utility received by customers, with respect to the price being paid, subject to a fixed supply curve. On the dual side the auctioneer is minimizing the cost of production for the supplier, with respect to quantity sold, subject to a fixed demand curve. It is the presumed neutrality of the auctioneer which justifies the honest statement of supply and demand.
An alternative formulation is a linear complementarity problem. Here the presence of an optimal space of trades ensures that there is a Pareto optimal front of possible trades. The perfect opposition of interests in dividing the consumer and producer surplus means that this is a zero sum game. Furthermore the solution to this zero-sum game maximizes societal welfare and is therefore the Hicks optimal solution.
Next Steps
A possible addition of this model would be to have a weekly varying demand of customers, for instance caused by the use of natural gas as a heating agent. This would require the bids and asks to be time varying, and for the market to be run over successive time periods. A second addition would be to create transport costs, or enable intermediate goods to be produced. This would need a more elaborate market operator.  Another possible addition would be to add a profit maximizing broker. This may require adding belief, fictitious play, or message passing.
The object-orientation of the models will probably need to be further rationalized. Right now the market requires very particular ordering of calls to function correctly.
Can you write Python code for it?

# timeit

stopit = time.time()
dtstopit = datetime.datetime.now()

print('it took us %s seconds to get to this conclusion' % (stopit-startit))
print('in another notation (h:m:s) %s'% (dtstopit - dtstartit))

# print the run results
price = df_hb['price']
fig = price.plot()
plt.ylabel('€ / unit')
plt.show()

quantity = df_hb['quantity']
fig = quantity.plot()
plt.ylabel('quantity')
plt.show()


I am working on the file ".ipynb_checkpoints/Matching Market v8-checkpoint.ipynb".
The first task is:
classes buyers and sellers
Below we are constructing the buyers and sellers in classes.
Can you write Python code for it?

# measure how long it takes to run the script
startit = time.time()
dtstartit = datetime.datetime.now()

class Seller():
    def __init__(self, name):
        self.name = name
        self.wta = []
        self.step = 0
        self.prod = 2000
        self.lb_price = 10
        self.ub_price = 20
        self.reserve = 500000
        #multiple market idea, also ga away from market
        self.subscr_market = {}

    # the supplier has n quantities that they can sell
    # they may be willing to sell this quantity anywhere from a lower price of l
    # to a higher price of u
    def set_quantity(self):
        n = self.prod
        l = self.lb_price
        u = self.ub_price
        wta = []
        for i in range(n):
            p = rnd.uniform(l, u)
            wta.append(p)
        self.wta = wta

    def get_name(self):
        return self.name

    def get_asks(self):
        return self.wta

    def clear_wta(self):
        self.wta = []

    def extract(self, cur_extraction):
        if self.reserve > 0:
            self.reserve = self.reserve - cur_extraction
        else:
            self.prod = 0

class Buyer():
    def __init__(self, name):
        self.name = name
        self.wtp = []
        self.step = 0
        self.base_demand = 0
        self.max_demand = 0
        self.lb_price = 10
        self.ub_price = 20

    # the supplier has n quantities that they can buy
    # they may be willing to sell this quantity anywhere from a lower price of l
    # to a higher price of u
    def set_quantity(self):
        n = int(self.consumption(self.step))
        l = self.lb_price
        u = self.ub_price
        wtp = []
        for i in range(n):
            p = rnd.uniform(l, u)
            wtp.append(p)
        self.wtp = wtp

    # gets a little to obvious
    def get_name(self):
        return self.name

    # return list of willingness to pay
    def get_bids(self):
        return self.wtp

    # is this neccesary?
    def clear_wtp(self):
        self.wtp = []

   ...


I am working on the file ".ipynb_checkpoints/Matching Market v8-checkpoint.ipynb".
The first task is:
run the model
To run the model we create the observer. The observer creates all the other objects and runs the model.
Can you write Python code for it?

# create observer and run the model
# first data about buyers then sellers and then model ticks
years = 1
timestep = 12

obser1 = Observer(init_buyer, init_seller, years*timestep)
obser1.run_it()
#get the info from the observer
hist_book = obser1.hist_book

# recording the total run
def write_to_csv(hist_book):
    f = open('hist_book.csv', 'a')
    for item in hist_book:
        f.write('%s,%s\n' % (item[0], item[1]))
    f.close()

#write_to_csv(hist_book)

# make a dataframe of clearing prices
df_hb = pd.DataFrame(hist_book)
df_hb = df_hb.set_index(0)
df_hb.index.name = 'month'
df_hb.rename(columns={1: 'price', 2: 'quantity'}, inplace=True)


I am working on the file ".ipynb_checkpoints/Matplotlib_tutorial-checkpoint.ipynb".
The first task is:
Styles
We can use a simple style usage for the plots made by plt.
Let's try and use the ggplot style
Can you write Python code for it?

# import the styles
from matplotlib import style

# select the 'ggplot' style from the default ones
style.use('ggplot')

# and make the same plot -- pretty cool heh? :)
plt.plot([5,6,7,8],[7,4,8,4])
plt.title("Random Plot")
plt.xlabel("$1/2\cdot x$")
plt.ylabel("Dispersion $[m]$")


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Create the connection to the MongoDB server. The first argument is the IP we've supplied above and the second is the port (TCP) through which we'll be talking to the EC2 server and the MongoDB instance running inside it.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Dang it's unicode. Unicode is a pain in the ass to deal with, it's some Python specific format. Let's try converting it to a more usable Python format (datetime). We care about the relative difference between the date variable. Doing this with a string doesn't make sense to a computer so we have to transform it into a quantitative measure of time.
Can you write Python code for it?

string_year = collection.find()[0]['date'][0:4]
year = datetime.strptime(string_year, '%Y')
year


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
First, I'm going to take a look at what one of the reviews looks like. I totally could have done something wrong earlier and the output is pure garbage. This is a good sanity check to make.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Let's grab a a subset reviews from the academic reviews collection. Suppose we want a random set of 5000, all from after 2010, from each city in our dataset.
Can you write Python code for it?

collection = db.get_collection('academic_reviews')

#I cheated and just had a list of all the states.
#You should try to find a unique list of all the states from mongoDB as an exercise.
states = [u'OH', u'NC', u'WI', u'IL', u'AZ', u'NV']


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Note that the datetime above is given as January-1st, 2014. We only gave it a year variable so it just defaults to the first day of that year. That's all good though, we just want stuff after 2010, we just define the beginning of 2010 to be January-1st 2010.
Can you write Python code for it?

threshold_year = datetime.strptime('2010', '%Y')


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Print the collection names
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Running the below code is going to take a little while. But it's essentially doing the following:
    For each review in the reviews database:
        If the review comes from one of our states:
            Check to see if the review was made after 2010:
              If it did, append it to the overall reviews dictionary.
              If it didn't, proceed to the next review.
Can you write Python code for it?

#reviews_dict = {}
num_reviews = 50000

for obj in collection.find():
    if obj['state'] == 'IL':
        try:
            if len(reviews_dict[obj['state']]) > num_reviews:
                continue
        except KeyError:
            pass
        if datetime.strptime(obj['date'][0:4], '%Y') >= threshold_year:
            del obj['_id']
            try:
                reviews_dict[obj['state']].append(obj)
            except KeyError:
                reviews_dict[obj['state']]=[obj]
    else:
        pass


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
So the new dictionary we created is structured with each state being a key and each entry being a list of reviews. Let's take a look at what Ohio looks like:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
Sweet, this is pretty much what we were expecting. Let's pull out the date field from this entry. We're going to filter on this in a second. Depending on its type, we're going to need to develop different strategies in constructing the logical statements that filter for the date.
Can you write Python code for it?

print collection.find()[0]['date']
print type(collection.find()[0]['date'])


I am working on the file ".ipynb_checkpoints/Mongo_Connect-checkpoint.ipynb".
The first task is:
You have to change this variable each time the EC2 server stops or restarts. Please email/text me to get the new IP address.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Another example showing morphological blackhat operation with circular SEs.
Can you write Python code for it?

img = cv2.imread('images/binary_circles.jpg',0)
kernel1 = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (9,9))
black1 = cv2.morphologyEx(img, cv2.MORPH_BLACKHAT, kernel1)
kernel2 = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (15,15))
black2 = cv2.morphologyEx(img, cv2.MORPH_BLACKHAT, kernel2)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(black1, cmap=plt.cm.gray)
plt.title(r'Blackhat with a $9\times9$ circular kernel')
plt.subplot(133)
plt.imshow(black2, cmap=plt.cm.gray)
plt.title(r'Blackhat with a $15\times15$ circular kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Binary Morphology
Morphological transformations are some simple operations based on the image shape. It is normally performed on binary images. It needs two inputs, one is our original image, second one is called structuring element or kernel which decides the nature of operation. Two basic morphological operators are Erosion and Dilation. Then its variant forms like Opening, Closing, Gradient etc also comes into play.
Images can be thought to be a mapping from the integer space $\mathbb{Z^2}$ to $\mathbb{R}$. For a binary image, the mapping reduces to $f: \mathbb{Z^2} \to {0,1}$. Every pixel at position $(x,y)\in\mathbb{Z^2}$ is either completely dark $(0)$ or completely bright $(1)$. We shall now introduce two very important set operations.
Reflection of a set $B\subseteq\mathbb{Z}^2$ is defined as $\hat{B}  = {w : w=-b, \forall b\in B}$. E.g. if $B$ is a set of points of the form $(x,y)$, then $\hat{B}$ can be found by replacing those points by $(-x,-y)$.
Translation of a set $B\subseteq\mathbb{Z}^2$ by a point $z=(z_1,z_2)$ is defined as $(B)_z={c : c=b+z,\forall b\in B}$. E.g. if $B$ is a set of points of the form $(x,y)$, then $(B)_z$ can be found by replacig those points by $(x+z_1, y+z_2)$.
Set reflection and translation are employed extensively in morphology to formulate operations based on so-called structuring elements (SEs) or kernels. SEs are basically small sets or subimages used to probe an image under study for properties of interest. Usually they are often taken in rectangular, circular, elliptical or cross shapes.
Erosion
With $A, B \subseteq \mathbb{Z}^2$, the erosion of $A$ by $B$ (SE) is defined as $A\ominus B = {z : (B)_z \subseteq A}$. In words, this equation indicated that the erosion of $A$ by $B$ is the set of all points $z$ such that $B$, translated by $z$, is contained in $A$. The basic idea of erosion is just like soil erosion only, it erodes away the boundaries of foreground object (Always try to keep foreground in white). So what does it do? The kernel (SE) slides through the...
Can you write Python code for it?

img = cv2.imread('images/binary_circles.jpg',0)
kernel1 = np.ones((3,3), np.uint8)
erosion1 = cv2.erode(img, kernel1, iterations = 1)
kernel2 = np.ones((5,5), np.uint8)
erosion2 = cv2.erode(img, kernel2, iterations = 1)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(erosion1, cmap=plt.cm.gray)
plt.title(r'Erosion with a $3\times3$ kernel')
plt.subplot(133)
plt.imshow(erosion2, cmap=plt.cm.gray)
plt.title(r'Erosion with a $5\times5$ kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Blackhat
It is the difference between the closing of the input image and input image.
Can you write Python code for it?

img = cv2.imread('images/binary_circles.jpg',0)
kernel1 = np.ones((5,5), np.uint8)
black1 = cv2.morphologyEx(img, cv2.MORPH_BLACKHAT, kernel1)
kernel2 = np.ones((11,11), np.uint8)
black2 = cv2.morphologyEx(img, cv2.MORPH_BLACKHAT, kernel2)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(black1, cmap=plt.cm.gray)
plt.title(r'Morphological blackhat with a $5\times5$ kernel')
plt.subplot(133)
plt.imshow(black2, cmap=plt.cm.gray)
plt.title(r'Morphological blackhat with a $11\times11$ kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Closing
The morphological closing of $A$ by $B$ is defined as $A \bullet B = (A\oplus B)\ominus B$, which is nothing but dilation followed by erosion. It is useful in removing noise. It is useful in closing small holes inside the foreground objects, or small black points on the object.
Can you write Python code for it?

img = cv2.imread('images/binary_circles.jpg',0)
kernel1 = np.ones((5,5), np.uint8)
closing1 = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel1)
kernel2 = np.ones((7,7), np.uint8)
closing2 = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel2)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(closing1, cmap=plt.cm.gray)
plt.title(r'Closing with a $5\times5$ kernel')
plt.subplot(133)
plt.imshow(closing2, cmap=plt.cm.gray)
plt.title(r'Closing with a $7\times7$ kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Dilation
With $A, B \subseteq \mathbb{Z}^2$, the dilation of $A$ by $B$ (SE) is defined as $A\oplus B = {z:(\hat{B})z\cap A \ne \phi}$. In words, the dilation of $A$ by $B$ is the set consisting of all the structuring element _origin locations where the reflected and translated $B$ overlaps at least one element of $A$.
It is just opposite of erosion. Here, a pixel element is $1$ if atleast one pixel under the kernel is $1$. So it increases the white region in the image or size of foreground object increases. Normally, in cases like noise removal, erosion is followed by dilation. Because, erosion removes white noises, but it also shrinks our object. So we dilate it. Since noise is gone, they won’t come back, but our object area increases. It is also useful in joining broken parts of an object.
Can you write Python code for it?

img = cv2.imread('images/binary_retina.png',0)
kernel1 = np.ones((3,3), np.uint8)
dilation1 = cv2.dilate(img, kernel1, iterations = 1)
kernel2 = np.ones((5,5), np.uint8)
dilation2 = cv2.dilate(img, kernel2, iterations = 1)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(dilation1, cmap=plt.cm.gray)
plt.title(r'Dilation with a $3\times3$ kernel')
plt.subplot(133)
plt.imshow(dilation2, cmap=plt.cm.gray)
plt.title(r'Dilation with a $5\times5$ kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Morphological Gradient
It is the difference between dilation and erosion of an image.
Can you write Python code for it?

img = cv2.imread('images/binary_circles.jpg',0)
kernel1 = np.ones((3,3), np.uint8)
grad1 = cv2.morphologyEx(img, cv2.MORPH_GRADIENT, kernel1)
kernel2 = np.ones((5,5), np.uint8)
grad2 = cv2.morphologyEx(img, cv2.MORPH_GRADIENT, kernel2)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(grad1, cmap=plt.cm.gray)
plt.title(r'Morphological gradient with a $3\times3$ kernel')
plt.subplot(133)
plt.imshow(grad2, cmap=plt.cm.gray)
plt.title(r'Morphological gradient with a $5\times5$ kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
Opening
The morphological opening of $A$ by $B$ is defined as $A \circ B = (A\ominus B)\oplus B = \cup{(B)_z:(B)_z\subseteq A}$, which is nothing but erosion followed by dilation. It is useful in removing noise.
Can you write Python code for it?

img = cv2.imread('images/binary_sand.png',0)
kernel1 = np.ones((5,5), np.uint8)
opening1 = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel1)
kernel2 = np.ones((7,7), np.uint8)
opening2 = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel2)

plt.figure(figsize=(15, 5))
plt.subplot(131)
plt.imshow(img, cmap=plt.cm.gray)
plt.title('Original image')
plt.subplot(132)
plt.imshow(opening1, cmap=plt.cm.gray)
plt.title(r'Opening with a $5\times5$ kernel')
plt.subplot(133)
plt.imshow(opening2, cmap=plt.cm.gray)
plt.title(r'Opening with a $7\times7$ kernel')


I am working on the file ".ipynb_checkpoints/Morphological_Image_Processing-checkpoint.ipynb".
The first task is:
SEs of different shapes
OpenCV provides built-in functions for creating SEs of custom shapes like circle, ellipse, cross, etc. They turn out to be useful for dufferent purposes.
Can you write Python code for it?

# Rectangular Kernel
rect = cv2.getStructuringElement(cv2.MORPH_RECT,(25,25))
# Elliptical Kernel
ellip = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(25,25))
# Cross-shaped Kernel
cross = cv2.getStructuringElement(cv2.MORPH_CROSS,(25,25))

plt.matshow(ellip, cmap=cm.gray)
plt.title(r'A $19\times 19$ elliptical / circular kernel')


I am working on the file ".ipynb_checkpoints/MultipleLinearRegression-checkpoint.ipynb".
The first task is:
$\mbox{rank} = \mbox{const}+ B\times \mbox{hoursplayed} + C \times \mbox{wins}$
Can you write Python code for it?

import pandas as pd
import numpy as np
import statsmodels.api as sm
data = pd.read_json("Overwatch090317.json")

rank=[x for x in data['rank']]
level=[x for x in data['level']]
wins=[x for x in data.get('comp', {})]
# winsdata=[x for x in wins.index[x]]
# wins=[x for x in data['comp'] if x <1E100]
a=wins[0]
b=a.get('total',{})
b.get('wins',{})
def getWins(x):
    b=x.get('total',{})
    return(b.get('wins',{}))
winsExtracted =[getWins(x) for x in  wins]

#[x for x in rank if x <1e100]
def filterNaN(x):
    if x<1e100:
        return(x)
    else:
        return(0)#0 ->treat NaN as no rank
rankFiltered=[filterNaN(x) for x in rank ]
levelFiltered= [filterNaN(x) for x in level]

from pandas.stats.api import ols
df = pd.DataFrame({"A": rankFiltered, "B": level, "C":wins})
res = ols(y=df['A'], x=df[['B','C']])
res

data.to_xarray()

rankdata=[x for x in data['rank'] if x <1E100] # is x < 'infinity'

import matplotlib.pyplot as plt


I am working on the file ".ipynb_checkpoints/NB1_IdentifyProblem+DataClean-checkpoint.ipynb".
The first task is:
From the above results, from the 33, variables,column id number 1 is an integer diagnosis 569 non-null object. and rest are float. More on python variables
Can you write Python code for it?

#check for missing variables
#data.isnull().any()

data.diagnosis.unique()


I am working on the file ".ipynb_checkpoints/NB1_IdentifyProblem+DataClean-checkpoint.ipynb".
The first task is:
In the result displayed, you can see the data has 569 records, each with 32 columns.
The “info()” method provides a concise summary of the data; from the output, it provides the type of data in each column, the number of non-null values in each column, and how much memory the data frame is using.
The method get_dtype_counts() will return the number of columns of each type in a DataFrame:
Can you write Python code for it?

# Review data types with "info()".
#data.info()

# Review number of columns of each data type in a DataFrame:
data.get_dtype_counts()


I am working on the file ".ipynb_checkpoints/NB1_IdentifyProblem+DataClean-checkpoint.ipynb".
The first task is:
Load Dataset
First, load the supplied CSV file using additional options in the Pandas read_csv function.
Inspecting the data
The first step is to visually inspect the new data set. There are multiple ways to achieve this, the easiest being to request the first few records using the DataFrame data.head()* method. By default, “data.head()” returns the first 5 rows from the DataFrame object df (excluding the header row). Alternatively, one can also use “df.tail()” to return the five rows of the data frame. For both head and  tail methods, there is an option to specify the number of records by including the required number in between the parentheses when calling either method.Inspecting the data
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/NLP-checkpoint.ipynb".
The first task is:
Add your own dictionary
Can you write Python code for it?

# Dict objects can also be used to check words against a custom list of correctly-spelled words
# known as a Personal Word List. This is simply a file listing the words to be considered, one word per line.
# The following example creates a Dict object for the personal word list stored in “mywords.txt”:
pwl = enchant.request_pwl_dict("../Data_nlp/mywords.txt")

pwl.check('pappapero'), pwl.suggest('cittin'), pwl.check('altro')

# PyEnchant also provides the class DictWithPWL which can be used to combine a language dictionary
# and a personal word list file:
d2 = enchant.DictWithPWL("it_IT", "../Data_nlp/mywords.txt")

d2.check('altro') & d2.check('pappapero'), d2.suggest('cittin')

%%timeit
d2.suggest('poliza')


I am working on the file ".ipynb_checkpoints/NLP-checkpoint.ipynb".
The first task is:
check entire phrase
Can you write Python code for it?

from enchant.checker import SpellChecker
chkr = SpellChecker("it_IT")

chkr.set_text("questo è un picclo esmpio per dire cm funziona")
for err in chkr:
    print(err.word)
    print(chkr.suggest(err.word))

print(chkr.word, chkr.wordpos)

chkr.replace('pippo')
chkr.get_text()


I am working on the file ".ipynb_checkpoints/NN Playground-VariX-checkpoint.ipynb".
The first task is:
GPU
Make sure you enabled GPU suppor, and have a compatible (ie. nvidia) GPU
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/NN Playground-VariX-checkpoint.ipynb".
The first task is:
Redraw
Allows you to define a new threshold (provided it's more aggressive than the original), without calculating everything.
Can you write Python code for it?

threshold = 0.96
img =  PIL.Image.open('test_vis4.jpg')
w, h = img.size
img = img.resize((int(w/orig_scale),int(h/orig_scale)), PIL.Image.NEAREST)
imbase = img
print "adding overlay"
for pair in found_pairs:
    if pair[2]>threshold:
        imbase = add_outline(np.array(imbase),pair[0],pair[1],pair[3], pair[2])
showarray(imbase)

img = PIL.Image.open('test_vis4.jpg').convert('LA')
arr = np.array(img)
w, h = img.size
print w

a


I am working on the file ".ipynb_checkpoints/NN Playground-VariX-checkpoint.ipynb".
The first task is:
The cell bellows checks that opencv and its python bindings are properly installed
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/NN Playground-VariX-checkpoint.ipynb".
The first task is:
Train dataset appender
Can you write Python code for it?

img =  PIL.Image.open('neg1.jpg')
w, h = img.size
#img = img.resize((int(w/4),int(h/4)), PIL.Image.NEAREST)
imbase = img
img = img.convert('L')
w, h = img.size
scale = 1
width = 36
height = 36
import array
findex = 4327
with open('posneg.txt', 'a') as f:
    print "starting processing"
    while w >=36*2 and h >36*2:
        img = img.resize((int(w/1.3),int(h/1.3)), PIL.Image.NEAREST)
        w, h = img.size
        print img.size
        im = np.array(img)
        scale*=1.3
        found_pairs = []
        i = 0
        j = 0
        last_result = 0
        while i < int(h-36):
            while j < int(w-36):
                imtmp = np.array(im [i:i+36, j:j+36]/256.0)
                face_prob = get_face_prob(imtmp)
                if  face_prob > 0.9:
                    showarray(imtmp*255)
                    buff=array.array('B')
                    for k in range(0, 36):
                        for l in range(0, 36):
                            buff.append(int(imtmp[k][l]*255))
                    findex += 1
                    # open file for writing
                    filename = '0/lbe%i.pgm'%findex
                    try:
                        fout=open("train_images/"+filename, 'wb')
                    except IOError, er:
                        print "Cannot open file ", filename, "Exiting … \n", er
                    # define PGM Header
                    pgmHeader = 'P5' + '\n' + str(width) + '  ' + str(height) + '  ' + str(255) + '\n'
                    # write the header to the file
                    fout.write(pgmHeader)
                    # write the data to the file
                    buff.tofile(fout)
                    # close the file
                    fout.close()
                    f.write(filename + " 0\n")
                j+=4
            i+=4
            j = 0
        print "adding overlay"
        for pair in found_pairs:
            imbase = add_outline(np.array(imbase),pair[0],pair[1],scale, pair[2])
showarray(imbase)

print findex


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
2. Basic interactions with the Census dataset API
Combining data from two API endpoints into a human-readable table
The dataset in our example is not excessively large, so we can explore different approaches to interacting with it:

Download some or all data to the computer we're using ('local'). Keep the data in memory and do stuff.
Only download what we need when we need it. Doing stuff may require additional calls to the API.

Both have pros and cons. Both are used in the following examples.
Some points of interest:

The data are not provided in tabular form
Human-readable variable names and definitions are stored separately from the data

In order to make a human readable table we need to:

Download variable definitions
Download data
Replace shorthand variable codes with human readable names
Reformat the data into a table

Get API and variable information:
Can you write Python code for it?

# First, get the basic info about the dataset.
# References: Dataset API (https://api.census.gov/data/2010/surname.html)
#             Requests API (http://docs.python-requests.org/en/master/)
#             Python 3 JSON API (https://docs.python.org/3/library/json.html)

api_base_url = "http://api.census.gov/data/2010/surname"
api_info = requests.get(api_base_url)
api_json = api_info.json()

# Uncomment the next line(s) to see the response content.
# NOTE: JSON and TEXT don't look much different to us. They can look very different to a machine!
#print(api_info.text)
print(json.dumps(api_json, indent=4))

# The output is a dictionary - data are stored as key:value pairs and can be nested.

# Request and store a local copy of the dataset variables.
# Note that the URL could be hard coded just from referencing the API, but
#   we are navigating the JSON data.
var_link = api_json['dataset'][0]['c_variablesLink']
print(var_link)

# Use the variable info link to make a new request

variables = requests.get(var_link)
jsonData = variables.json()
variable_data = jsonData['variables']

# Note that this is a dictionary of dictionaries.
print(json.dumps(variable_data, indent=4))

print(variable_data.keys())


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
4. Create a name demographic pie-chart
Can you write Python code for it?

# Using index positions is good for doing something quick, but in this case makes code easy to break.
# Selecting different surname dataset variables or re-ordering variables will result in errors.
print(d)
pcts = d[1][2:8]
print('\n\n',pcts)

# Create the labels and get the data for the pie chart.
# Note that we are using the downloaded source data, not the dataframe
#     used for the table above.
labels = ['White', 'Asian', '2+ Races', 'Native American', 'Black', 'Hispanic']
pcts = d[1][2:8]
#print(pcts)

# Create a pie chart (https://matplotlib.org/2.0.2/examples/pie_and_polar_charts/pie_demo_features.html)
plt.pie(
    # using data percentages
    pcts,
    # Use labels defined above
    labels=labels,
    # with no shadows
    shadow=False,
    # with the start angle at 90%
    startangle=90,
    # with the percent listed as a fraction
    autopct='%1.1f%%',
    )

# View the plot drop above
plt.axis('equal')

# View the plot
plt.tight_layout()
plt.show()


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
Build the query URL and send the request. Pass the response data into a Pandas dataframe for viewing.
Can you write Python code for it?

# References: Pandas (http://pandas.pydata.org/)

base_url = 'http://api.census.gov/data/2010/surname?get='
query_url = base_url + desired_vars + name_query
name_stats = requests.get(query_url)
d = name_stats.json()

# The response data are not very human readable.
print('Raw response data:\n')
print(d)

# Pass the data to a Pandas dataframe.
# In addition to being easier to read, dataframes simplify further analysis.

# The simplest dataframe would use the variable names returned with the data. Example: PCTWHITE
# It's easier to read the descriptive labels provide via the variables API.
# The code block below replaces variable names with labels as it builds the dataframe.
column_list = []

for each in d[0]:                                   # For each variable in the response data (stored as d[0])
    label = v[each]['label']                        #     look up that variable's label in the variable dictionary
    column_list.append(label)                       #     add the variable's label to the list of column headers

df = pd.DataFrame([d[1]], columns=column_list)      # Create a dataframe using the column names created above. Data
                                                    # for the dataframe comes from d[1]

print('\n\nPandas dataframe:')
df


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
Create a dataframe (table) with variable labels as column names and append data
Can you write Python code for it?

df = pd.DataFrame([surname_data[1]], columns=column_list)  # Create a dataframe using the column names created above. Data
                                                           # for the dataframe comes from rows 2-10 (positions 1-9)
                                                           # of surname_data.

# The table we just created is empty. Here we add the surname data:
for surname in d[2:]:
    tdf = pd.DataFrame([surname], columns=column_list)
    df = df.append(tdf)

print('\n\nPandas dataframe:')
df.sort_values(by=["National Rank"])


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
Get surname data:
Can you write Python code for it?

# References: Pandas (http://pandas.pydata.org/)

# Default vars: 'RANK,COUNT,PCTWHITE,PCTAPI,PCT2PRACE,PCTAIAN,PCTBLACK,PCTHISPANIC'
desired_vars = 'NAME,COUNT,PCTWHITE,PCTAPI,PCT2PRACE,PCTAIAN,PCTBLACK,PCTHISPANIC&RANK=1:10'     # Top ten names

base_url = 'http://api.census.gov/data/2010/surname?get='
query_url = base_url + desired_vars
name_stats = requests.get(query_url)
surname_data = name_stats.json()

# The response data are not very human readable.
# Note that this is a list of lists. Data within lists are typically accessed by position number. (There are no keys.)
print('Raw response data:\n')
print(json.dumps(surname_data, indent=4))


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
Laying out the API response like a table helps illustrate what we're doing here. For easier reading the "surname_data" variable has been replace with "d" in the image below.
The variable codes in d[0] will be replaced with human readable descriptions from the variable list (v).

Replace variable codes with human readable labels
Can you write Python code for it?

# Pass the data to a Pandas dataframe.
# In addition to being easier to read, dataframes simplify further analysis.

# The simplest dataframe would use the variable names returned with the data. Example: PCTWHITE
# It's easier to read the descriptive labels provide via the variables API.
# The code block below replaces variable names with labels as it builds the dataframe.
column_list = []

for each in surname_data[0]:                        # For each variable in the response data (stored as surname_data[0])
    label = variable_data[each]['label']            #     look up that variable's label in the variable dictionary
    column_list.append(label)                       #     add the variable's label to the list of column headers
    print(each, ":", label)

print('\n', column_list)


I am working on the file ".ipynb_checkpoints/Name_Data-checkpoint.ipynb".
The first task is:
Referring to the variables API, decide which variables are of interest and edit accordingly.
Can you write Python code for it?

# Default vars: 'RANK,COUNT,PCTWHITE,PCTAPI,PCT2PRACE,PCTAIAN,PCTBLACK,PCTHISPANIC'
desired_vars = 'RANK,COUNT,PCTWHITE,PCTAPI,PCT2PRACE,PCTAIAN,PCTBLACK,PCTHISPANIC'


I am working on the file ".ipynb_checkpoints/NeuralNetworks-checkpoint.ipynb".
The first task is:
The actual, known values (0,1,2,3,4,5,6,7,8,9) associated with each image can be found in the target array:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
Data Collection
Now we are ready to collect all our data. Let's first collect the training data into a DataFrame.
Can you write Python code for it?

# # First get the positive reviews for the train_df.
# train_df = data_collect('aclImdb/train/pos/', True, train_pos_dict, train_pos_urls)
# # Then append the negative reviews
# train_df = train_df.append(data_collect('aclImdb/train/pos/', False, train_neg_dict, train_neg_urls))


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
If we did this right for training positives, the length of the dictionary keys should be equal to the number of unique URLs in its URL list.
Can you write Python code for it?

# print len(train_pos_dict.keys()), len(list(set(list(train_pos_urls))))

# %time
# train_neg_dict = make_url_dict(train_neg_urls)
# fp = open("url_movie_train_neg.json","w")
# json.dump(train_neg_dict, fp)
# fp.close()

# %time
# test_pos_dict = make_url_dict(test_pos_urls)
# fp = open("url_movie_test_pos.json","w")
# json.dump(test_pos_dict, fp)
# fp.close()

# %time
# test_neg_dict = make_url_dict(test_neg_urls)
# fp = open("url_movie_test_neg.json","w")
# json.dump(test_neg_dict, fp)
# fp.close()

# Reload
# with open("url_movie_tr_pos.json", "r") as fd:
#     train_pos_dict = json.load(fd)
# with open("url_movie_train_neg.json", "r") as fd:
#     train_neg_dict = json.load(fd)
# with open("url_movie_test_pos.json", "r") as fd:
#     test_pos_dict = json.load(fd)
# with open("url_movie_test_neg.json", "r") as fd:
#     test_neg_dict = json.load(fd)


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
Let's see how long each list is.
Can you write Python code for it?

print len(train_pos_urls), len(train_neg_urls), len(test_pos_urls), len(test_neg_urls)


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
Now let's get the list of URLs for each of our data sets: both positive and negative for train and test.
Can you write Python code for it?

# get all urls for train and test, neg and pos
with open('aclImdb/train/urls_pos.txt','r') as f:
    train_pos_urls = f.readlines()

with open('aclImdb/train/urls_neg.txt','r') as f:
    train_neg_urls = f.readlines()

with open('aclImdb/test/urls_pos.txt','r') as f:
    test_pos_urls = f.readlines()

with open('aclImdb/test/urls_neg.txt','r') as f:
    test_neg_urls = f.readlines()


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
Now that we have saved movie names associated with each URL, we can finally create our data table of reviews. We will define a function data_collect which iterates over our directories, making a pandas dataframe out of all the reviews in a particular category (e.g. Test Set, Positive Reviews).
Can you write Python code for it?

def data_collect(directory, pos, url_dict, url_list):
    '''
    Inputs:
        directory: Directory to collect reviews from. ex) 'aclImdb/train/pos/'
        Pos: True or False, depending on whether the reviews are labelled positive or not.
        url_dict: the relevant URL-Movie dictionary (created above) for the particular category
        url_list: the list of URLs for that particular category
    '''
    # Column names for the data frame
    review_df = pd.DataFrame(columns=['movie_id', 'stars', 'positive', 'text', 'url', 'movie_name'])
    # Crawl over the directory, attaining relevant data for each of the .txt review files.
    train_pos_names = list(os.walk(directory))[0][2]
    for review in train_pos_names:
        # Andrew L. Maas's stanford group encoded the reviewID and number of stars for a review in the file's name.
        # For example, "0_10.txt" means reviewID 0 received 10 stars. The reviews are in the same order as the URLs,
        # so the reviewID is precisely the location of that movie's URL in the respective URL list.
        stars = int(review.split("_")[1].split(".")[0])
        movieID = int(review.split("_")[0]) #everything before the underscore
        fp = open('%(dir)s%(review)s' % {'dir': directory, 'review': review}, 'r')
        text = fp.read()
        url = url_list[movieID]
        movie_name = url_dict[url]
        reviewDict = {'movie_id': movieID, 'stars': stars, 'positive': pos, 'text': text, 'url': url, 'movie_name': movie_name}
        review_df = review_df.append(pd.DataFrame(reviewDict),index=[0])
    return review_df


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
Now we'll create a testing data frame.
Can you write Python code for it?

# # First get the positive reviews for the train_df.
# test_df = data_collect('aclImdb/test/pos/', True, test_pos_dict, test_pos_urls)
# # Then append the negative reviews
# test_df = test_df.append(data_collect('aclImdb/train/pos/', False, test_neg_dict, test_neg_urls))


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
There are 12500 reviews in each sub data set. Each review has a corresponding URL. However, the URL lists have duplicates, as two reviews can be for the same movie and thus be found on the same IMDB webpage.
We would like to save the URLs and their associated movies into a dictionary for later use. This way we can do all the scraping up front. Let's define a function which does this scraping for a given set of URLs.
Can you write Python code for it?

def make_url_dict(url_list):
    '''
    Input: List of URLs.
    Output: Dictionary of URL: movie based on scraped movie title.
    '''
    url_dict = dict(zip(url_list, [None]*len(url_list)))
    index = 0
    for url in url_list:
        if url_dict[url] == None:
            url_dict[url] = get_movie(url)
        # Every once in awhile, let us know how many URLs we have digested out of 12,500 total.
        if random.random() < 0.001:
            print i
        i += 1
        time.sleep(0.001)


I am working on the file ".ipynb_checkpoints/New-checkpoint.ipynb".
The first task is:
We gathered data from Andrew L. Maas from Standford University, based on IMDB movies review (describe this more). The data was automatically split into test and train sets, with each set containing polarized movie reviews (each review was a text file) in subdirectories. Since the creators of the original data set were not interested in predicted Box Office Scores, they didn't bother to save the names of th movies, only the URLs that the reviews were scraped from on IMDB.com. Thus we had to go back into all those URLs and scrape the movie names from the top of the pages. Below we write a function to scrape an IMDB url and return a movie name.
Can you write Python code for it?

# function to get name of movie from each URL
def get_movie(url):
    '''
    Scrapes a given URL from IMDB.com. The URL's page contains many reviews for one particular movie.
    This function returns the name of that  movie.
    '''
    pageText = requests.get(url)
    # Keep asking for the page until you get it. Sleep if necessary.
    while (pageText==None):
        time.sleep(5)
        pageText = requests.get(url)
    soup = BeautifulSoup(pageText.text,"html.parser")
    # Some of our URL's are expired! Return None if so.
    if soup == None or soup.find("div",attrs={"id":"tn15title"}) == None:
        return None
    return soup.find("div",attrs={"id":"tn15title"}).find("a").get_text()


I am working on the file ".ipynb_checkpoints/Noise axons and the binary DFORCE-checkpoint.ipynb".
The first task is:
Besides some small fluctations, the binary version does nearly as well as the real valued version above!
I need to quantify performance differences between the two models and,
need to train with many more kinds of inputs to be sure but
I think with your axon part and my FORCE part could really work nicely toghether in a short published report. Not sure where, or if, you are along in the publication process. But if you're interested in a joint thing, let me know.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Noise axons and the binary DFORCE-checkpoint.ipynb".
The first task is:
Binary FORCE
Implementation
I first define a function that converts unit rates into a binary output. Then I replace the standard learning FORCE math to use the binary decoded response generated by this function.
Can you write Python code for it?

def decode(x, rho):
    xd = zeros_like(x)

    xd[x > rho] = 1
    xd[x < -rho] = -1

    return xd


I am working on the file ".ipynb_checkpoints/Noise axons and the binary DFORCE-checkpoint.ipynb".
The first task is:
In a biologically constrained parameterization Tee's effect holds up perfectly.
FORCE
FORCE is a variant 'liquid state machine' or 'echo state machine' supervised learning system, that can tame chaotic patterns to match a target. Here's an classic example of FORCE wherein firing rates represent real values.
Neurons here are replaced units, or neural mass models that represent the aggregrate firing rates of many neurons. This is a common, and useful, theoretical abstraction, in case you're not familiar with it.
Neural masses
We simulate the model:
$$\frac{d\mathbf{x}}{dt} = -\mathbf{x} + g J \tanh{[\mathbf{x}]} $$
with $x \in \mathcal{R}^N$ (vector), $J \in \mathcal{R}^{N \times N}$ (matrix), $g \in \mathcal{R}$ (scalar). Randomly draw each element of $J$ from a Gaussian distribution with zero mean and variance $1/N$. Characterize the output of the system for increasing values of $g$. If $g$ is greater than 1.5 the system will behave chaotically. If you take $g$ = 1 it reduces the system from FORCE to a tradtional 'echo state' system.

Here's an example of what the firing of 10 random units looks like:
Can you write Python code for it?

def f1(x,t0):
    return -x + g*dot(J,tanh(x))

N = 100
J = normal(0,sqrt(1/N),(N,N))
x0 = uniform(-0.8,0.3,N)
t = linspace(0,50,1000)

plt.figure(figsize=(10,5))
for s,g in enumerate(linspace(0.5,1.5,3)):
    plt.subplot(1,3,s+1)
    x = odeint(f1,x0,t)
    plt.plot(t,x[:,choice(N,10)])
    plt.title('g = '+str(g),fontweight='bold')
plt.show()

x.shape


I am working on the file ".ipynb_checkpoints/Noise axons and the binary DFORCE-checkpoint.ipynb".
The first task is:
Setup the axon code
Can you write Python code for it?

from numpy.random import normal
from numpy import asarray
def axon(a0, n, sigma_l, sigma_b, min_v=-0.07, max_v=-0.04):
    """A Tee-type axon.

    An intial voltage a0 is sequentially corrupted
    by additive noise even when signal boosting
    is perfect.

    Params
    ------
    a0 : scalar
        The initial voltage
    n : scalar
        Number of nodes
    sigma_l : scalar
        Std dev of the leak
    sigma_b : scalar
        Std dev of the boost (should be very small)
    min_v : scalar
        Min voltage (-0.07 mV)
    max_v : scalar
        Max volate (-0.04 mV)
    """

    if a0 > max_v:
        raise ValueError("a0 must be smaller than {0}".format(max_v))
    if a0 < min_v:
        raise ValueError("a0 must be larger than {0}".format(min_v))

    ax = [a0, ]
    for i in range(1, n+1):
        ax.append(
            ax[-1] +
            normal(loc=0.0, scale=sigma_l, size=1)[0] +
            normal(loc=0.0, scale=sigma_b, size=1)[0]
        )

    ax = asarray(ax)

    # Clip
    ax[ax < min_v] = min_v
    ax[ax > max_v] = max_v

    return ax


I am working on the file ".ipynb_checkpoints/Noise axons and the binary DFORCE-checkpoint.ipynb".
The first task is:
The binary loss function
Now let's learn, with binary codes.
Can you write Python code for it?

target = lambda t0: cos(2 * pi * t0 / 50)  # target pattern

def f3(t0, x):
    return -x + g * dot(J, tanh_x) + dot(w, tanh_x) * u

dt = 1       # time step
tmax = 800   # simulation length
tstop = 500

N = 300
J = normal(0, sqrt(1 / N), (N, N))
x0 = uniform(-0.5, 0.5, N)
t = linspace(0, 50, 500)

rho = 0.1  # Set and rand vec
rho = uniform(0, 0.5, N)

g = 1.5
u = uniform(-1, 1, N)
w = uniform(-1 / sqrt(N), 1 / sqrt(N), N)  # initial weights
P = eye(N)  # Running estimate of the inverse correlation matrix
lr = .4  # learning rate

rho = repeat(.5, N)

# simulation data: state,
# output, time, weight updates
x, z, t, wu = [x0], [], [0], [0]

# Set up ode solver
solver = ode(f3)
solver.set_initial_value(x0)

# Integrate ode, update weights, repeat
while t[-1] < tmax:
    tanh_x = tanh(x[-1])
    tanh_xd = decode(tanh_x, rho)  # BINARY CODE INTRODUCED HERE!
    z.append(dot(w, tanh_xd))

    error = target(t[-1]) - z[-1]
    q = dot(P, tanh_xd)
    c = lr / (1 + dot(q, tanh_xd))
    P = P - c * outer(q, q)
    w = w + c * error * q

    # Stop training time
    if t[-1] > tstop:
        lr = 0

    wu.append(np.sum(np.abs(c * error * q)))

    solver.integrate(solver.t + dt)
    x.append(solver.y)
    t.append(solver.t)

# last update for readout neuron
z.append(dot(w, tanh_x))

# plot
x = np.array(x)
t = np.array(t)

plt.figure(figsize=(10, 5))
plt.subplot(2, 1, 1)
plt.plot(t, target(t), '-r', lw=2)
plt.plot(t, z, '-b')
plt.legend(('target', 'output'))
plt.ylim([-1.1, 3])
plt.xticks([])
plt.subplot(2, 1, 2)
plt.plot(t, wu, '-k')
plt.yscale('log')
plt.ylabel('$|\Delta w|$', fontsize=20)
plt.xlabel('time', fontweight='bold', fontsize=16)
plt.show()


I am working on the file ".ipynb_checkpoints/OpenStack-API-Lab2-PartI-checkpoint.ipynb".
The first task is:
Excercise 2:
Can you write Python code for it?

# Obtain a list of all the object names in your container.


I am working on the file ".ipynb_checkpoints/OpenStack-API-Lab2-PartI-checkpoint.ipynb".
The first task is:
First, we obtain a client connection to Swift (we are usign the v2 APIs)
Can you write Python code for it?

conn = swiftclient.client.Connection(auth_version=2, **config)

# Create a container, use a UUID to make sure it has a globally unique name.
import uuid
bucket_name = "lab2_{0}".format(str(uuid.uuid4()))
conn.put_container(bucket_name)


I am working on the file ".ipynb_checkpoints/OpenStack-API-Lab2-PartI-checkpoint.ipynb".
The first task is:
In the cell below, we obtain a client connection to the Nova endpoint. It can be used to for example start, stop and terminate instances.
Can you write Python code for it?

config = {'username':os.environ['OS_USERNAME'],
          'api_key':os.environ['OS_PASSWORD'],
          'project_id':os.environ['OS_TENANT_NAME'],
          'auth_url':os.environ['OS_AUTH_URL'],
           }
from novaclient.client import Client
nc = Client('2',**config)


I am working on the file ".ipynb_checkpoints/OpenStack-API-Lab2-PartI-checkpoint.ipynb".
The first task is:
Question 1:
What does it mean that the object store has a global, flat namespace? What is the practical consequence for you when using it?
Can you write Python code for it?

# List containers
(response, bucket_list) = conn.get_account()
for bucket in bucket_list:
    print bucket['name']

# Put an object in the container
object_id = conn.put_object(bucket_name, "test_object", "Hi Swift")


I am working on the file ".ipynb_checkpoints/OpenStack-API-Lab2-PartI-checkpoint.ipynb".
The first task is:
To establish a client connection, we will need to pass a dictionary with information about the tenant, user, credentials and the API Identity endpoint. Here, I have sourced the "openrc.sh file" obtained from the Horizon dashboard in the underlying shell prior to starting the notebook. Hence, in order to actually run the code below, you would need to do the same with your own credentials.
Can you write Python code for it?

config = {'user':os.environ['OS_USERNAME'],
          'key':os.environ['OS_PASSWORD'],
          'tenant_name':os.environ['OS_TENANT_NAME'],
          'authurl':os.environ['OS_AUTH_URL']}


I am working on the file ".ipynb_checkpoints/Overfitting_Demo_Ridge_Lasso-checkpoint.ipynb".
The first task is:
Above: Fit looks pretty wild, too.  Here's a clear example of how overfitting is associated with very large magnitude estimated coefficients.

#
#
Ridge Regression
Ridge regression aims to avoid overfitting by adding a cost to the RSS term of standard least squares that depends on the 2-norm of the coefficients $\|w\|$.  The result is penalizing fits with large coefficients.  The strength of this penalty, and thus the fit vs. model complexity balance, is controled by a parameter lambda (here called "L2_penalty").
Define our function to solve the ridge objective for a polynomial regression model of any degree:
Can you write Python code for it?

def polynomial_ridge_regression(data, deg, l2_penalty):
    model = graphlab.linear_regression.create(polynomial_features(data,deg),
                                              target='Y', l2_penalty=l2_penalty,
                                              validation_set=None,verbose=False)
    return model


I am working on the file ".ipynb_checkpoints/Overfitting_Demo_Ridge_Lasso-checkpoint.ipynb".
The first task is:
Create a function that prints the polynomial coefficients in a pretty way :)
Can you write Python code for it?

def print_coefficients(model):
    # Get the degree of the polynomial
    deg = len(model.coefficients['value'])-1

    # Get learned parameters as a list
    w = list(model.coefficients['value'])

    # Numpy has a nifty function to print out polynomials in a pretty way
    # (We'll use it, but it needs the parameters in the reverse order)
    print 'Learned polynomial for degree ' + str(deg) + ':'
    w.reverse()
    print numpy.poly1d(w)


I am working on the file ".ipynb_checkpoints/Overfitting_Demo_Ridge_Lasso-checkpoint.ipynb".
The first task is:
Create a function to plot the data, since we'll do it many times
Can you write Python code for it?

def plot_data(data):
    plt.plot(data['X1'],data['Y'],'k.')
    plt.xlabel('x')
    plt.ylabel('y')

plot_data(data)


I am working on the file ".ipynb_checkpoints/Overfitting_Demo_Ridge_Lasso-checkpoint.ipynb".
The first task is:
Create random values for x in interval [0,1)
Can you write Python code for it?

random.seed(98103)
n = 30
x = graphlab.SArray([random.random() for i in range(n)]).sort()