I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Copies & Views :
No Copy:
Simple assignments do not make the copy of array object. Instead, it uses the same id() of the original array to access it. The id() returns a universal identifier of Python object, similar to the pointer in C.
View or Shallow Copy:
NumPy has ndarray.view() method which is a new array object that looks at the same data of the original array.
Deep copy:
The ndarray.copy() function creates a deep copy. It is a complete copy of the array and its data, and doesn’t share with the original array.
Can you write Python code for it?

#no copy
a = np.arange(6)

print ('Our array is:' )
print (a )

print ('Applying id() function:')
print (id(a))

print ('a is assigned to b:' )
b = a
print (b)

print ('b has same id():')
print (id(b))

print ('Change shape of b:')
b.shape = 3,2
print (b)

print ('Shape of a also gets changed:')
print (a)

#view
a = np.array([1,2,3,4])

#print 'Array a:'
print (a )
print(id(a))
#Create view of a:
b = a.view()
print( b )
b.shape=(2,2)
print(id(b))
print (b is a)
print(b.shape)
print(a.shape)

#copy
a = np.array([[10,10], [2,3], [4,5]])

print ('Array a is:')
print( a)

# 'Create a deep copy of a:'
b = a.copy()
print ('Array b is:')
print (b)

#b does not share any memory of a
print ('Can we write b is a')
print (b is a)


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Create an array using repeating list (pythonic way)
Can you write Python code for it?

#using numpy
np.repeat([1, 2, 3], 3)

p = np.ones([2, 3], int)
p

#vstack to stack arrays in sequence vertically (row wise).
np.vstack([p, 2*p])

#hstack to stack arrays in sequence horizontally (column wise)
np.hstack([p, 2*p])


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Dot Product:
[x1,x2,x2]clo[y1,y2,y3] = x1y1+x2xy2+x3y3
Can you write Python code for it?


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Every ndarray is a homogeneous collection of exactly the same data-type
every item takes up the same size block of memory
each block of memory in the array is interpreted in exactly the same way.
Array Creation :
There are a number of ways to initialize new numpy arrays, for example from
– a Python list or tuples
– using functions that are dedicated to generating numpy arrays, such as arange, linspace, etc.
– reading data from files
The basic ndarray is created using an array function in NumPy: numpy.array
syntax : numpy.array(object, dtype = None, copy = True, order = None, subok = False, ndmin = 0)
returns a array object
Can you write Python code for it?

# Eg : one dimensional
a = np.array([1,2,3,4])

print("One dim ")
print(a)
print(type(a))
#more than one dimension
b = np.array([[1, 2], [3, 4]])
print("Two dims")
print(b)

#using ndim
c=np.array([1,2,3,4,5], ndmin = 2)
print("Two dimensional")
print(c.ndim)
print(c.shape)
print(c)

#dtype:
np.array([1, 2, 3], dtype = complex)


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Iterating Over Arrays
create a new 4 by 3 array of random numbers 0-9.
Can you write Python code for it?

tp = np.random.randint(0, 10, (4,3))
tp

#Iterate by row:

for row in tp:
    print(row)

#Iterate by index:

for i, row in enumerate(tp):
    print('row', i, 'is', row)

#Use zip to iterate over multiple iterables.

tp2=tp*2
tp2

for i, j in zip(tp, tp2):
    print(i,'+',j,'=',i+j)


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Note : If the dimensions of two arrays are dissimilar, element-to-element operations are not possible. However, operations on arrays of non-similar shapes is still possible in NumPy, because of the broadcasting capability.
Can you write Python code for it?

#Matrix operations
z = np.array([y, y**2])
print(len(z)) # number of rows of array


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
NumPy package contains an iterator object numpy.nditer. It is an efficient multidimensional iterator object using which it is possible to iterate over an array.
Can you write Python code for it?

a = np.arange(0,60,5)
a = a.reshape(3,4)

print ('Original array is:')
print (a)
print ('\n')

print ('Modified array is:')
for x in np.nditer(a):
   print (x)


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Numeric ranges
This function returns an ndarray object containing evenly spaced values within a given range.
syntax: numpy.arange(start, stop, step, dtype)
Can you write Python code for it?


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Statistical Functions:
NumPy has quite a few useful statistical functions for finding minimum, maximum, percentile standard deviation and variance, etc. from the given elements in the array.
Can you write Python code for it?

a = np.array([-4, -2, 1, 3, 5])

a.sum()

a.max()

a.min()

np.average(a)

a.mean()

a.std() #Standard deviation is the square root of the average of squared deviations from mean


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
To indicate a range. array[start:stop]
Leaving start or stop empty will default to the beginning/end of the array.
Can you write Python code for it?

s[1:5]

#Use negatives to count from the back.
s[-4:]

#can be used to indicate step-size. array[start:stop:stepsize]
#Here we are starting 5th element from the end, and counting backwards by 2 until the beginning of the array is reached.
s[5::2]

#Let's look at a multidimensional array.
m = np.arange(36)
m.resize((6, 6))
m

#Use bracket notation to slice: array[row, column]
m[2, 2]

#to select a range of rows or columns

m[3, 3:]

#We can also perform conditional indexing. Here we are selecting values from the array that are greater than 30.
m[m > 30]

#Here we are assigning all values in the array that are greater than 30 to the value of 30
m[m > 30] = 30
m

x = np.arange(10)
print(x)
s=slice(2,7,2)
print("Done",x[s])


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
Variance is the average of squared deviations, i.e., mean(abs(x - x.mean())**2).
In other words, the standard deviation is the square root of variance.
Can you write Python code for it?

np.var([1,2,3,4])

a.argmax()

a.argmin()


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
cs-4
Topics :
Matlib subpackage,
matrix,
linear algebra method,
matplotlib using numpy.
Can you write Python code for it?

#NumPy package contains a Matrix library numpy.matlib.
import numpy.matlib
#matlib.empty()
#numpy.matlib.empty(shape, dtype, order)
print (np.matlib.empty((2,2)))
print('\n')
#ones
print (np.matlib.ones((2,2)))
print('\n')
#random
print (np.matlib.rand(3,3))
print('\n')
#This function returns the matrix filled with zeros.
#numpy.matlib.zeros()
print (np.matlib.zeros((2,2)))
print('\n')
#numpy.matlib.eye()
#This function returns a matrix with 1 along the diagonal elements and the zeros elsewhere. The function takes the following parameters.
#numpy.matlib.eye(n, M,k, dtype)
print (np.matlib.eye(n = 3, M = 3, k = 1, dtype = float))
print('\n')
#numpy.matlib.identity()
#The numpy.matlib.identity() function returns the Identity matrix of the given size.
#An identity matrix is a square matrix with all diagonal elements as 1.
np.matlib.identity(3)

#creation matrix
i = np.matrix('1,2,3,4')
print(i)
#array to matrix
list=[1,2,3,4]
k = np.asmatrix (list)
print(k)
print(type(k))


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
ndarray.dtype :
an object describing the type of the elements in the array. One can create or specify dtype’s using standard Python types. Additionally NumPy provides types of its own. numpy.int32, numpy.int16, and numpy.float64 are some examples.
Can you write Python code for it?


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
ndarray.shape
the dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the rank, or number of dimensions, ndim.
Can you write Python code for it?

arrey = np.array([[1,2,3],[4,5,6]])
print(arrey)

print(arrey.shape)

#resize ndarray
arrey = np.array([[1,2,3],[4,5,6]])
arrey.shape = (3,2)
print(arrey)

#Resize: NumPy also provides a reshape function to resize an array.
barray = arrey.reshape(2,3)
print(barray)


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
numpy.around()
This is a function that returns the value rounded to the desired precision. The function takes the following parameters.
syntax : numpy.around(a,decimals)
Can you write Python code for it?

#round off
a = np.array([1.0,5.55, 123, 0.567, 25.532])

print ('Original array:')
print (a )
print ('\n')

print ('After rounding:')
print (np.around(a))
print (np.around(a, decimals = 1))


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
numpy.floor()
This function returns the largest integer not greater than the input parameter. The floor of the scalar x is the largest integer i, such that i <= x. Note that in Python, flooring always is rounded away from 0.
Can you write Python code for it?

a = np.array([-1.7, 1.5, -0.2, 0.6, 10])

print ('array:')
print (a)
print ('\n')

print ('The modified array:')
#returns largest intgres

print (np.floor(a))
#returns lowest intgers
print (np.ceil(a))


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
numpy.linspace
This function is similar to arange() function. In this function, instead of step size, the number of evenly spaced values between the interval is specified.
syntax: numpy.linspace(start, stop, num, endpoint, retstep, dtype)
retstep : If true, returns samples and step between the consecutive numbers.
endpoint : True by default, hence the stop value is included in the sequence. If false, it is not included
Can you write Python code for it?


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
numpy.ones
Returns a new array of specified size and type, filled with ones.
Syntax : numpy.ones(shape, dtype = None, order = 'C')
Can you write Python code for it?


I am working on the file "Numpy/numpyclass.ipynb".
The first task is:
resize changes the shape and size of array in-place.
Can you write Python code for it?

o = np.linspace(0, 4, 9)

print(o)

o.resize(3, 3)

o


I am working on the file "Numpy_PyData_DEL_2019.ipynb".
The first task is:
List of Supported Data Types
Can you write Python code for it?

type(numpy_array)

type(numpy_array[0])

numpy_array.append(9.0) # Should throw error


I am working on the file "Numpy_PyData_DEL_2019.ipynb".
The first task is:
NOTE
Numpy arrays are fixed size arrays unlike Python List/Dict ojects which are dynamic in nature.
Any insertion/deletion/concatination when/if performed creates a new copy of the array.
How this strict rule of homogenity and size helps Numpy?
We will witness shortly.
Code Slide Part-2
Can you write Python code for it?

a = np.array([[1,2,3],[4,5,6]])

a

a.shape

a.strides


I am working on the file "Numpy_PyData_DEL_2019.ipynb".
The first task is:
NumPy
Provides
  1. An array object of arbitrary homogeneous items
  2. Fast mathematical operations over arrays
  3. Linear Algebra, Fourier Transforms, Random Number Generation
For more information about introduction to NumPy. Read this Doc
Can you write Python code for it?


I am working on the file "Numpy_PyData_DEL_2019.ipynb".
The first task is:
ndarray = block of memory + indexing scheme + data type descriptor
    raw data
    how to locate an element
    how to interpret an element

Ref: Link
Can you write Python code for it?

numpy_array #Typecasted to float to maintain homogenity

numpy_array.dtype


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Array creation Functions
Can you write Python code for it?

np.arange(4)  # arange(start, stop=None, step=1)

np.arange(1, 5, 2)

np.ones(4)  # ones(shape)

np.ones((2, 3))

np.zeros((2, 3)) # zeros(shape)

np.linspace(0, 1, 5)  # Generate N evenly spaced elements between start and stop values


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Boxplots
Can you write Python code for it?

r1 = np.random.normal(0, 10, 100)
r2 = np.random.gamma(1, 2, 100)
r3 = r1 + r2
_ = plt.boxplot(r3)


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Draw two plots on one coordinate
Can you write Python code for it?

x = np.linspace(0, 2*np.pi, 20)
y1 = np.sin(x)
y2 = np.cos(x)
plt.plot(x, y1)
plt.plot(x, y2)


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Exercises 1
Write a Python program to reverse an array (first element becomes last)
Can you write Python code for it?


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
For more detail of matplotlib please refer to matplotlib.org
Can you write Python code for it?

HTML("<iframe src=https://matplotlib.org/index.html width=800 height=350></iframe>")


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
For more detail of matplotlib please refer to scipy.org
Can you write Python code for it?

HTML("<iframe src=https://www.scipy.org/scipylib/index.html width=800 height=350></iframe>")


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Histogram
Can you write Python code for it?

r1 = np.random.normal(0, 10, 1000)
_ = plt.hist(r1, bins=50, normed=True)


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Scatterplot
Can you write Python code for it?

x = np.random.normal(0, 10, 100)
y = np.random.normal(5, 3, 100)
plt.scatter(x, y)


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
SciPy
SciPy is a collection of packages addressing a number of different standard problem domain in scientific computing. Here is some packages you may use:
- scipy.integrate: numerical integration routines and differential equation solvers
- scipy.linalg: linear algebra routines and matrix decompositions
- scipy.optimize: function optimizers and root finding algorithms
- scipy.stats: standard probability distributions (cdf, pdf, sampler and etc)

Example of linear algebra operations
Can you write Python code for it?

from scipy import linalg  # import linear algebra operations

arr = np.array([[1, 2],
               [3, 4]])
linalg.det(arr)  # compute the determinant of a square matrix

linalg.inv(arr)  # compute the inverse of a squar matrix


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
Working with multiple figures and axes

figure represents the whole picture
axes represents each plot with independ coordinate
The subplot() command specifies (numrows, numcols, fignum) where fignum ranges from 1 to numrows * numcols.
Can you write Python code for it?

x = np.linspace(0, 2*np.pi, 20)
y1 = np.sin(x)
y2 = np.cos(x)

fig = plt.figure()
ax1 = plt.subplot(211)
ax1.plot(x, y1)

ax2 = plt.subplot(212)
ax2.plot(x, y2)


I am working on the file "Numpy_matplotlib_and_Scipy.ipynb".
The first task is:
index and slicing
var[lower:upper:step]
Can you write Python code for it?

a

a[1]

a[1:3]

# omitted boundaries are assumed to be the beginning (or end) of the array

a[:3]  # grap first three elements

a[::2] # element with step size 2


I am working on the file "Nursery/Nursery.ipynb".
The first task is:
If we look at the above result clearly, we can infer the following. Let us take example of the feature 'parent'
count : 12960  (It gives the number of instances of that feature. Here all features have same count. So, we can infer that there         are no missing values.)
unique : 3     (This number says that there 3 different categories of feature parent.)
top : usual    (This gives the category which occured the highest number of times in a feature)
freq : 4320    (This gives the frequency of the top occurence category in that feature.)

4. Class distribution
We can find out how the data is distributed. Because sometimes we might get some data which may be skewed. This leads to unexpected behaviour of our prediction model.
Can you write Python code for it?

# The group by function summarizes a particular feature.

print(df.groupby('class').size())
print("\n")
print(df.groupby('parents').size())
print("\n")
print(df.groupby('has_nurs').size())
print("\n")

print(df.groupby('form').size())
print("\n")

print(df.groupby('children').size())
print("\n")

print(df.groupby('housing').size())
print("\n")

print(df.groupby('finance').size())
print("\n")

print(df.groupby('social').size())
print("\n")

print(df.groupby('health').size())


I am working on the file "Nursery/Nursery.ipynb".
The first task is:
In the above summary data, we ca see that the class feature has 5 different observations.  So, we should look at the data distribution of the expected output for every instance.
Here, as we can see, the recommended variable is very less. Almost negligible.  very_recom variable is seen in 328 observations which is not too high but enough to train our machine.
The remaining three variables are almost equally distributed. So, we are good to go

Data Vizualization
Now that we have a rough idea of our dataset, we can understand it much better by vizualising it. Data can be vizualised by either Univariate plots or Multivariate plots.
Univariate plots helps us in understanding a particular feature.
Where as Multivariate plots helps us in understanding the relationship between different features.
1. Univariate plots
Now we will start with univariate plots to understand each feature in our dataset. Run the cell below to get the same information.
Can you write Python code for it?

plt.rcParams['figure.figsize'] = (5,5)

df['parents'].value_counts().plot(kind='bar')
plt.title('Parents')
plt.show()

df['has_nurs'].value_counts().plot(kind='bar')
plt.title('has_nurs')
plt.show()

df['form'].value_counts().plot(kind='bar')
plt.title('Form')
plt.show()

df['children'].value_counts().plot(kind='bar')
plt.title('Children')
plt.show()

df['housing'].value_counts().plot(kind='bar')
plt.title('Housing')
plt.show()

df['finance'].value_counts().plot(kind='bar')
plt.title('Finance')
plt.show()

df['social'].value_counts().plot(kind='bar')
plt.title('Social')
plt.show()

df['health'].value_counts().plot(kind='bar')
plt.title('Health')
plt.show()

df['class'].value_counts().plot(kind='bar')
plt.title('Class')
plt.show()


I am working on the file "Nursery/Nursery.ipynb".
The first task is:
Summarizing the Dataset
1. Dimensions Of the DataSet.
We have some data and if we don't know how many features and instances our dataset has, it is of no use. So, we should find the dimensions of the dataset.
Can you write Python code for it?

# shape attribute gives the dimensions of a dataframe

print df.shape


I am working on the file "Nursery/Nursery.ipynb".
The first task is:
The above displayed result shows that the dataframe has 12960 rows and 9 colums. It means that we have 9 features and 12960 instances.
2. Looking at the data
It's a very good practice to look at your data before you proceed. We can look how our data is structured
Can you write Python code for it?

# Output printing out first 5 columns

df.head()


I am working on the file "OCR_Example.ipynb".
The first task is:
Classification
Let's prepare to run some classification on the digits data set. As a first step we want to split the data in training and test data sets.
Discussion:
* Why we need to training data sets and test data sets ?
* Why not test on training data set ?
* How much overlap should there be between training and test data sets ?
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
Xtrain, Xtest, ytrain, ytest = train_test_split(digits.data, digits.target, random_state=2)
print(Xtrain.shape, ytrain.shape, Xtest.shape, ytest.shape)


I am working on the file "OCR_Example.ipynb".
The first task is:
Data for our algorithms
Most algorithms in scikit expect data as two dimensional numpy arrays. Luckily scikit learn load_digits does most of this work for us. However if you are using the MINST data set, then you might want to reshape your arrays to be two dimensional. Say do something like:
CODING TIP:
Xtrain = train_dataset.reshape(len(train_dataset),-1)
Can you write Python code for it?

# The data for use in our algorithms
print(digits.data.shape)
print(digits.data[0])

# The target label
print(digits.target)


I am working on the file "OCR_Example.ipynb".
The first task is:
Let's visualize how we did. We will mark mistakes in red:
Can you write Python code for it?

fig, axes = plt.subplots(10, 10, figsize=(8, 8))
fig.subplots_adjust(hspace=0.1, wspace=0.1)

for i, ax in enumerate(axes.flat):
    ax.imshow(Xtest[i].reshape(8, 8), cmap='binary')
    ax.text(0.05, 0.05, str(ypred[i]),
            transform=ax.transAxes,
            color='green' if (ytest[i] == ypred[i]) else 'red')
    ax.set_xticks([])
    ax.set_yticks([])


I am working on the file "OCR_Example.ipynb".
The first task is:
Segway to TensorFlow
What is TensorFlow ? Well at a 100K feet level it is a computational framework for distributed machine learning. Well what does that mean ? It means it is a framework that has built in efficient basic computational constructs (viz. matrix manipulation, softmax computations) and an expressive graph based descriptor language that makes it real easy to express complex machine learning algorithms.
OCR is a problem that is particularly suited to neural networks. So let's see how we could use tensorflow to do a quick model and measure its accuracy. This section is intended to give you a quick flavor of tensorflow. So do not worry as yet if you cannot get all of it.
We will use Stochastic Gradient Descent (SGD) combined with 1-hidden layer neural network with rectified linear units nn.relu()(https://www.tensorflow.org/versions/r0.7/api_docs/python/nn.html#relu) and 1024 hidden nodes.
Reformat into a shape that's more adapted to the models we're going to train:
* data as a flat matrix,
* labels as float 1-hot encodings.
Can you write Python code for it?

image_size = 8
num_labels = 10

def reformat(dataset, labels):
  dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)
  # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]
  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
  return dataset, labels
train_dataset, train_labels = reformat(Xtrain, ytrain)
test_dataset, test_labels = reformat(Xtest, ytest)
print('Training set', train_dataset.shape, train_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Combine team averages
Can you write Python code for it?

ground = {}
for name in set(groundmeans['short name']) :
    ground[name] = mean(groundmeans[ groundmeans['short name'] == name ]['Mean'])

space = {}
for name in set(spacemeans['short name']) :
    space[name] = mean(spacemeans[ spacemeans['short name'] == name ]['Mean'])

data = pd.DataFrame( { 'tecan_ground' : ground, 'tecan_space' : space } )


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Correlations between experiments
For the two machines used, how well do the space-based observations and the
ground-based observations correlate?
Can you write Python code for it?

figure(figsize=(8, 4))
subplot(1,2,1)
plot( data['tecan_ground'], data['tecan_space'], 'ro' )
title( 'Space vs. Ground (Tecan)' )
xlabel('ground')
ylabel('space')
subplot(1,2,2)
plot( data['ground 96h OD'], data['space 96h OD'], 'ro' )
title( 'Space vs. Ground (SpectraMax)' )
xlabel('ground')
ylabel('space')
tight_layout()

# Pearson's correlation of ground and space data for Tecan data

pcc, p = pearsonr( data['tecan_ground'], data['tecan_space'] )

print 'PCC :', pcc
print 'p value :', p

# Pearson's correlation of ground and space data for SpectraMax data

pcc, p = pearsonr( data['ground 96h OD'], data['space 96h OD'] )

print 'PCC :', pcc
print 'p value :', p


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Correlations between machines
For the space-based observations and the ground-based observations,
how well does the data from the two machines correlate?
Can you write Python code for it?

figure(figsize=(8, 4))

subplot(1,2,1)
plot( data['space 96h OD'], data['tecan_space'], 'ro' )
title('Space')
xlabel('SpetraMax OD600')
ylabel('Tecan OD600')
subplot(1,2,2)
plot( data['ground 96h OD'], data['tecan_ground'], 'ro' )
title('Ground')
xlabel('SpetraMax OD600')
ylabel('Tecan OD600')
tight_layout()

from scipy.stats import pearsonr

# Pearson's correlation of Tecan and SpectraMax data for space

pcc, p = pearsonr( data['space 96h OD'], data['tecan_space'] )
print 'PCC :', pcc
print 'p value :', p

# Pearson's correlation of Tecan and SpectraMax data for ground

pcc, p = pearsonr( data['ground 96h OD'], data['tecan_ground'] )
print 'PCC :', pcc
print 'p value :', p


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Join SpectraMax and Tecan averages
Can you write Python code for it?

df_sptmx = pd.read_csv( 'Space_vs_Ground.tsv', sep='\t', index_col=False ).set_index('n')
data = df_sptmx.join( data, on='short name' )
data.head()


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Load raw Tecan data
Can you write Python code for it?

ground1 = pd.read_csv( 'Tecan_final_ground1.csv', index_col='Well' )
ground2 = pd.read_csv( 'Tecan_final_ground2.csv', index_col='Well' )
ground3 = pd.read_csv( 'Tecan_final_ground3.csv', index_col='Well' )

space1 = pd.read_csv( 'Tecan_final_space1.csv', index_col='Well' )
space2 = pd.read_csv( 'Tecan_final_space2.csv', index_col='Well' )
space3 = pd.read_csv( 'Tecan_final_space3.csv', index_col='Well' )

ground1.head()


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Load the plate maps
Can you write Python code for it?

rows = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H' ]
cols = [ '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12' ]

ground1_map = pd.read_csv( 'platemap_ground1.csv', index_col=False, header=None, names=cols )
ground2_map = pd.read_csv( 'platemap_ground2.csv', index_col=False, header=None, names=cols )
ground3_map = pd.read_csv( 'platemap_ground3.csv', index_col=False, header=None, names=cols )

space1_map = pd.read_csv( 'platemap_space1.csv', index_col=False, header=None, names=cols )
space2_map = pd.read_csv( 'platemap_space2.csv', index_col=False, header=None, names=cols )
space3_map = pd.read_csv( 'platemap_space3.csv', index_col=False, header=None, names=cols )

ground1_map.index = rows
ground2_map.index = rows
ground3_map.index = rows

space1_map.index = rows
space2_map.index = rows
space3_map.index = rows

ground1_map


I am working on the file "OD600 Tecan.ipynb".
The first task is:
Map team number to well name
Can you write Python code for it?

def flatten_platemap( platemap ) :
    rows = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H' ]
    cols = [ '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12' ]
    flatmap = {}
    for row in rows :
        for col in cols :
            flatmap[row+col] = ground1_map[col][row]
    return flatmap

ground1 = ground1.join( pd.DataFrame( { 'team' : flatten_platemap(ground1_map) } ) )
ground2 = ground2.join( pd.DataFrame( { 'team' : flatten_platemap(ground2_map) } ) )
ground3 = ground3.join( pd.DataFrame( { 'team' : flatten_platemap(ground3_map) } ) )

space1 = space1.join( pd.DataFrame( { 'team' : flatten_platemap(space1_map) } ) )
space2 = space2.join( pd.DataFrame( { 'team' : flatten_platemap(space2_map) } ) )
space3 = space3.join( pd.DataFrame( { 'team' : flatten_platemap(space3_map) } ) )

ground1.head()


I am working on the file "ODEs.ipynb".
The first task is:
The cell below defines the differential form of the Logistic equation:
$\dfrac{dx}{dt}=rx(1-x)$
Can you write Python code for it?

def LogisticEquation(y,t,r,k=1):
    dydt=r*y*(1-y/k)
    return(dydt)


I am working on the file "ODEs.ipynb".
The first task is:
The function below describes a simple pendulum which has had a small angle approximation in normalised units
$$\dfrac{d^2x}{dt^2} + x = 0$$
This can be split as follows and solved numerically:
$$
\begin{align}
\dfrac{dx}{dt}&=y\
\dfrac{dy}{dt}&=-x
\end{align}
$$
Can you write Python code for it?

def SimplePendulum(y,t,b=0,omega=1):
    x,v  = y
    dydt = np.array([v,-b*v-(omega**2)*x])
    return dydt


I am working on the file "ODEs.ipynb".
The first task is:
The function below describes the equation $\dfrac{dy}{dt}=-y^3+\sin{t}$
Can you write Python code for it?

def Nonlinear2(y,t,args=None):
    dydt=-y**3 + np.sin(t)
    return(dydt)


I am working on the file "ODEs.ipynb".
The first task is:
The function below describes the equation $\dfrac{dy}{dt}=\dfrac{-y}{\tau}$ which has the analytical solution $y(t)=y_0e^{-\dfrac{t}{\tau}}$
Can you write Python code for it?

def RadioactiveDecay(y, t, tau):
    dydt=-1.0*y/tau
    return(dydt)


I am working on the file "ODEs.ipynb".
The first task is:
The function below describes the equation $\dfrac{dy}{dt}=t-y^2$
Can you write Python code for it?

def Nonlinear1(y,t,args=None):
    dydt=t-np.square(y)
    return(dydt)


I am working on the file "ODEs/ODEsEx01.ipynb".
The first task is:
I worked with Hunter, Jessica, and Brett.
Can you write Python code for it?

assert True # leave this for grading the plots


I am working on the file "ODEs/ODEsEx01.ipynb".
The first task is:
In the following cell you are going to solve the above ODE using four different algorithms:

Euler's method
Midpoint method
odeint
Exact

Here are the details:

Generate an array of x values with $N=11$ points over the interval $[0,1]$ ($h=0.1$).
Define the derivs function for the above differential equation.
Using the solve_euler, solve_midpoint, odeint and solve_exact functions to compute
  the solutions using the 4 approaches.

Visualize the solutions on a sigle figure with two subplots:

Plot the $y(x)$ versus $x$ for each of the 4 approaches.
Plot $\left|y(x)-y_{exact}(x)\right|$ versus $x$ for each of the 3 numerical approaches.

Your visualization should have legends, labeled axes, titles and be customized for beauty and effectiveness.
While your final plot will use $N=10$ points, first try making $N$ larger and smaller to see how that affects the errors of the different approaches.
Can you write Python code for it?

f, (ax1, ax2) = plt.subplots(2, 1,figsize=(12,8))
ax1.plot(x,y1,label = 'Euler',color='red');
ax1.plot(x,y2,label = 'Midpoint',color='blue');
ax1.plot(x,y3,label = 'Exact',color='black');
ax1.plot(x,y4,label = 'ODE',color='green');
ax1.set_title('Four Differential Methods')
ax1.set_xlabel('x')
ax1.set_ylabel('y')

ax2.plot(x,(abs(y1-y3)),label = 'Euler Difference',color='red');
ax2.plot(x,(abs(y2-y3)),label = 'Midpoint Difference',color='blue');
ax2.plot(x,(abs(newy4-y3)),label = 'ODE Difference',color='green');
ax2.set_title('Error for Each Differential Method');
ax2.set_ylim(-.1,1);

ax2.set_xlabel('x')
ax2.set_ylabel('y')

plt.tight_layout()
ax2.legend();
ax1.legend();


I am working on the file "ODEs/ODEsEx01.ipynb".
The first task is:
You are now going to solve the following differential equation:
$$
\frac{dy}{dx} = x + 2y
$$
which has the analytical solution:
$$
y(x) = 0.25 e^{2x} - 0.5 x - 0.25
$$
First, write a solve_exact function that compute the exact solution and follows the specification described in the docstring:
Can you write Python code for it?

def solve_exact(x):
    """compute the exact solution to dy/dx = x + 2y.

    Parameters
    ----------
    x : np.ndarray
        Array of x values to compute the solution at.

    Returns
    -------
    y : np.ndarray
        Array of solutions at y[i] = y(x[i]).
    """
    def function(point):
        return .25*np.exp(2*point) - .5*point - .25
    y = np.array([function(i) for i in x])
    return y

assert np.allclose(solve_exact(np.array([0,1,2])),np.array([0., 1.09726402, 12.39953751]))


I am working on the file "ODEs/ODEsEx02.ipynb".
The first task is:
Lorenz system
The Lorenz system is one of the earliest studied examples of a system of differential equations that exhibits chaotic behavior, such as bifurcations, attractors, and sensitive dependence on initial conditions. The differential equations read:
$$ \frac{dx}{dt} = \sigma(y-x) $$
$$ \frac{dy}{dt} = x(\rho-z) - y $$
$$ \frac{dz}{dt} = xy - \beta z $$
The solution vector is $[x(t),y(t),z(t)]$ and $\sigma$, $\rho$, and $\beta$ are parameters that govern the behavior of the solutions.
Write a function lorenz_derivs that works with scipy.integrate.odeint and computes the derivatives for this system.
Can you write Python code for it?

def lorentz_derivs(yvec, t, sigma, rho, beta):
    """Compute the the derivatives for the Lorentz system at yvec(t)."""
    x = yvec[0]
    y = yvec[1]
    z = yvec[2]
    dx = sigma*(y-x)
    dy = x*(rho-z)-y
    dz = x*y - beta*z
    return np.array([dx,dy,dz])

assert np.allclose(lorentz_derivs((1,1,1),0, 1.0, 1.0, 2.0),[0.0,-1.0,-1.0])


I am working on the file "ODEs/ODEsEx02.ipynb".
The first task is:
Use interact to explore your plot_lorenz function with:

max_time an integer slider over the interval $[1,10]$.
N an integer slider over the interval $[1,50]$.
sigma a float slider over the interval $[0.0,50.0]$.
rho a float slider over the interval $[0.0,50.0]$.
beta fixed at a value of $8/3$.
Can you write Python code for it?


I am working on the file "ODEs/ODEsEx02.ipynb".
The first task is:
Write a function plot_lorentz that:

Solves the Lorenz system for N different initial conditions. To generate your initial conditions, draw uniform random samples for x, y and z in the range $[-15,15]$. Call np.random.seed(1) a single time at the top of your function to use the same seed each time.
Plot $[x(t),z(t)]$ using a line to show each trajectory.
Color each line using the hot colormap from Matplotlib.
Label your plot and choose an appropriate x and y limit.

The following cell shows how to generate colors that can be used for the lines:
Can you write Python code for it?

colors = plt.cm.hot(1)

colors

N = 5
colors = plt.cm.hot(np.linspace(0,1,N))
for i in range(N):
    # To use these colors with plt.plot, pass them as the color argument
    print(colors[i])

def plot_lorentz(N=10, max_time=4.0, sigma=10.0, rho=28.0, beta=8.0/3.0):
    """Plot [x(t),z(t)] for the Lorenz system.

    Parameters
    ----------
    N : int
        Number of initial conditions and trajectories to plot.
    max_time: float
        Maximum time to use.
    sigma, rho, beta: float
        Parameters of the differential equation.
    """
    colors = plt.cm.hot(np.linspace(0,1,N))
    np.random.seed(1)
    for i in range(N):
        ic = (np.random.random(3)-.5)*30
        t,soln = solve_lorentz(ic, max_time, sigma, rho, beta)
        x = [e[0] for e in soln]
        y = [e[1] for e in soln]
        z = [e[2] for e in soln]
        plt.plot(x,z,color=colors[i])
        plt.title('Lorenz System for Multiple Trajectories')
        plt.xlabel('X Position')
        plt.ylabel('Y Position')

plot_lorentz()

assert True # leave this to grade the plot_lorenz function


I am working on the file "ODEs/ODEsEx02.ipynb".
The first task is:
Write a function solve_lorenz that solves the Lorenz system above for a particular initial condition $[x(0),y(0),z(0)]$. Your function should return a tuple of the solution array and time array.
Can you write Python code for it?

def solve_lorentz(ic, max_time=4.0, sigma=10.0, rho=28.0, beta=8.0/3.0):
    """Solve the Lorenz system for a single initial condition.

    Parameters
    ----------
    ic : array, list, tuple
        Initial conditions [x,y,z].
    max_time: float
        The max time to use. Integrate with 250 points per time unit.
    sigma, rho, beta: float
        Parameters of the differential equation.
        "
    Returns
    -------
    soln : np.ndarray
        The array of the solution. Each row will be the solution vector at that time.
    t : np.ndarray
        The array of time points used.

    """
    t = np.linspace(0,max_time,250*max_time)
    soln = odeint(lorentz_derivs,ic,t,args=(sigma,rho,beta))
    return t,soln

assert True # leave this to grade solve_lorenz


I am working on the file "ODEs/ODEsEx03.ipynb".
The first task is:
Here is an example of the output of your plot_pendulum function that should show a decaying spiral.
Can you write Python code for it?


I am working on the file "ODEs/ODEsEx03.ipynb".
The first task is:
Use interact to explore the plot_pendulum function with:

a: a float slider over the interval $[0.0,1.0]$ with steps of $0.1$.
b: a float slider over the interval $[0.0,10.0]$ with steps of $0.1$.
omega0: a float slider over the interval $[0.0,10.0]$ with steps of $0.1$.
Can you write Python code for it?


I am working on the file "ODEs/ODEsEx03.ipynb".
The first task is:
Write a function derivs for usage with scipy.integrate.odeint that computes the derivatives for the damped, driven harmonic oscillator. The solution vector at each time will be $\vec{y}(t) = (\theta(t),\omega(t))$.
Can you write Python code for it?

def derivs(y, t, a, b, omega0):
    """Compute the derivatives of the damped, driven pendulum.

    Parameters
    ----------
    y : ndarray
        The solution vector at the current time t[i]: [theta[i],omega[i]].
    t : float
        The current time t[i].
    a, b, omega0: float
        The parameters in the differential equation.

    Returns
    -------
    dy : ndarray
        The vector of derviatives at t[i]: [dtheta[i],domega[i]].
    """
    theta = y[0]
    dtheta = y[1]
    dw = -(g/l)*np.sin(theta) - a*dtheta - b*np.sin(omega0*t)
    return [dtheta,dw]

assert np.allclose(derivs(np.array([np.pi,1.0]), 0, 1.0, 1.0, 1.0), [1.,-1.])

def energy(y):
    """Compute the energy for the state array y.

    The state array y can have two forms:

    1. It could be an ndim=1 array of np.array([theta,omega]) at a single time.
    2. It could be an ndim=2 array where each row is the [theta,omega] at single
       time.

    Parameters
    ----------
    y : ndarray, list, tuple
        A solution vector

    Returns
    -------
    E/m : float (ndim=1) or ndarray (ndim=2)
        The energy per mass.
    """
    theta = y[0]
    omega = y[1]
    if y.ndim == 1:
        theta = y[0]
        omega = y[1]
        EperM = g*l*(1-np.cos(theta))+.5*(l**2)*omega**2
        return EperM
    if y.ndim == 2:
        theta = y[:,0]
        omega = y[:,1]
        EperM = g*l*(1-np.cos(theta))+.5*(l**2)*omega**2
        return EperM

assert np.allclose(energy(np.array([np.pi,0])),g)
assert np.allclose(energy(np.ones((10,2))), np.ones(10)*energy(np.array([1,1])))


I am working on the file "ODEtests.ipynb".
The first task is:
Ah ha! This looks like the negative derivative of the solution, which indicates we have a phase error. Because with phase error, we see the difference
$$\cos(t + \delta) - \cos(t) \approx -\delta\cdot \sin(t)$$
while with an amplitude error, we would see
$$(1+\delta)\cos(t) - \cos(t) \approx \delta\cdot \cos(t).$$
Let's plot the soluton and the error difference, to see if the derivative zeros line up with peaks.
Can you write Python code for it?


I am working on the file "ODEtests.ipynb".
The first task is:
Try playing with the mu parameter. mu=0 gives the harmonic oscillator. mu=10 starts giving pointy spikes. For mu big, you might want to increase the range of to values, from [0,30] to a larger interval like [0,100]. Etc.
Can you write Python code for it?


I am working on the file "ODEtests.ipynb".
The first task is:
We start with a second order linear equation, that has the usual harmonic oscillator solutions.
$$ y''(t) + \omega^2 y(t) = 0, \qquad y(0) = 1, y'(0) = 0.$$
To put this into the numerical solver, we need to reformulate as a 2 dimensional, first order system:
$$\mathbf{y} = \left[\begin{array}{r} y \ \dot{y}  \end{array}\right] \
\mathbf{\dot{y}}=
\left[\begin{array}{r}  \dot{y} \ -\omega^2 y  \end{array}\right],$$
with initial conditions
$$\mathbf{y}_{init} = \left[\begin{array}{r} 1 \ 0  \end{array}\right]$$
Here is a simple code snippet that solves the problem numerically.
Can you write Python code for it?

def rhs(Y,t,omega):  # this is the function of the right hand side of the ODE
    y,ydot = Y
    return ydot,-omega*omega*y

t_arr=np.linspace(0,2*np.pi,101)
y_init =[1,0]
omega = 2.0
y_arr=odeint(rhs,y_init,t_arr, args=(omega,))
y,ydot = y_arr[:,0],y_arr[:,1]
plt.ion()
plt.plot(t_arr,y,t_arr,ydot)

# Let's draw a phase portrait, plotting y and ydot together
plt.plot(y,ydot)
plt.title("Solution curve when omega = %4g" % omega)
plt.xlabel("y values")
plt.ylabel("ydot values")


I am working on the file "ODEtests.ipynb".
The first task is:
With epsilon = 0.1 (radians, which is about 5.7 degrees), it is hard to see a period shift. With epsilon = 0.5 (radians, which just under 30 degrees), we clearly see a shift after ten cycles of oscillation.
How big is the shift? Can we figure this out easily with numerics? And how does it compare with the estimate given by the Poincare-Lindstedt method?
The Lorenz equation
You've probably heard of the butterfly effect (it's even a movie). The idea is that weather can demonstrate chaotic behaviour, so that the flap of a butterfly wing in Brazil could eventually result in a tornado in Alabama.
Lorenz was studying a simplified model for weather, given by a system of three simple ODEs:
$$x' = \sigma(y-x), \quad y'=\rho x - y - xz, \quad z' = xy - \beta z$$
where $x,y,z$ are functions of time, and $\sigma, \rho,\beta$ are fixed constants.
When $\rho$ is small, the behaviour is quite predictable. But when $\rho gets better than about 24.7, you get strange, aperiodic behaviour.
Here is some code that demonstrates the behaviour. We also include 3D plots
Can you write Python code for it?

def rhsLZ(u,t,beta,rho,sigma):
    x,y,z = u
    return [sigma*(y-x), rho*x-y-x*z, x*y-beta*z]

sigma = 10.0
beta = 8.0/3.0
rho1 = 29.0
rho2 = 28.8  # two close values for rho give two very different curves

u01=[1.0,1.0,1.0]
u02=[1.0,1.0,1.0]

t=np.linspace(0.0,50.0,10001)
u1=odeint(rhsLZ,u01,t,args=(beta,rho1,sigma))
u2=odeint(rhsLZ,u02,t,args=(beta,rho2,sigma))

x1,y1,z1=u1[:,0],u1[:,1],u1[:,2]
x2,y2,z2=u2[:,0],u2[:,1],u2[:,2]

from mpl_toolkits.mplot3d import Axes3D

plt.ion()
fig=plt.figure()
ax=Axes3D(fig)
ax.plot(x1,y1,z1,'b-')
ax.plot(x2,y2,z2,'r:')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
ax.set_title('Lorenz equations with rho = %g, %g' % (rho1,rho2))

fig=plt.figure()
ax=Axes3D(fig)
ax.plot(x1,y1,z1)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')

fig=plt.figure()
ax=Axes3D(fig)
ax.plot(x2,y2,z2)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')


I am working on the file "ODEtests.ipynb".
The first task is:
You should play around with the time interval (in the definition of varible t) to observe the predictable, followed by chaotic behaviour. ANd play with other parameters.
van der Pol equation
This equation is used as a testbed for numerical software. It is nonlinear, and collapses to a periodic orbit very quickly. We can include information about the Jacobian, the derivative of the vector function $\mathbf{f}$ in the ODE system $\mathbf{y}' = \mathbf{f}(\mathbf{y},t)$, into the ODE solver, to help it choose appropriate step sizes and reduce errors.
The van der Pol equation is this:
$$y'' - \mu(1-y^2)y' + y = 0$$
with the usual initial conditons for $y(0), y'(0)$. In vector form, we write
$$\mathbf{y} = (y,y')^T, \qquad \mathbf{f}(\mathbf{y}) = (y', \mu(1-y^2)y' - y)^T.$$
The Jacobian is a 2x2 matrix of partial derivatives for $\mathbf{f}$, which is
$$\mathbf{J}(\mathbf{y}) =
\left(
\begin{array}{cc}
0 & 1 \
-2\mu y * y' -1 & \mu(1-y^2)
\end{array}
\right)
$$
We set up the code as follows:
Can you write Python code for it?

def rhsVDP(y,t,mu):
    return [ y[1], mu*(1-y[0]**2)*y[1] - y[0]]

def jac(y,t,mu):
    return [ [0,1], [-2*mu*y[0]*y[1]-1, mu*(1-y[0]**2)]]

mu=100
t=np.linspace(0,300,10001)
y0=np.array([2.0,0.0])
y,info=odeint(rhsVDP,y0,t,args=(mu,),Dfun=jac,full_output=True)

print("mu = %g, number of Jacobian calls is %d", mu, info['nje'][-1])

plt.plot(t,y)


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
All OLPS Algos Market Sectors comparison
Can you write Python code for it?

#list all the algos
olps_algos_sectors = [
algos.Anticor(),
algos.BAH(),
algos.BCRP(),
algos.BNN(),
algos.CORN(),
algos.CRP(),  # removed weights, and thus equivalent to UCRP
algos.CWMR(),
algos.EG(),
algos.Kelly(),
algos.OLMAR(),
algos.ONS(),
algos.PAMR(),
algos.RMR(),
algos.UP()
]

olps_sectors_train = pd.DataFrame(index=algo_names, columns=algo_data)
olps_sectors_train.algo = olps_algos_sectors

# run all algos - this takes more than a minute
for name, alg in zip(olps_sectors_train.index, olps_sectors_train.algo):
    olps_sectors_train.ix[name,'results'] = alg.run(train_sectors)

# we need 14 colors for the plot
n_lines = 14
color_idx = np.linspace(0, 1, n_lines)
mpl.rcParams['axes.color_cycle']=[plt.cm.rainbow(i) for i in color_idx]

# plot as if we had no fees
# get the first result so we can grab the figure axes from the plot
olps_df = olps_sectors_train
ax = olps_df.results[0].plot(assets=False, weights=False, ucrp=True, portfolio_label=olps_df.index[0])
for k, r in olps_df.results.iteritems():
    if k == olps_df.results.keys()[0]: # skip the first item because we have it already
        continue
    r.plot(assets=False, weights=False, ucrp=False, portfolio_label=k, ax=ax[0])

# Kelly went wild, so let's remove it
# get the first result so we can grab the figure axes from the plot
olps_df = olps_sectors_train
ax = olps_df.results[0].plot(assets=False, weights=False, ucrp=True, portfolio_label=olps_df.index[0])
for k, r in olps_df.results.iteritems():
    if k == olps_df.results.keys()[0] or k == 'Kelly' : # skip the first item because we have it already
        continue
    r.plot(assets=False, weights=False, ucrp=False, portfolio_label=k, ax=ax[0])

olps_stats(olps_sectors_train)
olps_sectors_train[metrics].sort_values('profit', ascending=False)

# create the test set dataframe
olps_sectors_test  = pd.DataFrame(index=algo_names, columns=algo_data)
olps_sectors_test.algo  = olps_algos_sectors

# run all algos
for name, alg in zip(olps_sectors_test.index, olps_sectors_test.algo):
   ...


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Comparing the Algorithms
We want to train on market data from a number of years, and test out of sample for a duration smaller than the train set. To get started we accept the default parameters for the respective algorithms and we essentially are just looking at two independent time periods.  In the future we will want to optimize the paramaters on the train set.
Can you write Python code for it?

#list all the algos
olps_algos = [
algos.Anticor(),
algos.BAH(),
algos.BCRP(),
algos.BNN(),
algos.CORN(),
algos.CRP(b=swensen_allocation), # Non Uniform CRP (the Swensen allocation)
algos.CWMR(),
algos.EG(),
algos.Kelly(),
algos.OLMAR(),
algos.ONS(),
algos.PAMR(),
algos.RMR(),
algos.UP()
]

# put all the algos in a dataframe
algo_names = [a.__class__.__name__ for a in olps_algos]
algo_data = ['algo', 'results', 'profit', 'sharpe', 'information', 'annualized_return', 'drawdown_period','winning_pct']
metrics = algo_data[2:]
olps_train = pd.DataFrame(index=algo_names, columns=algo_data)
olps_train.algo = olps_algos


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Kelly went wild and crashed, so let's remove it from the mix
Can you write Python code for it?

# plot as if we had no fees
# get the first result so we can grab the figure axes from the plot
ax = olps_test.results[0].plot(assets=False, weights=False, ucrp=True, portfolio_label=olps_test.index[0])
for k, r in olps_test.results.iteritems():
    if k == olps_test.results.keys()[0] or k == 'Kelly': # skip the first item because we have it already
        continue
    r.plot(assets=False, weights=False, ucrp=False, portfolio_label=k, ax=ax[0])

olps_stats(olps_test)
olps_test[metrics].sort_values('profit', ascending=False)


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Let's add fees of 0.1% per transaction (we pay \$1 for every \$1000 of stocks bought or sold).
Can you write Python code for it?

best_olmar.fee = 0.001
print(best_olmar.summary())
best_olmar.plot(weights=False, assets=False, bah=True, ucrp=True, logy=True, portfolio_label='OLMAR')


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Let's see how individual ETFs contribute to portfolio equity.
Can you write Python code for it?


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Loading the data
We want to train on market data from 2005-2012 inclusive (8 years), and test on 2013-2014 inclusive (2 years). But at this point we accept the default parameters for the respective algorithms and we essentially are looking at two independent time periods.  In the future we will want to optimize the paramaters on the train set.
Can you write Python code for it?

# load data from Yahoo
# Be careful if you cange the order or types of ETFs to also change the CRP weight %'s in the swensen_allocation
etfs = ['VTI', 'EFA', 'EEM', 'TLT', 'TIP', 'VNQ']
#etfs = ['SCHX','VOO','FNDX','PRF','SCHA','VB','FNDA','PRFZ','SCHF','VEA',
#       'FNDF','PXF','SCHC','VSS','FNDC','PDN','SCHE','IEMG','FNDE',
#       'PXH','SCHH','VNQ','VNQI','IFGL','SCHD','VYM','DWX','IQDF','MLPA',
#       'ZMLP','SCHR','VGIT','ITR','VCIT','VMBS','MBG','SCHP','STIP','SHYG',
#       'JNK','BNDX','IGOV','EMLC','VWOB','PSK','PFF','BKLN','VTEB','TFI',
#       'PWZ','CMF','IAU','GLTR']
# Swensen allocation from http://www.bogleheads.org/wiki/Lazy_portfolios#David_Swensen.27s_lazy_portfolio
# as later updated here : https://www.yalealumnimagazine.com/articles/2398/david-swensen-s-guide-to-sleeping-soundly
swensen_allocation = [0.3, 0.15, 0.1, 0.15, 0.15, 0.15]
benchmark = ['SPY']
train_start = datetime(2008,1,1)#'2005-01-01'
train_end   = datetime(2012,12,31)#'2012-12-31'
test_start  = datetime(2013,1,1)#'2013-01-01'
test_end    = datetime(2014,12,31)#'2014-12-31'
#train = DataReader(etfs, 'yahoo', start=train_start, end=train_end)['Adj Close']
#test  = DataReader(etfs, 'yahoo', start=test_start, end=test_end)['Adj Close']
#train_b = DataReader(benchmark, 'yahoo', start=train_start, end=train_end)['Adj Close']
#test_b  = DataReader(benchmark, 'yahoo', start=test_start, end=test_end)['Adj Close']
train = pdr.get_data_yahoo(etfs, train_start, train_end)['Adj Close']
test  = pdr.get_data_yahoo(etfs, test_start, test_end)['Adj Close']
train_b = pdr.get_data_yahoo(benchmark, train_start, train_end)['Adj Close']
test_b  = pdr.get_data_yahoo(benchmark, test_start, test_end)['Adj Close']

# plot normalized prices of the train set
#idx = pd.IndexSlice
#ax1 = (train / train.iloc[idx[0,:]]).plot()
#(train_b / train_b.iloc[idx[0,:]]).plot(ax=ax1)
(train / train.iloc[0,:]).plot()

# plot normalized prices of the test set
#ax2 = (test / test.iloc[0,:]).plot()
#(test_b / test_b.iloc[0,:]).plot(ax=ax2)
(test / test.iloc[0,:]).plot()


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Notice how Kelly crashes right away and how RMR and OLMAR float to the top after some high volatility.
Can you write Python code for it?


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Now OLMAR looks better!
OLMAR Market Sectors comparison
Let's look at algo behavior on market sectors:

XLY Consumer Discrectionary SPDR Fund
XLF Financial SPDR Fund
XLK Technology SPDR Fund
XLE Energy SPDR Fund
XLV Health Care SPRD Fund
XLI Industrial SPDR Fund
XLP Consumer Staples SPDR Fund
XLB Materials SPDR Fund
XLU Utilities SPRD Fund
Can you write Python code for it?

sectors = ['XLY','XLF','XLK','XLE','XLV','XLI','XLP','XLB','XLU']
#train_sectors = DataReader(sectors, 'yahoo', start=train_start_2010, end=train_end_2010)['Adj Close']
#test_sectors  = DataReader(sectors, 'yahoo', start=test_start_2010,  end=test_end_2010)['Adj Close']
train_sectors = pdr.get_data_yahoo(sectors, train_start_2010, train_end_2010)['Adj Close']
test_sectors = pdr.get_data_yahoo(sectors, test_start_2010, test_end_2010)['Adj Close']

# plot normalized prices of these stocks
(train_sectors / train_sectors.iloc[0,:]).plot()

# plot normalized prices of these stocks
(test_sectors / test_sectors.iloc[0,:]).plot()

train_olmar_sectors = algos.OLMAR().run(train_sectors)
train_olmar_sectors.plot(assets=True, weights=False, ucrp=True, bah=True, portfolio_label='OLMAR')

print(train_olmar_sectors.summary())

train_olmar_sectors.plot(assets=False, weights=False, ucrp=True, bah=True, portfolio_label='OLMAR')

test_olmar_sectors = algos.OLMAR().run(test_sectors)
test_olmar_sectors.plot(assets=True, weights=False, ucrp=True, bah=True, portfolio_label='OLMAR')

test_olmar_sectors = algos.OLMAR().run(test_sectors)
test_olmar_sectors.plot(assets=False, weights=False, ucrp=True, bah=True, portfolio_label='OLMAR')


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
OLMAR Starting in 2010
The 2008-2009 recession was unique.  Let's try it all again starting in 2010, with a train set from 2010-2013 inclusive, and a test set of 2014.
Can you write Python code for it?

# set train and test time periods
train_start_2010= datetime(2010,1,1)
train_end_2010 = datetime(2013,12,31)
test_start_2010 = datetime(2014,1,1)
test_end_2010 = datetime(2014,12,31)

# load data from Yahoo
#train_2010 = DataReader(etfs, 'yahoo', start=train_start_2010, end=train_end_2010)['Adj Close']
#test_2010  = DataReader(etfs, 'yahoo', start=test_start_2010,  end=test_end_2010)['Adj Close']
train_2010 = pdr.get_data_yahoo(etfs, train_start_2010, train_end_2010)['Adj Close']
test_2010  = pdr.get_data_yahoo(etfs, train_start_2010, train_end_2010)['Adj Close']

# plot normalized prices of these stocks
(train_2010 / train_2010.iloc[0,:]).plot()

# plot normalized prices of these stocks
(test_2010 / test_2010.iloc[0,:]).plot()

train_olmar_2010 = algos.OLMAR().run(train_2010)
train_crp_2010 = algos.CRP(b=swensen_allocation).run(train_2010)
ax1 = train_olmar_2010.plot(assets=True, weights=False, ucrp=True, bah=True, portfolio_label='OLMAR')
train_crp_2010.plot(ucrp=False, bah=False, weights=False, assets=False, ax=ax1[0], portfolio_label='CRP')

print(train_olmar_2010.summary())

train_olmar_2010.plot_decomposition(legend=True, logy=True)


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Run on the Test Set
Can you write Python code for it?

# create the test set dataframe
olps_test  = pd.DataFrame(index=algo_names, columns=algo_data)
olps_test.algo  = olps_algos

# run all algos
for name, alg in zip(olps_test.index, olps_test.algo):
    olps_test.ix[name,'results'] = alg.run(test)

# Let's make sure the fees are 0 at first
for k, r in olps_test.results.iteritems():
    r.fee = 0.0

# plot as if we had no fees
# get the first result so we can grab the figure axes from the plot
ax = olps_test.results[0].plot(assets=False, weights=False, ucrp=True, portfolio_label=olps_test.index[0])
for k, r in olps_test.results.iteritems():
    if k == olps_test.results.keys()[0]: # skip the first item because we have it already
        continue
    r.plot(assets=False, weights=False, ucrp=False, portfolio_label=k, ax=ax[0])


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
SPY / TLT portfolio comparison
Let's step back and simplify this by looking at OLMAR on a SPY and TLT portfolio.  We should also compare this portfolio to a rebalanced 70/30 mix of SPY and TLT.
Can you write Python code for it?

# load data from Yahoo
#spy_tlt_data = DataReader(['SPY', 'TLT'], 'yahoo', start=datetime(2010,1,1))['Adj Close']
spy_tlt_data = pdr.get_data_yahoo(['SPY', 'TLT'], datetime(2010,1,1))['Adj Close']

# plot normalized prices of these stocks
(spy_tlt_data / spy_tlt_data.iloc[0,:]).plot()

import plotly.plotly as py
import plotly.graph_objs as go

import pandas as pd

TLT = pd.read_csv("stock_datas/TLT.csv")
SPY = pd.read_csv("stock_datas/SPY.csv")
df = pd.concat([TLT,SPY.rename(columns={'date':'datex'})], ignore_index=True)

trace_spy = go.Scatter(
                x=df.Date,
                y=df['SPY.Adjusted'],
                name = "SPY Adjusted",
                line = dict(color = '#17BECF'),
                opacity = 0.8)

trace_tlt = go.Scatter(
                x=df.Date,
                y=df['TLT.Adjusted'],
                name = "TLT Adjusted",
                line = dict(color = '#7F7F7F'),
                opacity = 0.8)

data = [trace_spy,trace_tlt]

layout = dict(
    title = "SPY / TLT portfolio comparison",
    xaxis = dict(
        range = ['2010-01-01','2017-07-17'])
)

fig = dict(data=data, layout=layout)

py.iplot(fig)

spy_tlt_olmar_2010 = algos.OLMAR().run(spy_tlt_data)
spy_tlt_olmar_2010.plot(assets=True, weights=True, ucrp=True, bah=True, portfolio_label='OLMAR')

spy_tlt_olmar_2010.plot_decomposition(legend=True, logy=True)

print(spy_tlt_olmar_2010.summary())

spy_tlt_2010 = algos.CRP(b=[0.7, 0.3]).run(spy_tlt_data)

ax1 = spy_tlt_olmar_2010.plot(assets=False, weights=False, ucrp=True, bah=True, portfolio_label='OLMAR')
spy_tlt_2010.plot(assets=False, weights=False, ucrp=False, bah=False, portfolio_label='CRP', ax=ax1[0])


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
So VNQ (Real Estate) is the big driver after the market crash of 2008, which makes sense.
Let's look at portfolio allocations
Can you write Python code for it?

best_olmar.plot(weights=True, assets=True, ucrp=False, logy=True, portfolio_label='OLMAR')


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
The results now fall, with a Sharpe Ratio below the ~0.5 market Sharpe, and an annualized return that has been cut in half due to fees. It's as if all the trading makes OLMAR underperform for the first 4 years until it can grab some volatility in 2008 to beat UCRP.
Let's look at OLMAR in the test time frame
Can you write Python code for it?

test_olmar = algos.OLMAR(window=5, eps=5).run(test)
#print(train_olmar.summary())
test_olmar.plot(ucrp=True, bah=True, weights=False, assets=False, portfolio_label='OLMAR')


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
VNQ is the big driver of wealth (log scale). Let's test the strategy by removing the most profitable stock and comparing Total Wealth.
Can you write Python code for it?

# find the name of the most profitable asset
most_profitable = best_olmar.equity_decomposed.iloc[-1].argmax()

# rerun algorithm on data without it
result_without = algos.OLMAR().run(train.drop([most_profitable], 1))

# and print results
print(result_without.summary())
result_without.plot(weights=False, assets=False, bah=True, ucrp=True, logy=True, portfolio_label='OLMAR-VNQ')

result_without.plot_decomposition(legend=True, logy=False)


I am working on the file "OLPS_Comparison.ipynb".
The first task is:
Wow, ONS and OLMAR are at the bottom of the list. Remember, we really didn't do any training, but if we had selected ONS or OLMAR at the beginning of 2013 based on past performance, we would not have beat BAH. Hm.
Focusing on OLMAR
Instead of using the default parameters, we will test several window parameters to see if we can get OLMAR to improve.
Can you write Python code for it?

# we need need fewer colors so let's reset the colors_cycle
mpl.rcParams['axes.prop_cycle']= default_color_cycle

train_olmar = algos.OLMAR.run_combination(train, window=[3,5,10,15], eps=10)
train_olmar.plot()

print(train_olmar.summary())

train_olmar = algos.OLMAR.run_combination(train, window=5, eps=[3,5,10,15])
train_olmar.plot()

print(train_olmar.summary())


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
As an example, we consider a harmonic oscillator that is driven by a periodic external force with frequency $ \omega $.
The system is described by the differential equation
$$ \ddot x + k^2 x = F \sin(\omega t) $$
with $ k = \sqrt{D/m} $ where $ D $ is the spring constant and $ m $ mass, furthermore we set $ F = 1 $.
Reformulation as a system of first order is given by
$$ \begin{pmatrix} \dot x_0 \ \dot x_1 \end{pmatrix} = \begin{pmatrix} x_1 \ -k^2 x_0 + \sin(\omega t) \end{pmatrix}. $$
We have to define the function $ f $ for the right hand side in python where we set $ p = \begin{pmatrix} k \ \omega \end{pmatrix} $.
Can you write Python code for it?

def harm_osc(t, x, p):
    k = p[0]
    omega = p[1]
    return np.array([x[1], -k*k*x[0]+np.sin(omega*t)])


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Booleans
Comparison for equality is done with == in python, for numerical datatypes also <, <=, >, >= exist.
Membership in lists and dictionaries can be tested by in:
Can you write Python code for it?

print(3 == 7)
print(7 < 7)
print(7 <= 7)
print(3 in range(0,9))
print(4 in ['apple', 'pear'])


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Booleans can be composed with and and or and negated with not:
Can you write Python code for it?

print(cubesum % 100 == 0 and cubesum < 0)
print(cubesum % 100 == 0 or cubesum < 0)
print(not cubesum in ['apple', 'pear'])


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
First we use $ k = 1, \omega = 2 $ and solve the initial value problem with $ x(0) = 1, \dot x(0) = 0 $ as initial value on the time horizon $ [0, 25] $ and store the trajectory as computed by the initial value solver:
Can you write Python code for it?

result = solve_ivp(harm_osc, np.array([0., 25.]), np.array([1., 0.]), p=np.array([1., 2.]), store_trajectory=True)
print(result.xs)
fig = plt.figure()
plt.plot(result.trajectory_t, result.trajectory_x)
plt.show()


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
For sanity reasons we test first if our implementation has no obvious bugs by comparing with finite differences at a random point.
Can you write Python code for it?

x0 = np.random.randn(4) # x0 is an array with 4 entries chosen from a normal distribution
h = 1e-8                # perturbation in finite differences
# get unit vectors
unit = np.eye(4)
# compute finite difference approximation to gradient in x0
fx0  = ffcn(x0) # store f(x0) to compute it only once
dfx0 = np.array([ffcn(x0+h*unit[ii,:])-fx0 for ii in range(4)])/h # approximation to gradient
print(np.linalg.norm(dfx0 - dfcn(x0)), np.inf)
# compute finite difference approximation to inequality jacobian in x0
cix0  = cfcn_ieq(x0)
jcix0 = (np.array([cfcn_ieq(x0+h*unit[ii,:])-cix0 for ii in range(4)])/h).transpose()
print(np.linalg.norm(jcix0 - jfcn_ieq(x0)), np.inf)
# compute finite difference approximation to equality jacobian in x0
cex0  = cfcn_eq(x0)
jcex0 = (np.array([cfcn_eq(x0+h*unit[ii,:])-cex0 for ii in range(4)])/h).transpose()
print(np.linalg.norm(jcex0 - jfcn_eq(x0)), np.inf)


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
In functions declared with def it is also possible to have optional arguments.
They have to appear after nonoptional arguments:
Can you write Python code for it?

def newton_squareroot(a, x0 = 1.0, tol = 1e-12, itmax = 100):
    """
    Compute square root of a by Newton iteration, with initial guess x0
    """
    x = x0
    it = 0
    while (abs(x*x-a) > tol and it < itmax):
        x = .5*(x+a/x)
        it += 1
    return x, it                 # return the computed square root x and the number of iterations

print(newton_squareroot(3.4))


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
It is sometimes necessary to "vectorize" matrices, forming the vector with all matrix rows appended.
Can you write Python code for it?

vecA = A.reshape(-1) # vectorize A
print(vecA)
reshapedA = vecA.reshape([3,2])
print(reshapedA)     # change shape of A to 3 x 2 instead 3 x 2, order of elements is preserved

# compute trace of matrix
print(B.trace())

# compute sum and norms of vector
print(u'\u03a3\u2096a\u2096 =', a.sum())
print(u'\u2016a\u2016\u2082 =', np.linalg.norm(a))
print(u'\u2016a\u2016\u2081 =', np.linalg.norm(a, 1))
print(u'\u2016a\u2016\u221e =', np.linalg.norm(a, np.inf) )

# solve linear system (backslash in matlab)
np.linalg.solve(B, np.ones([3])) # find x with B*x = [1,1,1]

# compute inverse of A (you should always prefer solve or getting a decomposition)
print(np.linalg.inv(B))


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Iterators
Lists can be used as iterators, some examples:
Can you write Python code for it?

for elem in a:
    print(elem, type(elem))


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Many of the standard functions are available in numpy and vectorized:
They can be applied to a numpy vector and return a vector with the function applied to the components.
This is very efficiently implemented:
Can you write Python code for it?

print(np.sin(a))
print(np.array([np.sin(x) for x in a])) # same result, but slower


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Note that all these datatypes are immutable, this will be explained in the functions paragraph.
Composed datatypes: lists and dictionaries
Furthermore there are the composed datatypes list and dict (short for dictionary).
A list is a ordered collection of arbitrary python objects, a dictionary a mapping between arbitrary python objects (the object on the source side have to be hashable).
In contrast to the previously encountered datatypes, list and dict are mutable, explained in the function section.
Can you write Python code for it?

a = [1, 2, 14.0, 3]                # declaration of a list, note that members of the collection may have different types
print(a[0], a[2], a[-1])           # indexing is zero based: this gives the first, third and last element of the list
print(len(a))                      # len gives the length of a list

b = list(range(10))                # range is a function that constructs a generator giving a list with integers
c = list(range(3,7))
d = [x*x for x in b if x % 2 == 0] # list comprehension is a method to construct new lists out of a given one,
                                   # here all even squares < 100
print(b, c, d)

cd = c + d                         # concatenate lists c and d
print(cd)

print(c)
c.append(5)                        # extend the list c with another element 5
print(c)

e = 'hello world'                  # declares a string which is a list of characters
print(e, e[3])

f = {                              # declare a dictionary with the keys 'algorithm', 'tolerance', 'suspicious iterates'
    'algorithm': 'downhill',
    'tolerance': 1e-6,
    'suspicious iterates': [3,7,11]
}
print(f['algorithm'], f.keys())   # access mapped elements of dictionary by their key, the function keys gives a list of keys


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Note that the second line that constitutes the inner block of the for loop has been indented by 4 characters.
This is one syntax rule in python, which you will also encounter in if-else and while statements:
Blocks are announced by a : and have to be indented. End of indentation indicates end of block. In Java/C++ you would use { to indicate the begin of a block and } to indicate its end.
Can you write Python code for it?

# less elegant, but same functionality as in previous example
for ii in range(len(a)):
    print(a[ii], type(a[ii]))

for key in f:
    print(key, ':', f[key])

# sum all cubes of integers from 0,...,99
cubesum = 0
for ii in range(100):
    cubesum += ii*ii*ii
print(cubesum)


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Now the case of resonance disaster: we set $ k = \omega = 1 $:
Can you write Python code for it?

result = solve_ivp(harm_osc, np.array([0., 25.]), np.array([1., 0.]), p=np.array([1., 1.]), store_trajectory=True)
print(result.xs)
fig = plt.figure()
plt.plot(result.trajectory_t, result.trajectory_x)
plt.show()


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Primitive Datatypes
Similar to Matlab, python uses dynamical typing with implicit type conversion.
Primitive datatypes in python are integers, floating point numbers, boolean and characters.
Here are some examples using these primitive and the implicit type conversion:
Can you write Python code for it?

a = 1           # new integer variable a that is assigned the value 1
b = 2           # new integer variable b that is assigned the value 2
c = 2.0         # new floating point variable c that is assigned the value 2.0
d = True        # new boolean variable d that is assignes the value True

print(b*c)      # automatic type conversion to float

print(a/b)      # what you expect with automatic type conversion, beware: this is different in standard python 2!

print(a//b)     # integer division


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Solving an initial value problems with scipy.integrate
Initial value problems as considered in the lecture are of the form
$$ \dot x(t) = f(t, x(t), p), \quad x(t_0) = x_0 $$
where $ p $ is a parameter.
Scipy provides with ODE a class to solve ordinary differential equations.
The following function provides a convenience interface to solve an initial value problem as specified above:
Can you write Python code for it?

import scipy.integrate

class IVPResult:
    pass

def solve_ivp(f, ts, x0, p=None, integrator='dopri5', store_trajectory=False):
    """
    Solve initial value problem
        d/dt x = f(t, x, p);  x(t0) = x0.

    Evaluate Solution at time points specified in ts, with ts[0] = t0.
    """

    ivp = scipy.integrate.ode(f)
    ivp.set_integrator(integrator)

    if store_trajectory:
        times = []
        points = []
        def solout(t, x):
            if len(times) == 0 or t != times[-1]:
                times.append(t)
                points.append(np.copy(x[:x0.shape[0]]))
        ivp.set_solout(solout)

    ivp.set_initial_value(x0, ts[0])
    ivp.set_f_params(p)

    result = IVPResult()
    result.ts = ts
    result.xs = np.zeros([ts.shape[0], x0.shape[0]])
    result.success = True

    result.xs[0,:] = x0
    for ii in range(1,ts.shape[0]):
        ivp.integrate(ts[ii])
        result.xs[ii,:] = ivp.y[:x0.shape[0]]
        if not ivp.successful():
            result.success = False
            break

    if store_trajectory:
        result.trajectory_t = np.array(times)
        result.trajectory_x = np.array(points)

    return result


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
Solving an optimization problem using the SQP solver SLSQP provided by scipy
We use the problem HS71 from the Hock-Schittkowski collection of optimization problems.
This is given by
$$ \begin{array}{ll} \min_{x \in \mathbb R^4} & x_0 x_3 (x_0+x_1+x_2) \ \text{s.t.} & x_0 x_1 x_2 x_3 -25 \ge 0 \ & \Vert x \Vert_2^2 - 40 = 0 \ & 1 \le x_i \le 5 \quad (i=0,\ldots,3) \end{array}. $$
Now we define functions
- ffcn that computes $ f(x) := x_0 x_3 (x_0+x_1+x_2) + x_2 $,
- dfcn that computes $ \nabla f(x) = \begin{pmatrix} x_3 (x_0+x_1+x_2) + x_0 x_3 \ x_0 x_3 \ x_0 x_3 + 1 \ x_0  (x_0+x_1+x_2) + x_0 x_3 \end{pmatrix} $
- cfcn_ieq that computes $ c_I(x) := x_0 x_1 x_2 x_3 - 25 $
- cfcn_eq that computes $ c_E(x) := \Vert x \Vert_2^2 - 40  $
- jfcn_ieq that computes $ Jc_I(x) = \nabla c_I(x)^T = \begin{pmatrix} x_1 x_2 x_3 & x_0 x_2 x_3 & x_0 x_1 x_3 & x_0 x_1 x_2 \end{pmatrix}$
- jfcn_eq that computes $ Jc_E(x) = \nabla c_E(x)^T = \begin{pmatrix} 2 x_0 & 2 x_1 & 2 x_2 & 2 x_3 \end{pmatrix}$
Can you write Python code for it?

ffcn = lambda x: x[0]*x[3]*np.sum(x[:3]) + x[2]
dfcn = lambda x: np.array([x[3]*np.sum(x[:3])+x[0]*x[3], x[0]*x[3], x[0]*x[3]+1, x[0]*np.sum(x[:3])])
cfcn_ieq = lambda x: np.array([np.prod(x)-25.0])
cfcn_eq = lambda x: np.array([np.inner(x,x)-40])

def jfcn_ieq(x):
    return np.array([
            [np.prod(x[1:]), np.prod(x[[0,2,3]]), np.prod(x[[0,1,3]]), np.prod(x[:3])]
        ])
jfcn_eq = lambda x: np.array([2*x])


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
The resulting deviations from the finite difference approximations give some confidence that we have done a correct implementation of the gradient and jacobian. Let us now solve the optimization problem with initial guess $ x^0 = \begin{pmatrix} 1 \ 5 \ 5 \ 1 \end{pmatrix} $. Using the specified options shows a one line summary per SQP iteration.
Can you write Python code for it?

import scipy.optimize

n  = 4                          # number of variables
m  = 2                          # number of constraints
l  = np.ones([n])               # lower bound variables
u  = 5.0*np.ones([n])           # upper bound variables
x0 = np.array([1., 5., 5., 1.]) # initial guess

result = scipy.optimize.minimize(ffcn, x0, method='SLSQP', jac=dfcn,
                        bounds=[(l[i], u[i]) for i in range(l.shape[0])],
                        constraints=[
                            {'type': 'eq', 'fun': cfcn_eq, 'jac': jfcn_eq},
                            {'type': 'ineq', 'fun': cfcn_ieq, 'jac': jfcn_ieq}
                        ],
                        options={'disp': True, 'iprint': 2})


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
To get the execution time of a code block in python, you can use the time.time() of the time module that gives you the current system time:
Can you write Python code for it?

def fib(n):
    if n in [0, 1]:
        return 1
    return fib(n-1) + fib(n-2)
import time
zerotime = time.time()
print('30th fibonacci number:', fib(30))
executiontime = time.time() - zerotime
print('took {:.2f} s to compute using recursion'.format(executiontime))


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
To verify that the first variant is faster, we use %timeit, that determines the average execution time of a python statement. A command starting with % is a IPython magic command that only works in ipython and ipython notebooks, but not in python scripts.
Can you write Python code for it?

%timeit test = np.sin(np.linspace(0.0, 1.0, 1000))

%timeit test = np.array([np.sin(ii*1e-3) for ii in range(1000)])


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
You can also create a custom vectorized function out of a function that is defined for scalar values:
Can you write Python code for it?

newton_squareroot_vectorized = np.vectorize(newton_squareroot)

print(newton_squareroot_vectorized(np.linspace(1.0, 10.0, 5)))

%timeit test = newton_squareroot_vectorized(np.linspace(1.0, 10.0, 100))

%timeit test = [newton_squareroot_vectorized(x) for x in np.linspace(1.0, 10.0, 100)]


I am working on the file "OODE/.ipynb_checkpoints/introduction_hints-checkpoint.ipynb".
The first task is:
some operations with matrices and vectors
Can you write Python code for it?

# get dimensions
print(a.shape)
print(A.shape)

# transposition
print(A.transpose())
print(A.transpose().shape)

# matrix vector product
print(B.dot(a))

# access first row of matrix A:
print(A[0,:])

# access second column of matrix A:
print(A[:,1])


I am working on the file "OOI/Creating_Geopackage_Layers.ipynb".
The first task is:
Geopandas reading a geopackage
Can you write Python code for it?

import pandas as pd
from geopandas import GeoDataFrame
from shapely.geometry import Point
import matplotlib.pyplot as plt
import geopandas as gpd
import pyproj

print "geopandas version: {}".format(gpd.__version__)

# Test reading geopackage
geopackage = gpd.read_file(os.path.join(pth,fname))

geopackage.head(2)


I am working on the file "OOI/Creating_Geopackage_Layers.ipynb".
The first task is:
Reading csv and printing as dictionary
Can you write Python code for it?

with open(os.path.join(pth,fcsv),'rb') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print row # Notice that numbers are strings in this case


I am working on the file "OOI/OOI_equipment_mapping_Emilio.ipynb".
The first task is:
Testing with vizer google credentials
Can you write Python code for it?

import vizer.tsharvest.util as vhutil

vizer = vhutil.Vizer('nvs', False)

gdoc = get_googlesheet_doc(vizer.vizerspath, "sensor_configurations_mappings")

sheet = gdoc.worksheet('instruments')


I am working on the file "OOI/from_ooi_json-pwrsys-deployment.ipynb".
The first task is:
Add data variables
Can you write Python code for it?

df.columns.tolist()

for c in df.columns:
    if c in ts._nc.variables:
        print("Skipping '{}' (already in file)".format(c))
        continue
    if c in ['time', 'lat', 'lon', 'depth', 'cpm_date_time_string']:
        print("Skipping axis '{}' (already in file)".format(c))
        continue
    print("Adding {}".format(c))
    try:
        ts.add_variable(c, df[c].values)
    except:
        print('skipping, hit object')

df['error_flag1'].dtype.name


I am working on the file "OOI/from_ooi_json-pwrsys-deployment.ipynb".
The first task is:
Create initial file
Can you write Python code for it?

ts = TimeSeries(
    output_directory=odir,
    latitude=44.64,
    longitude=-124.31,
    station_name='ce02shsm',
    global_attributes=global_attributes,
    times=df.time.values.astype(np.int64) // 10**9,
    verticals=df.depth.values,
    output_filename=ofile,
    vertical_positive='down'
)


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
Custom functions
Can you write Python code for it?

remove_extension = lambda x: x.split('.')[0]


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
Generate logs (triplet)
Here we generate four summary files that will be used for QC and place them in the 'output' directory.

mapped.log: The number of mapped reads per cell
extracted.log: The number of remaining reads after filtering for ribosomal DNA and unreadable UMIs
filtered.log: The detailed number of ribosomal DNA extracted per cell
transcript_count.log: The exact number of unique transcript per cell
Can you write Python code for it?

total_cmd       = "grep 'total input reads'      {tagdust_r1}_logfile.txt    | cut -f 2"
extracted_cmd   = "grep 'successfully extracted' {tagdust_r1}_logfile.txt    | cut -f 2"
cleaned_cmd     = "grep 'successfully extracted' {cleaned_reads}_logfile.txt | cut -f 2"
rdna_cmd        = "grep 'ribosomal'              {cleaned_reads}_logfile.txt | cut -f 2"
spikes_cmd      = "grep 'ERCC'                   {cleaned_reads}_logfile.txt | cut -f 2 | paste -s -d+ - | bc"
mapped_cmd      = "{samtools} view -u -f 0x40 {genome_mapped}.bam | {samtools} flagstat - | grep 'mapped (' | cut -f 1 -d ' '"
properpairs_cmd = "{samtools} view -u -f 0x42 {genome_mapped}.bam | {samtools} flagstat - | grep 'mapped (' | cut -f 1 -d ' '"
counts_cmd      = "wc -l {cagescan_fragments}.bed | cut -f 1 -d ' '"

#remove _R1 from the file's name
custom_rename = lambda x: x.replace('_R1', '')

total, cleaned, extracted, rdna, spikes, mapped, properpairs, counts = ([], [], [], [], [], [], [], [])

def run_qc(command, dest, keyword):
    output = subprocess.check_output(command.format(**args), shell=True).strip().decode()
    dest.append([keyword, custom_rename(r1_shortname), output])

for read1 in reads1:
    r1_shortname = remove_extension(os.path.basename(read1))

    args = {'tagdust_r1':         os.path.join('output', 'tagdust_r1',         r1_shortname),
            'genome_mapped':      os.path.join('output', 'genome_mapped',      r1_shortname),
            'cagescan_fragments': os.path.join('output', 'cagescan_fragments', r1_shortname),
            'cleaned_reads':      os.path.join('output', 'cleaned_reads',      r1_shortname),
            'samtools': samtools_path}

    run_qc(total_cmd,       total,       'total')
    run_qc(extracted_cmd,   extracted,   'extracted')
    run_qc(cleaned_cmd,     cleaned,     'cleaned')
    run_qc(rdna_cmd,        rdna,        'rdna')
    run_qc(spikes_cmd,      spikes,      'spikes')
    run_qc(mapped_cmd,      mapped,      'mapped')
    run_qc(properpairs_cmd, properpairs, 'properpairs')
   ...


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
Generate the level1 file
Can you write Python code for it?

root, folders, files = os.walk('./output/genome_mapped/').__next__()
files = [os.path.join(root, f) for f in files if f.endswith('bam')]
level1 = 'level1.py -o output/mylevel1file.l1.osc.gz -f 0x0042 -F 0x0104 --fingerprint {files}'.format(files=' '.join(files))

subprocess.call(level1, shell=True)


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
Parameters
If the required softwares are not in the PATH you can manually set their location here
Can you write Python code for it?

tagdust2_path = 'tagdust'
bwa_path = 'bwa'
samtools_path = 'samtools'
paired_bam_to_bed12_path = 'pairedBamToBed12'
umicountFP_path = 'umicountFP'
syncpairs_path = 'syncpairs'


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
Run the commands for all the pairs
Can you write Python code for it?

for read1, read2 in zip(reads1, reads2):
    args = get_args(read1, read2, ref_genome, output_folders)
    args = dict(args, **softwares)

    for cmd in cmds:
#        print(cmd.format(**args))
        subprocess.call(cmd.format(**args), preexec_fn=lambda: signal.signal(signal.SIGPIPE, signal.SIG_DFL), shell=True)


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
The actual command to run. See the tutorial section for more details about each command
Can you write Python code for it?

cmds = [

    '{tagdust} -t8 -o {tagdust_r1} -1 F:NNNNNNNN -2 S:TATAGGG -3 R:N {r1_input}',

    'gunzip -c {r2_input} > {unzip_r2}.fq',

    '{syncpairs} {tagdust_r1}.fq {unzip_r2}.fq {extracted_r1}.fq {extracted_r2}.fq',

    '{tagdust} -arch SimpleArchitecture.txt -ref ercc_and_human_rRNA_and_tagdust.fa -o {cleaned_reads} {extracted_r1}.fq {extracted_r2}.fq',

    'cp {cleaned_reads}_READ1.fq {cleaned_r1}.fq',

    'cp {cleaned_reads}_READ2.fq {cleaned_r2}.fq',

    '{bwa} aln -t8 {ref_genome} {cleaned_r1}.fq > {r1_sai}.sai',

    '{bwa} aln -t8 {ref_genome} {cleaned_r2}.fq > {r2_sai}.sai',

    '{bwa} sampe -a 2000000 {ref_genome} {r1_sai}.sai {r2_sai}.sai {cleaned_r1}.fq {cleaned_r2}.fq > {sampe}.sam',

    '{samtools} view -uSo - {sampe}.sam | {samtools} sort - -o {genome_mapped}.bam',

    '{samtools} view -f 0x0002 -F 0x0100 -uo - {genome_mapped}.bam | {samtools} sort -n - -o {properly_paired}.bam',

    '{pairedBamToBed12} -i {properly_paired}.bam > {cagescan_pairs}.bed',

    '{umicountFP} -f {cagescan_pairs}.bed > {cagescan_fragments}.bed'

]


I am working on the file "OP-WORKFLOW-CAGEscan-short-reads-v2.0.ipynb".
The first task is:
The name of the output folders for each command
Can you write Python code for it?

output_folders = [ 'tagdust_r1', 'unzip_r2'                    # Demultiplexed R1, unziped R2
                 , 'extracted_r1', 'extracted_r2'              # Synced R1 and R2
                 , 'cleaned_reads', 'cleaned_r1', 'cleaned_r2' # rRNA reads removed
                 , 'r1_sai', 'r2_sai', 'sampe'                 # Intermediate files from BWA
                 , 'genome_mapped', 'properly_paired'          # Final output in BAM format
                 , 'cagescan_pairs', 'cagescan_fragments'      # Final output in BED12 format
                 ]


I am working on the file "OP_high.ipynb".
The first task is:
Operational intensity of  differential operation
We consder  differential operation on a vector $u$ at a given point in 3D with a 1D stencil size $k$ (number of points in the stencil) for every order, the subindex $i$ represent the dimension number $1$ for z, $2$ for $x$ and 3 for $y$,
First order :
$
\frac{d u}{dx_i}
$
Second order :
$
\frac{d^2 u}{dx_i^2}
$
Second order cross derivative
$
\frac{d^2 u}{dx_i dx_j}
$
Can you write Python code for it?

# Arithmetic operations
k = symbols('k')
s = symbols('s')
# 1D stencil
#                multiplication         addition
AI_dxi   =           k + 1           +    k - 1
AI_dxxi  =           k + 1           +    k - 1
AI_dxxij =            2*k            +   2*k-1

# square stencil (all uses the same stencil mask)
#                multiplication         addition
AI_dxis   =          k**2            +    k**2 - 1
AI_dxxis  =          k**2            +    k**2 - 1
AI_dxxijs =          k**2            +    k**2 - 1

# I/O operations
#                    load
IO_dxi   =           k
IO_dxxi  =           k
IO_dxxij =           2*k
IO_square =          k**2

# Operational intensity in single precision

print(AI_dxi/(4*IO_dxi))
print(AI_dxxi/(4*IO_dxxi))
print(AI_dxxij/(4*IO_dxxij))

print(AI_dxis/(4*IO_square))
print(AI_dxxis/(4*IO_square))
print(AI_dxxijs/(4*IO_square))

OI_dxi   = lambdify(k,AI_dxi/(4*IO_dxi))
OI_dxxi  = lambdify(k,AI_dxxi/(4*IO_dxxi))
OI_dxxij = lambdify(k,AI_dxxij/(4*IO_dxxij))

OI_dxis   = lambdify(k,AI_dxis/(4*IO_dxxij))
OI_dxxis  = lambdify(k,AI_dxxis/(4*IO_dxxij))
OI_dxxijs = lambdify(k,AI_dxxijs/(4*IO_dxxij))


I am working on the file "OP_high.ipynb".
The first task is:
Operational intensity of  wave equations
We now consider geophysical wave equations to obtain the theoretical expression of the operational intensity. We write directly the expression of a single time step as a function of differential operators. An operation on a wavefield is counted only once as we consider the minimum of arithmetic operations required.
Acoustic isotropic
$ u(x,y,z,t+dt) = dt^2 v^2(x,y,z) ( 2 u(x,y,z,t) + u(x,y,z,t-dt) + \nabla^2 u(x,y,z,t) +q )  $
VTI
$ p(x,y,z,t+dt) = dt^2 v^2(x,y,z) \left( 2 p(x,y,z,t) + p(x,y,z,t-dt) +(1+2\epsilon)(\frac{d^2 p(x,t)}{dx^2}+\frac{d^2 p(x,t)}{dyx^2}) + \sqrt{(1+2\delta)} \frac{d^2 r(x,t)}{dz^2} + q \right)  $
$ r(x,y,z,t+dt) = dt^2 v^2(x,y,z) \left( 2 r(x,y,z,t) + r(x,y,z,t-dt) +\sqrt{(1+2\delta)}(\frac{d^2 p(x,t)}{dx^2}+ \frac{d^2 p(x,t)}{dy^2}) +  \frac{d^2 r(x,t)}{dz^2} + q \right)   $
TTI
$  p(x,y,z,t+dt) = dt^2 v^2(x,y,z) \left( 2 p(x,y,z,t) + p(x,y,z,t-dt) + (1+2\epsilon) (G_{\bar{x}\bar{x}} + G_{\bar{y}\bar{y}}) p(x,y,z,t) + \sqrt{(1+2\delta)} G_{\bar{z}\bar{z}} r(x,y,z,t) + q \right)  $
$  r(x,y,z,t+dt) = dt^2 v^2(x,y,z) \left( 2 r(x,y,z,t) + r(x,y,z,t-dt) + \sqrt{(1+2\delta)}(G_{\bar{x}\bar{x}} + G_{\bar{y}\bar{y}}) p(x,y,z,t) + G_{\bar{z}\bar{z}} r(x,y,z) +q  \right)  $
where
$
\begin{cases}
G_{\bar{x}\bar{x}} & = cos(\phi)^2 cos(\theta)^2 \frac{d^2}{dx^2} +sin(\phi)^2 cos(\theta)^2 \frac{d^2}{dy^2}+ sin(\theta)^2 \frac{d^2}{dz^2} + sin(2\phi) cos(\theta)^2 \frac{d^2}{dx dy} - sin(\phi) sin(2\theta) \frac{d^2}{dy dz} -cos(\phi) sin(2\theta) \frac{d^2}{dx dz} \
G_{\bar{y}\bar{y}} & =  sin(\phi)^2 \frac{d^2}{dx^2} +cos(\phi)^2  \frac{d^2}{dy^2} - sin(2\phi)^2 \frac{d^2}{dx dy}\
G_{\bar{z}\bar{z}} & = cos(\phi)^2 sin(\theta)^2 \frac{d^2}{dx^2} +sin(\phi)^2 sin(\theta)^2 \frac{d^2}{dy^2}+ cos(\theta)^2 \frac{d^2}{dz^2} + sin(2\phi) sin(\theta)^2 \frac{d^2}{dx dy} + sin(\phi) sin(2\theta) \frac{d^2}{dy dz} +cos(\phi) sin(2\theta) \frac{d^2}{dx dz} \
\end{cases}
$
Can you write Python code for it?

# Arithmetic
#                    dxi             dxxi             dxxij        multiplications     additions   duplicates
AI_acou =        0*AI_dxi    +    3*AI_dxxi   +    0*AI_dxxij    +        3          +     5     -  2 * 2
AI_vti  = 2 * (  0*AI_dxi    +    3*AI_dxxi   +    0*AI_dxxij    +        5          +     5     -  2    )
AI_tti  = 2 * (  0*AI_dxi    +    3*AI_dxxi   +    3*AI_dxxij    +        44         +     17    -  8    )

AI_acoums =        0*AI_dxi  +  3*s*AI_dxxi   +    0*AI_dxxij    +        3*s        +     5*s   -  2 * 2 *s
AI_vtims  = 2 * (  0*AI_dxi  +  3*s*AI_dxxi   +    0*AI_dxxij    +        5*s        +     5*s   -  2*s  )
AI_ttims  = 2 * (  0*AI_dxi  +  3*s*AI_dxxi   +  3*s*AI_dxxij    +        44*s       +     17*s  -  8*s  )

AI_acous =        0*AI_dxis  +    3*AI_dxxis  +    0*AI_dxxijs    +        3          +     5     -  2 * 2
AI_vtis  = 2 * (  0*AI_dxis  +    3*AI_dxxis  +    0*AI_dxxijs    +        5          +     5     -  2 * 2  )
AI_ttis  = 2 * (  0*AI_dxis  +    3*AI_dxxis  +    3*AI_dxxijs    +        44         +     17    -  3*k**2 )

# I/O operations , the full domain of size N is in cache for best case scenario
#
IO_acou   =  4 * N
IO_vti    =  9 * N
IO_tti    =  15 * N

IO_acoums =  3 * s * N + N
IO_vtims  =  6 * s * N + 3 * N
IO_ttims  =  6 * s * N + 9 * N

IO_acous  =  4 * N
IO_vtis   =  9 * N
IO_ttis   =  15 * N

print(simplify(N*AI_acou/(4*IO_acou)))
print(simplify(N*AI_vti/(4*IO_vti)))
print(simplify(N*AI_tti/(4*IO_tti)))

print(simplify(N*AI_acoums/(4*IO_acoums)))
print(simplify(N*AI_vtims/(4*IO_vtims)))
print(simplify(N*AI_ttims/(4*IO_ttims)))

print(simplify(N*AI_acous/(4*IO_acous)))
print(simplify(N*AI_vtis/(4*IO_vtis)))
print(simplify(N*AI_ttis/(4*IO_ttis)))

OI_acou  = lambdify(k,N*AI_acou/(4*IO_acou))
OI_vti   = lambdify(k,N*AI_vti/(4*IO_vti))
OI_tti   = lambdify(k,N*AI_tti/(4*IO_tti))

OI_acoums  = lambdify((k,s),N*AI_acoums/(4*IO_acoums))
OI_vtims   = lambdify((k,s),N*AI_vtims/(4*IO_vtims))
OI_ttims   =...


I am working on the file "OQResultsPath-withDamage_RT_Correlation.ipynb".
The first task is:
Calculate collapse and disruption probability
Can you write Python code for it?

import numpy as np
probColl = []
DamState_array = np.array(DamState_total)
for i in xrange(len(DamState_array[0])):
    sample_coll = np.where(DamState_array[:,i] == 5)
    if len(sample_coll[0]) != 0:
        probColl.append(1)
    else:
        probColl.append(0)

print "Collapse Probability = %f" %(float(sum(probColl))/len(probColl))

probDisr = []
for i in xrange(len(DamState_array[0])):
    sample_coll = np.where(DamState_array[:,i]==5)
    saples_exte = np.where(DamState_array[:,i]==4)
    if len(sample_coll[0]) or len(saples_exte[0]) != 0:
        probDisr.append(1)
    else:
        probDisr.append(0)

print "Disruption Probability = %f" %(float(sum(probDisr))/len(probDisr))


I am working on the file "OQResultsPath-withDamage_RT_Correlation.ipynb".
The first task is:
Calculate repair and disruption time.
Option 1: sum of the repair times for all bridges (consider only one construction team)
Option 2: maximum repair time among all bridges (consider unlimited number of construction teams)
Can you write Python code for it?

#Option 1
RT_final = np.sum(np.array(RT_total),axis=0)
DT_final = np.sum(np.array(DT_total),axis=0)

#Option 2
RT_final = np.max(np.array(RT_total),axis=0)
DT_final = np.max(np.array(DT_total),axis=0)

utils.print_stats(RT_final)
utils.print_stats(DT_final)


I am working on the file "OQResultsPath-withDamage_RT_Correlation.ipynb".
The first task is:
Damage levels calculation - considering damage correlation
Can you write Python code for it?

damage_levels = utils.calculate_damage(gmfs['gmv'], exposure_model, fragility_model)


I am working on the file "OQResultsPath-withDamage_RT_Correlation.ipynb".
The first task is:
Define the correlation coefficient for damage and for repair time
Define the number of samples
Can you write Python code for it?

damCorr = 1
RTCorr = 1
num_dam_samples = 100

DamState_total, RT_total, DT_total = utils.apply_dam_RT_corr(damCorr, RTCorr, num_dam_samples,
                damage_levels, gmfs, fragility_model, exposure_model, timeShinozuka)


I am working on the file "OQResultsPath-withDamage_RT_Correlation.ipynb".
The first task is:
Importing ground motion fields .xml file
Option A: If you only have one file for the ground motion fields (.xml format)
Option B: If you want to combine the gmfs of more than one file using specific weights
Can you write Python code for it?

#Option A
#gmfs_file='/media/sf_Shared_Folder/Paper_Scenarios/2016-09-Revision/Sensitivity_analysis/gmpesLx/out/gmf-rlz-000_134.xml'
#gmfs_file='/media/sf_Shared_Folder/Paper_Scenarios/2016-09-Revision/ScenarioD/out/gmf-rlz-000_90.xml'

#gmfs = utils.read_gmfs_file(gmfs_file, exposure_model)

#Option B
#Atkinson and Boore
gmfs_file1='/media/sf_Shared_Folder/Paper_Scenarios/2016-09-Revision/scenarioA/out/gmf-rlz-000_96.xml'
weight_1 = float(0.7)
#Akkar and Bommer
gmfs_file2 = '/media/sf_Shared_Folder/Paper_Scenarios/2016-09-Revision/scenarioA/out/gmf-rlz-000_92.xml'
weight_2 = float(0.3)

gmfs = utils.combine_gmfs(gmfs_file1, gmfs_file2, exposure_model, weight_1, weight_2)


I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
Get Data
Pull all the reports from the database and convert to individual sentences.
Can you write Python code for it?

if GRABREPORTS:
    conn = sq.connect(os.path.join(RADDIR,"NLP","DBs",
                                   "criticalFindingsAll.sqlite"))
    cursor = conn.cursor()

    r_date  = re.compile(r"""([0-9]{1,2}(/[0-9]{1,2})?/[0-9]{2,4})""")
    r_time = re.compile(r"""([0-9]{1,2}:\d\d(pm|am)?)""")
    cursor.execute("""SELECT rowid,impression from reports""")
    data = [(d[0],d[1].lower()) for d in cursor.fetchall()]
    data =[(d[0],r_date.sub("",r_time.sub("",d[1]))) for d in data]

    reports = [d[1] for d in data]

    def report2sentences(report):
        return TextBlob(report).raw_sentences

    sentences = list(itertools.chain.from_iterable(dview.map_async(report2sentences,reports).get()))


I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
Markup sentences with pyConTextNLP items generated from RadLex
Keep non-zero markups
Can you write Python code for it?

if GRABMARKUPS:
    allMarkups = create_sentence_conTextMarkup_parallel(sentences,
                                           all_items,
                                           mode="target")

    detected_all = [p for p in allMarkups if p.nodes()]
    print(len(detected_all))

    import gzip
    with gzip.open(os.path.join(RADDIR,"NLP","DBs","radlex_found_sentences.pickle"),"wb") as f0:
        pickle.dump(detected_all,f0)
else:
    with gzip.open(os.path.join(RADDIR,"NLP","DBs","radlex_found_sentences.pickle"),"rb") as f0:
        detected_all = pickle.load(f0)


I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
Use the IPython parallel function to speed up the processing
I've had difficulty getting functools partial to build up a robust set of functions with the arguments frozen.
Can you write Python code for it?

def create_sentence_conTextMarkup_parallel(sentences, targets, mode):
    mark_items_in_text = functools.partial(CM.mark_items_in_text,
                                           items=targets,
                                           mode=mode)
    dropMarks = functools.partial(CM.dropMarks, category='Exclusion')
    sm = [CM.create_ConTextMarkup() for i in range(len(sentences))]
    sm = dview.map_async(CM.setRawText, sm, sentences).get()
    sm = dview.map_async(CM.cleanText,sm).get()
    sm = dview.map_async(mark_items_in_text,sm).get()
    sm = dview.map_async(CM.pruneMarks,sm).get()
    sm = dview.map_async(dropMarks,sm).get()
    return sm

    # apply modifiers to any targets within the modifiers scope
    return markup

all_items = []
for key,items in itemData.items():
    all_items.extend(items)

len(all_items)


I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
count = 0
for r in query_results["pathophysiology_metaclass"]:
    print(r[1],r[2])
    count += 1
    if count >=50:
        break
Do some printing for verifying things look as expected
Can you write Python code for it?

for i in itemData['pathophysiology_metaclass'][0:50]:
    print("LITERAL: %s; RE: %s"%(i.literal, i.re))


I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
g=rdflib.Graph()
g.load(os.path.join(RADDIR,"Radlex_3.12.owl"))
Example RadLex queries
qres = g.query("""SELECT DISTINCT ?type WHERE {?s a ?type.}""")
In creating the regular expressions from the RadLex terms, I've learned from sad (and slow) experience the need to have the word boundaries around the phrases.
Can you write Python code for it?

def create_regex(r1,r2):
    if r2:
        return r"\b%s\b|\b%s\b"%(r1,r2)
    else:
        return r"\b%s\b"%r1

if GRABRADLEX:
    g = rdflib.Graph()
    result = g.parse("http://data.bioontology.org/ontologies/RADLEX/download?apikey=8b5b7825-538d-40e0-9e9e-5ab9274a9aeb&download_format=rdf",
                     format="xml")

    query_results = {}
    types = ["anatomy_metaclass","pathophysiology_metaclass","imaging_observation_metaclass"]
    q1 = """

    PREFIX Ontology1447432460: <http://www.owl-ontologies.com/Ontology1447432460.owl#>

    SELECT DISTINCT ?s ?preferred_name ?synonym ?type

    WHERE {

        ?s Ontology1447432460:Preferred_name ?preferred_name.
        ?s rdf:type ?type
        OPTIONAL {?s Ontology1447432460:Synonym ?synonym.}

        FILTER  regex(str(?type), "%s")}"""
    #print(q1)

    for t in types:
        q = q1%t
        print(q)
        query_results[t] = g.query(q)

    for t,r in query_results.items():
        print(t,len(r))
        print("*"*42)

    rslts = g.query(q)

    itemData = {}
    for t in types:
        itemData[t] = [CI.create_ConTextItem([r[1],
                                              r[3].split("#")[1],
                                              create_regex(r[1],r[2]),
                                              ""]) \
                    for r in query_results[t]]
    with open(os.path.join(RADDIR,"NLP","DBs","radlex.pickle"),"wb") as f0:
        pickle.dump(itemData,f0)
else:
    with open(os.path.join(RADDIR,"NLP","DBs","radlex.pickle"),"rb") as f0:
        itemData = pickle.load(f0)


I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
pathoMarkups = create_sentence_conTextMarkup_parallel(sentences,
                                       itemData['pathophysiology_metaclass'],
                                       mode="target")
detected_patho = [p for p in pathoMarkups if p.nodes()]
print(len(detected_patho))
anatMarkups = create_sentence_conTextMarkup_parallel(sentences,
                                       itemData['anatomy_metaclass'],
                                       mode="target")
detected_anat = [p for p in anatMarkups if p.nodes()]
print(len(detected_anat))
Can you write Python code for it?

for i in range(20):
    print(detected_all[i].nodes())
    print(detected_all[i].graph["__text"])

def analyzeReport(report, targets ):
        """
        given an individual radiology report, creates a pyConTextGraph
        object that contains the context markup
        report: a text string containing the radiology reports
        """
        sentences = report2sentences(report)
        markups = list(map(create_sentence_conTextMarkup,sentences, report, targets,[]))
        return markups

c2 = sq.connect(os.path.join(RADDIR,"radlex_terms_expanded.sqlite"))
cu2 = c2.cursor()