I am working on the file "Perceptrons_Basic_Understanding.ipynb".
The first task is:
IMDB Movie Review : Text Classification
Step 1: Download the dataset of IMDB review from TF Dataset library
Can you write Python code for it?

## check if the dataset holds the imdb dataset or not
"imdb_reviews" in ds.list_builders()

## Load the dataset

if "imdb_reviews" in ds.list_builders():
  imdb_ds = ds.load("imdb_reviews")

for key,value in imdb_ds.items():
  print(key, value)

### Using TF DS to split the training data into 60:40 validation dataset.

if "imdb_reviews" in ds.list_builders():

  train_data, validation_data, test_data = ds.load(name="imdb_reviews",split=('train[:60]','train[60:]','test'),as_supervised=True)

type(train_data)

## Explore the data set

train_examples_batch, train_labels_batch = next(iter(train_data.batch(10)))
train_examples_batch

train_labels_batch


I am working on the file "Perceptrons_Basic_Understanding.ipynb".
The first task is:
Step 2 : Building the Model Layers
Can you write Python code for it?

## Embedding the word from TF Hub pre-build BoW

embedding = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1"

hub_layer = hub.KerasLayer(embedding,trainable=True,input_shape=[],dtype=tf.string)

hub_layer(train_examples_batch[:3])

### Define the layers

model = tf.keras.Sequential(name='IMDBSEQModel')
model.add(hub_layer)
model.add(tf.keras.layers.Dense(16, activation='relu'))
model.add(tf.keras.layers.Dense(1))

model.summary()

### Define the Loss Function and optimizer

model.compile(optimizer='adam',loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), metrics=['accuracy'])


I am working on the file "Perceptrons_Basic_Understanding.ipynb".
The first task is:
Step 3 : Train the Model
Can you write Python code for it?

history = model.fit(train_data.shuffle(10000).batch(512), epochs=20, validation_data=validation_data.batch(512), verbose=1)


I am working on the file "Perceptrons_Basic_Understanding.ipynb".
The first task is:
Step 4 : Evaluate the Model
Can you write Python code for it?

results = model.evaluate(test_data.batch(512),verbose=2)

for name, value in zip(model.metrics_names, results):
  print("{} {}".format(name,value))


I am working on the file "Perceptual Uniform Color Interpolation.ipynb".
The first task is:
Conversion from RGB (assumed sRGB) to XYZ
The formulas copied from Wikipedia article on sRGB.
Can you write Python code for it?

Scale = 255.0
srgb = colors_array/Scale
srgb_thres=0.04045
srgb_linear = np.empty(srgb.shape)
srgb_linear[srgb<=srgb_thres] = (srgb/12.92)[srgb<=srgb_thres]
srgb_linear[srgb>srgb_thres] = np.power(((srgb+0.055)/1.055),2.4)[srgb>srgb_thres]
srgb2xyz = np.asarray([ [ 0.4124, 0.3576, 0.1805 ],
                        [ 0.2126, 0.7152, 0.0722 ],
                        [ 0.0193, 0.1192, 0.9505 ] ])
# xyz = srgb2xyz.dot(srgb_linear[i])
xyz = srgb_linear.dot(srgb2xyz.T)
xyz


I am working on the file "Perceptual Uniform Color Interpolation.ipynb".
The first task is:
Conversion from XYZ to RGB (assumed sRGB)
The formulas copied from Wikipedia article on sRGB.
Can you write Python code for it?

xyz2srgb = np.asarray([ [  3.2406, -1.5372, -0.4986 ],
                        [ -0.9689,  1.8758,  0.0415 ],
                        [  0.0557, -0.2040,  1.0570 ] ])
# srgb_linear_interp = xyz2srgb.dot(xyz_interp[i])
srgb_linear_interp = xyz_interp.dot(xyz2srgb.T)
srgb_interp = np.empty(srgb_linear_interp.shape)

srgb_interp[srgb_linear_interp<=0.0031308] = (12.92*srgb_linear_interp)[srgb_linear_interp<=0.0031308]
srgb_interp[srgb_linear_interp>0.0031308] = ((1.055)*np.power(srgb_linear_interp,(1.0/2.4)) - 0.055)[srgb_linear_interp>0.0031308]
srgb_interp[srgb_interp<0.0] = 0.0
srgb_interp[srgb_interp>1.0] = 1.0
srgb_device_interp = np.round(srgb_interp*Scale)
# print(srgb_linear_interp, srgb_interp, srgb_device_interp, colors, sep='\n\n')
srgb_device_interp


I am working on the file "Perceptual Uniform Color Interpolation.ipynb".
The first task is:
Formatting the output
Can you write Python code for it?

# This --- finally --- is the desired output
colors_interpolated = [ 'rgb(%d,%d,%d)' % tuple(c) for c in srgb_device_interp ]
after_hsl_and_back = cl.to_rgb(cl.to_hsl(colors_interpolated))
rgb_interpolated = [ 'rgb(%d,%d,%d)' % tuple(c) for c in rgb_interp ]

print(colors)
HTML(cl.to_html( colors ))

print(colors_interpolated)
print(after_hsl_and_back)
print(rgb_interpolated)
HTML(cl.to_html(after_hsl_and_back))
HTML('
'+
     '
Int. on L*a*b*:
'+
     cl.to_html( colors_interpolated )+'
'+
     '
HSL and back:
'+
     cl.to_html( after_hsl_and_back )+'
'+
     '
Int. on RGB:
'+
     cl.to_html( rgb_interpolated ))


I am working on the file "Perceptual Uniform Color Interpolation.ipynb".
The first task is:
Parsing
Can you write Python code for it?

colors_list = [ re.search('rgb\(([0-9]+),([0-9]+),([0-9]+)\)', c).groups() for c in colors ]
colors_array = np.asarray([ [ float(p) for p in c ] for c in colors_list ])
# DEBUG : forces sRGB correspondence on my hardware
# colors = np.asarray([ [172, 205, 229], [59, 117, 184], [181, 225, 128] ], dtype='float')
colors_array


I am working on the file "PerceptualIntelligence/Cortana Analytics Lab - Face API.ipynb".
The first task is:
Step 2: Define functions
The following five functions allow you to pass your API key and other relevant parameters based on the API reference.
Run the cell to create the various functions.
    * detect(apiKey, imageUrl)
    * createFaceList(apiKey, faceListId)
    * getFaceList(apiKey, faceListId)
    * deleteFaceList(apiKey, faceListId)
    * addFaceToFaceList(apiKey, faceListId, imageUrl)
    * findSimilars(apiKey, faceListId, faceId, numberOfCandidates)
Can you write Python code for it?

def detect(apiKey, imageUrl):
    jsonBody = '{"url":"' + imageUrl + '"}'

    headers = {
        'Content-Type': 'application/json',
        'Ocp-Apim-Subscription-Key': apiKey,
    }

    params = urllib.urlencode({
        'returnFaceId': 'true',
        'returnFaceLandmarks': 'false',
        'returnFaceAttributes': 'age,gender',
    })

    try:
        conn = httplib.HTTPSConnection('api.projectoxford.ai')
        conn.request("POST", "/face/v1.0/detect?%s" % params, jsonBody, headers)
        response = conn.getresponse()
        data = response.read()
        print(data)
        conn.close()
    except Exception as e:
        print("[Errno {0}] {1}".format(e.errno, e.strerror))

    return data

def createFaceList(apiKey, faceListId):
    faceList = faceListId
    jsonBody = '{"name":"' + faceList + '", "userData":"User-provided data attached to the face list"}'

    headers = {
        'Content-Type': 'application/json',
        'Ocp-Apim-Subscription-Key': apiKey,
    }

    params = urllib.urlencode({
    })

    try:
        conn = httplib.HTTPSConnection('api.projectoxford.ai')
        conn.request("PUT", "/face/v1.0/facelists/" + faceList + "?%s" % params, jsonBody, headers)
        response = conn.getresponse()
        data = response.read()
        print(data)
        conn.close()
    except Exception as e:
        print("[Errno {0}] {1}".format(e.errno, e.strerror))

def getFaceList(apiKey, faceListId):
    faceList = faceListId
    jsonBody = ''

    headers = {
        'Ocp-Apim-Subscription-Key': apiKey,
    }

    params = urllib.urlencode({
    })

    try:
        conn = httplib.HTTPSConnection('api.projectoxford.ai')
        conn.request("GET", "/face/v1.0/facelists/" + faceList + "?%s" % params, jsonBody, headers)
        response = conn.getresponse()
        data = response.read()
        print(data)
        conn.close()
    except Exception as e:
        print("[Errno {0}] {1}".format(e.errno, e.strerror))

def deleteFaceList(apiKey, faceListId):
   ...


I am working on the file "PerceptualIntelligence/Cortana Analytics Lab - Face API.ipynb".
The first task is:
Step 4: Detect the face on the initial image
Run the following cell to call the detect method, which returns a face box for the image. Note the values of "top" and "left" in "faceRectangle" and compare that point to the axes on the original image. The face rectangle then defines a box using the appropriate "width" and "height" values starting from that point. In addition, gender and age are displayed as attributes.
Can you write Python code for it?

thumbsUpData = detect(faceApiSubscriptionKey, thumbsUp)


I am working on the file "PerceptualIntelligence/Cortana Analytics Lab - Face API.ipynb".
The first task is:
Step 5: Crop the face
Run the following cell to take the JSON returned by the API and display only the faceRectangle
Can you write Python code for it?

face = json.loads(thumbsUpData)
thumbsUpFaceId = face[0]["faceId"]
faceTop = face[0]["faceRectangle"]["top"]
faceLeft = face[0]["faceRectangle"]["left"]
faceWidth = face[0]["faceRectangle"]["width"]
faceHeight = face[0]["faceRectangle"]["height"]

img = mpimg.imread(thumbsUp)
plt.imshow(img[faceTop:faceTop + faceHeight, faceLeft:faceLeft + faceWidth])


I am working on the file "PerceptualIntelligence/Cortana Analytics Lab - Face API.ipynb".
The first task is:
Step 6: Detect multiple faces in an image
Run the following cells in sequence to display an image with multiple faces, call the API to detect faces, and display the results. Note that while there are three people in the image, the API only returns two faces due to the side-facing orientation of the third person.
Can you write Python code for it?

img = mpimg.imread(multipleFaces)
plt.imshow(img)

multipleFacesData = detect(faceApiSubscriptionKey, multipleFaces)

face = json.loads(multipleFacesData)
faceTop = face[0]["faceRectangle"]["top"]
faceLeft = face[0]["faceRectangle"]["left"]
faceWidth = face[0]["faceRectangle"]["width"]
faceHeight = face[0]["faceRectangle"]["height"]

img = mpimg.imread(multipleFaces)
plt.imshow(img[faceTop:faceTop + faceHeight, faceLeft:faceLeft + faceWidth])

face = json.loads(multipleFacesData)
faceTop = face[1]["faceRectangle"]["top"]
faceLeft = face[1]["faceRectangle"]["left"]
faceWidth = face[1]["faceRectangle"]["width"]
faceHeight = face[1]["faceRectangle"]["height"]

img = mpimg.imread(multipleFaces)
plt.imshow(img[faceTop:faceTop + faceHeight, faceLeft:faceLeft + faceWidth])


I am working on the file "PerceptualIntelligence/Cortana Analytics Lab - Face API.ipynb".
The first task is:
Step 8: Add faces to your Face List
Run the following five cells in sequence to display images of five distinct people and load their face information into your face list. Your faces are referenced in the list using "persistedFaceId".
Can you write Python code for it?

img = mpimg.imread(faceA)
plt.imshow(img)
rawFaceDataA = detect(faceApiSubscriptionKey, faceA)
addFaceToFaceList(faceApiSubscriptionKey, "sample", faceA)

img = mpimg.imread(faceB)
plt.imshow(img)
rawFaceDataB = detect(faceApiSubscriptionKey, faceB)
addFaceToFaceList(faceApiSubscriptionKey, "sample", faceB)

img = mpimg.imread(faceC)
plt.imshow(img)
rawFaceDataC = detect(faceApiSubscriptionKey, faceC)
faceDataC = json.loads(rawFaceDataC)
addFaceToFaceList(faceApiSubscriptionKey, "sample", faceC)

img = mpimg.imread(faceD)
plt.imshow(img)
rawFaceDataD = detect(faceApiSubscriptionKey, faceD)
addFaceToFaceList(faceApiSubscriptionKey, "sample", faceD)

img = mpimg.imread(faceE)
plt.imshow(img)
rawFaceDataE = detect(faceApiSubscriptionKey, faceE)
addFaceToFaceList(faceApiSubscriptionKey, "sample", faceE)


I am working on the file "Perevedentsev/tensorflow.ipynb".
The first task is:
Basic usage
Launch the default graph
To run the matmul op we call the session 'run()' method, passing 'product'
which represents the output of the matmul op.  This indicates to the call
that we want to get the output of the matmul op back.
All inputs needed by the op are run automatically by the session.  They
typically are run in parallel.
The call 'run(product)' thus causes the execution of threes ops in the
graph: the two constants and matmul.
The output of the op is returned in 'result' as a numpy ndarray object.
Can you write Python code for it?

import tensorflow as tf

with tf.Graph().as_default():
    # Create a Constant op that produces a 1x2 matrix.  The op is
    # added as a node to the default graph.
    #
    # The value returned by the constructor represents the output
    # of the Constant op.
    matrix1 = tf.constant([[3., 3.]])

    # Create another Constant that produces a 2x1 matrix.
    matrix2 = tf.constant([[2.],[2.]])

    # Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.
    # The returned value, 'product', represents the result of the matrix
    # multiplication.
    product = tf.matmul(matrix1, matrix2)
    with tf.Session() as sess:
        result = sess.run([product])
        print(result)


I am working on the file "Perevedentsev/tensorflow.ipynb".
The first task is:
MNIST example
<img src="chart3.png" alt="logo" style="width: 100%;"/>
Can you write Python code for it?

import math
import tensorflow.python.platform
import tensorflow as tf
NUM_CLASSES = 10
IMAGE_SIZE = 28
IMAGE_PIXELS = IMAGE_SIZE * IMAGE_SIZE

def inference(images, hidden1_units, hidden2_units):
  # Hidden 1
  with tf.name_scope('hidden1'):
    weights = tf.Variable(
        tf.truncated_normal([IMAGE_PIXELS, hidden1_units],
                            stddev=1.0 / math.sqrt(float(IMAGE_PIXELS))),
        name='weights')
    biases = tf.Variable(tf.zeros([hidden1_units]),
                         name='biases')
    hidden1 = tf.nn.relu(tf.matmul(images, weights) + biases)
  # Hidden 2
  with tf.name_scope('hidden2'):
    weights = tf.Variable(
        tf.truncated_normal([hidden1_units, hidden2_units],
                            stddev=1.0 / math.sqrt(float(hidden1_units))),
        name='weights')
    biases = tf.Variable(tf.zeros([hidden2_units]),
                         name='biases')
    hidden2 = tf.nn.relu(tf.matmul(hidden1, weights) + biases)
  # Linear
  with tf.name_scope('softmax_linear'):
    weights = tf.Variable(
        tf.truncated_normal([hidden2_units, NUM_CLASSES],
                            stddev=1.0 / math.sqrt(float(hidden2_units))),
        name='weights')
    biases = tf.Variable(tf.zeros([NUM_CLASSES]),
                         name='biases')
    logits = tf.matmul(hidden2, weights) + biases
  return logits

def loss_function(logits, labels):
  batch_size = tf.size(labels)
  labels = tf.expand_dims(labels, 1)
  indices = tf.expand_dims(tf.range(0, batch_size), 1)
  concated = tf.concat(1, [indices, labels])
  onehot_labels = tf.sparse_to_dense(
      concated, tf.pack([batch_size, NUM_CLASSES]), 1.0, 0.0)
  cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,
                                                          onehot_labels,
                                                          name='xentropy')
  loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
  return loss

def training(loss, learning_rate):
  tf.scalar_summary(loss.op.name, loss)
  optimizer =...


I am working on the file "Perevedentsev/tensorflow.ipynb".
The first task is:
There is an opportunity to explicitly select computational device for session:
Can you write Python code for it?

with tf.Graph().as_default():
    with tf.Session() as sess:
        with tf.device("/cpu:0"):
            matrix1 = tf.constant([[3., 3.]])
            matrix2 = tf.constant([[2.],[2.]])
            product = tf.matmul(matrix1, matrix2)
            result = sess.run([product])
            print(result)


I am working on the file "Perevedentsev/tensorflow.ipynb".
The first task is:
Variables
Variables maintain state across executions of the graph. The following example shows a variable serving as a simple counter.
Can you write Python code for it?

with tf.Graph().as_default():
    # Create a Variable, that will be initialized to the scalar value 0.
    state = tf.Variable(0, name="counter")

    # Create an Op to add one to `state`.

    one = tf.constant(1)
    new_value = tf.add(state, one)
    update = tf.assign(state, new_value)

    # Variables must be initialized by running an `init` Op after having
    # launched the graph.  We first have to add the `init` Op to the graph.
    init_op = tf.initialize_all_variables()

    # Launch the graph and run the ops.
    with tf.Session() as sess:
        sess.run(init_op)
        print(sess.run(state))
        for _ in range(3):
            sess.run(update)
            print(sess.run(state))


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on data.ipynb".
The first task is:
Actually create the object, give it a data, run anomally detection.
This part can take significant time. It takes 10-30 seconds per hour of data analyzed. Total number of steps will be equal to number of subject intervals in the period tested. For every 5th step and  intervals where anomaly has been detected ROC curve will be shown.
Can you write Python code for it?

ann = ANN(n_series)
ann.set_data(df, auc_df)
ann.loop_over_intervals()


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on data.ipynb".
The first task is:
fix NANs and add accuracy column
Can you write Python code for it?

# full_df.interpolate(method='nearest', axis=0, inplace=True)
df=df.fillna(0)
auc_df = pd.DataFrame(np.nan, index=df.index, columns=['accuracy'])


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on data.ipynb".
The first task is:
parameters to set
Can you write Python code for it?

n_series = 20

start_date = '2017-05-13 00:00:00'
end_date = '2017-05-16 23:59:59'

# tuning parameters
ref = 24
sub = 1
chance = ref/(sub+ref)
cut = chance + (1-chance) * 0.05
print('chance:',chance, '\tcut:', cut)
ref = ref * Hour()
sub = sub * Hour()

srcSiteOWDServer = "128.142.223.247" # CERN site

# destSiteOWDServer = "193.109.172.188"  # pic site


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on data.ipynb".
The first task is:
preselecting X worst links but not really the worst
Can you write Python code for it?

del full_df['134.158.73.243']
means=full_df.mean()
means.sort_values(ascending=False, inplace=True)
means=means[:n_series]
print(means)
df = full_df[means.index.tolist()]
df.shape


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on data.ipynb".
The first task is:
shade regions where an anomaly has been dected
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(20,7))
auc_df['Detected'] = 0
auc_df.loc[auc_df.accuracy>0.05, ['Detected']]=1
ax.plot( auc_df.accuracy,'black')
ax.fill( auc_df.Detected, 'b', alpha=0.3)
ax.legend(loc='upper left')
ax.set_xlim([pd.to_datetime('2017-05-13'),pd.to_datetime('2017-05-17')])
plt.show()
fig.savefig('ANN_shaded_actual_data.png')


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on simulated data.ipynb".
The first task is:
Plots all the series, now it includes AUC values
Can you write Python code for it?

ax = df.plot(figsize=(20,7))
ax.set_xlabel("time", fontsize=14)
plt.savefig('ANN_simulated_score.png')


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on simulated data.ipynb".
The first task is:
Plots auc and shades periods were anomaly has been detected
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(20,7))
ax.set_xlabel("time", fontsize=14)
df.loc[:,'Detected'] = 0
df.loc[df.score>cut,'Detected']=1
df.head()
ax.plot(df.flag, 'r')
ax.plot(df.score,'g')
ax.fill( df.Detected, 'b', alpha=0.3)
ax.legend(loc='upper left')
plt.show()
fig.savefig('ANN_simulated_shaded.png')


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on simulated data.ipynb".
The first task is:
This function actually checks for anomaly in one time window. It receives both referent period and subject period (one under investigation). If splits in samples in training and testing parts, shuffle them and trains model. If anomaly has been detected it plots ROC. It returns both loss and accuracy.
Can you write Python code for it?

def check_for_anomaly(ref, sub, count):

    y_ref = pd.DataFrame([0] * ref.shape[0])
    y_ref.index=ref.index
    X_ref=ref
    del X_ref['flag']
    del X_ref['score']

    y_sub = pd.DataFrame([1] * sub.shape[0])
    y_sub.index=sub.index
    X_sub=sub
    del X_sub['flag']
    del X_sub['score']

    # separate Reference and Subject into Train and Test
    X_ref_train, X_ref_test, y_ref_train, y_ref_test = train_test_split(X_ref, y_ref, test_size=0.3, random_state=42)
    X_sub_train, X_sub_test, y_sub_train, y_sub_test = train_test_split(X_sub, y_sub, test_size=0.3, random_state=42)

    # combine training ref and sub samples
    X_train = pd.concat([X_ref_train, X_sub_train])
    y_train = pd.concat([y_ref_train, y_sub_train])

    # combine testing ref and sub samples
    X_test = pd.concat([X_ref_test, X_sub_test])
    y_test = pd.concat([y_ref_test, y_sub_test])

    X_train = X_train.reset_index(drop=True)
    y_train = y_train.reset_index(drop=True)

    X_train_s, y_train_s = shuffle(X_train, y_train)

    m=getModel()
    hist = m.fit(X_train_s.values, y_train_s.values, epochs=epochs, verbose=0, shuffle=True, batch_size=256)

    loss_and_metrics = m.evaluate(X_test.values, y_test.values)#, batch_size=256)
    #print(loss_and_metrics)

    if loss_and_metrics[1] > cut:# or not count%5:
        plotHist(hist)

    return loss_and_metrics[1]


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on simulated data.ipynb".
The first task is:
create NN
Can you write Python code for it?

def getModel():
    model = Sequential()
    model.add(Dense(units=n_series, input_shape=(n_series,), activation='relu' ))
#         model.add(Dropout(0.5))
    model.add(Dense(units=n_series, activation='relu'))
#         model.add(Dropout(0.5))
    model.add(Dense(units=1, activation='sigmoid') )

    model.compile(loss='binary_crossentropy',optimizer='rmsprop', metrics=['accuracy'])
#         model.compile(loss='hinge', optimizer='sgd', metrics=['binary_accuracy'])
#         model.compile(loss='mse',optimizer='rmsprop', metrics=['accuracy'])
#         model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['binary_accuracy'])

    return model

def plotHist(hist):
    es=len(hist.history['loss'])
    x = np.linspace(0,es-1,es)
    plt.plot(x, hist.history['loss'], '--', linewidth=2, label='loss')
    plt.plot(x, hist.history['acc'], '-', linewidth=2, label='acc')
    plt.legend()
    plt.show()


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on simulated data.ipynb".
The first task is:
generate normal data
Can you write Python code for it?

df =  simulated_data.get_simulated_data()
# df =  simulated_data.get_simulated_fixed_data()
df.head()


I am working on the file "PerfSONAR/AnomalyDetection/ANN/Testing NN AD on simulated data.ipynb".
The first task is:
plot timeseries
Can you write Python code for it?

ax = df.plot(figsize=(20,7))
ax.set_xlabel("time", fontsize=14)


I am working on the file "PerfSONAR/AnomalyDetection/BDT/Testing BDT AD on simulated data.ipynb".
The first task is:
Looping over time intervals
This function actully runs anomaly dection over all the intervals. It takes only few seconds per interval, but plot generation takes 10-20 seconds.
Can you write Python code for it?

df['auc_score']=0.5

#find min and max timestamps

start = df.index.min()
end = df.index.max()

#round start
start.seconds=0
start.minutes=0

ref = window * Hour()
sub = 1 * Hour()

# loop over them
ti=start+ref+sub
count=0
while ti < end + 1 * Minute():
    ref_start = ti-ref-sub
    ref_end = ti-sub
    ref_df = df[(df.index >= ref_start) & (df.index < ref_end)]
    sub_df = df[(df.index >= ref_end) & (df.index < ti)]
    auc_score = check_for_anomaly(ref_df, sub_df)
    df.loc[(df.index>=ref_end) & (df.index<=ti),['auc_score']] = auc_score
    print(ti,"\trefes:" , ref_df.shape[0], "\tsubjects:", sub_df.shape[0], '\tauc:', auc_score)
    ti = ti + sub
    count=count+1
    #if count>2: break

ax = df.plot(figsize=(20,7))
ax.set_xlabel("time", fontsize=14)
plt.savefig('BDT_simulated_fixed.png')


I am working on the file "PerfSONAR/AnomalyDetection/BDT/Testing BDT AD on simulated data.ipynb".
The first task is:
functions to check for anomaly and plot roc curves.
check_for_anomaly fuction receives both reference and subject dataframes, creates training and testing frames, does classification, tests it, prints auc results, and creates ROC curves when abouve the cut.
No outut is expected form this cell.
Can you write Python code for it?

def check_for_anomaly(ref, sub):

    y_ref = pd.Series([0] * ref.shape[0])
    X_ref = ref
    del X_ref['flag']
    del X_ref['auc_score']

    y_sub = pd.Series([1] * sub.shape[0])
    X_sub=sub
    del X_sub['flag']
    del X_sub['auc_score']

    # separate Reference and Subject into Train and Test
    X_ref_train, X_ref_test, y_ref_train, y_ref_test = train_test_split(X_ref, y_ref, test_size=0.3, random_state=42)
    X_sub_train, X_sub_test, y_sub_train, y_sub_test = train_test_split(X_sub, y_sub, test_size=0.3, random_state=42)

    # combine training ref and sub samples
    X_train = pd.concat([X_ref_train, X_sub_train])
    y_train = pd.concat([y_ref_train, y_sub_train])

    # combine testing ref and sub samples
    X_test = pd.concat([X_ref_test, X_sub_test])
    y_test = pd.concat([y_ref_test, y_sub_test])

    clf = AdaBoostClassifier() #dtc
#     clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),algorithm="SAMME",n_estimators=200)

    #train an AdaBoost model to be able to tell the difference between the reference and subject data
    clf.fit(X_train, y_train)

    #Predict using the combined test data
    y_predict = clf.predict(X_test)

    # scores = cross_val_score(clf, X, y)
    # print(scores)

    fpr, tpr, thresholds = roc_curve(y_test, y_predict) # calculate the false positive rate and true positive rate
    auc_score = auc(fpr, tpr) #calculate the AUC score
    print ("auc_score = ", auc_score, "\tfeature importances:", clf.feature_importances_)

    if auc_score > cut:
        plot_roc(fpr, tpr, auc_score)
        filename='tree_'+sub.index.min().strftime("%Y-%m-%d_%H")
        tree.export_graphviz(clf.estimators_[0] , out_file=filename +'_1.dot')
        tree.export_graphviz(clf.estimators_[1] , out_file=filename +'_2.dot')

    return auc_score

def plot_roc(fpr,tpr, roc_auc):
    plt.figure()
    plt.plot(fpr, tpr, color='darkorange', label='ROC curve (area = %0.2f)' % roc_auc)
    plt.plot([0, 1], [0, 1], color='navy',...


I am working on the file "PerfSONAR/AnomalyDetection/BDT/Testing BDT AD on simulated data.ipynb".
The first task is:
generate data
Can you write Python code for it?

# df =  simulated_data.get_simulated_data()
df =  simulated_data.get_simulated_fixed_data()
df.head()


I am working on the file "PerfSONAR/AnomalyDetection/BDT/Testing BDT AD on simulated data.ipynb".
The first task is:
make plot of created anomalies, auc values, and shade periods where anomaly is suspected.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(20,7))
ax.set_xlabel("time", fontsize=14)
df.loc[:,'Detected'] = 0
df.loc[df.auc_score>0.55,'Detected']=1
df.head()
ax.plot(df.flag, 'r')
ax.plot(df.auc_score,'g')
ax.fill( df.Detected, 'b', alpha=0.3)
ax.legend(loc='upper left')
plt.show()
fig.savefig('BDT_shaded_simulated_fixed.png')


I am working on the file "PerfSONAR/AnomalyDetection/BDT/Testing BDT AD on simulated data.ipynb".
The first task is:
parameters to set
Can you write Python code for it?


I am working on the file "PerfSONAR/AnomalyDetection/BDT/Testing BDT AD on simulated data.ipynb".
The first task is:
plot timeseries. Can take a minute to appear due to plot size and complexity.
Can you write Python code for it?

ax = df.plot(figsize=(20,7))
ax.set_xlabel("time", fontsize=16)
plt.savefig('simulated_fixed.png')


I am working on the file "PerfSONAR/AnomalyDetection/SVC/OneClass SVC/One-class+SVM+with+non-linear+kernel+%28RBF%29.ipynb".
The first task is:
generate training data
Can you write Python code for it?

X = 0.3 * np.random.randn(100,2)
X_train = np.r_[X + 2, X - 2]
# Generate some regular novel observations
X = 0.3 * np.random.randn(20, 2)
X_test = np.r_[X + 2, X - 2]
# Generate some abnormal novel observations
X_outliers = np.random.uniform(low=-4, high=4, size=(20, 2))


I am working on the file "PeriodWindowSimulation.ipynb".
The first task is:
As a test, we shall see what the cumulative distribution of periods looks like.
Can you write Python code for it?

d = DrawPeriods.large_neptunes()
periods = [d.draw() for _ in range(1000)]
_, _, lines = plt.hist(periods, bins=100, normed=True, cumulative=True, histtype='step', lw=2., color=colours[0])
plt.ylim(0, 1)
plt.twinx()
prob_dist = plt.plot(d.periods, d.values / d.values.max(), color=colours[1])
plt.ylim(0, 1)
plt.legend([lines[0], prob_dist[0]], ['Planet distribution', 'Probability distribution'], loc='best')
None


I am working on the file "PeriodWindowSimulation.ipynb".
The first task is:
Drawing planets
Now we can start drawing planets and determining if they are detectable. First we define the "is visible" function:
Can you write Python code for it?

def separation_from_period(period):
    period = (period * u.d).to(u.s)
    mass = (1. * u.astrophys.M_sun).to(u.kg)
    norm = ((const.G * mass / (4. * np.pi ** 2))) ** (1. / 3.)
    return (norm * period ** (2. / 3.)).to(u.m)

def transit_probability(period):
    radius = 1. * u.astrophys.R_sun.to(u.m)
    return (radius / separation_from_period(period)).value

def compute_twighlight_limits(dt):
    observer = ephem.Observer()
    observer.date = datetime.datetime(dt.datetime.year,
                                      dt.datetime.month,
                                      dt.datetime.day,
                                      0, 0, 1)
    observer.lat = site_coordinates.latitude.value
    observer.long = site_coordinates.longitude.value
    observer.horizon = '-18'

    return (observer.previous_rising(ephem.Sun(), use_center=True).datetime(),
            observer.next_setting(ephem.Sun(), use_center=True).datetime())

def transit_is_visible(dt, sky_position, minimum_elevation=30. * u.degree):
    night = dt.datetime.date()

    if night not in valid_dates:
        return False

    twighlight_limits = compute_twighlight_limits(dt)
    if dt < twighlight_limits[0] or dt > twighlight_limits[1]:
        return False

    if altitude(sky_position) < minimum_elevation:
        return False

    return True

def run_simulation(index):
    periods, fraction = [], []
    i = 0
    while True:
        period = d.draw()
#         if np.random.uniform() > transit_probability(period):
#             continue

        sky_position = coord.SkyCoord(np.random.choice(ra_choices), np.random.choice(dec_choices), unit=(u.degree, u.degree))
        epoch = time.Time(np.random.uniform(start_jd, end_jd), format='jd')

        n_visible_transits = 0
        for transit in np.arange(365.25 / period):
            midpoint = epoch + time.TimeDelta(transit * period, format='jd')
            if transit_is_visible(midpoint, sky_position):
                n_visible_transits += 1

        if...


I am working on the file "PeriodWindowSimulation.ipynb".
The first task is:
Now a class to draw periods, as the setup is relatively expensive.
Can you write Python code for it?

class DrawPeriods(object):

    def __init__(self, values):
        self.periods = np.asarray([2, 3.4, 5.9, 10, 17])
        self.values = values
        self.interp = interpolate.interp1d(self.values, self.periods)

    def draw(self):
        chosen_value = np.random.uniform(self.values.min(), self.values.max())
        return float(self.interp(chosen_value))

    @property
    def min_period(self):
        return self.periods.min()

    @property
    def max_period(self):
        return self.periods.max()

    @classmethod
    def large_neptunes(cls):
        values = np.asarray([0.004, 0.01, 0.12, 0.21, 0.49]) / 100.
        return cls(values)

    @classmethod
    def small_neptunes(cls):
        values = np.asarray([0.035, 0.22, 0.95, 2.88, 6.55]) / 100.
        return cls(values)


I am working on the file "PeriodWindowSimulation.ipynb".
The first task is:
Save the results for later analysis
Can you write Python code for it?

out = np.array(list(zip(periods, fraction)), dtype=[('periods', float), ('fraction', float)])
np.save('results', out)


I am working on the file "PeriodWindowSimulation.ipynb".
The first task is:
These are a series of delta functions, so we need to bin up in the x axis. We also need to normalise by the number of planets in each period window.
Can you write Python code for it?

by, bx, _ = stats.binned_statistic(np.log10(periods), fraction, bins=25)
norm, _, _ = stats.binned_statistic(np.log10(periods), periods, bins=bx)
by = by.astype(float) / norm
by[by != by] = 0.


I am working on the file "Periodic_Trends.ipynb".
The first task is:
PART 1: Trends Related to Atomic Number

Use and modify the section of code below to answer questions 3-5 in your coding booklet.
Can you write Python code for it?

ax = data.plot('AtomicNumber', 'AtomicMass', title="Trends Related to Atomic Number", legend=False)
ax.set(xlabel="Atomic Number", ylabel="Comparison Factor")


I am working on the file "Periodic_Trends.ipynb".
The first task is:
PART 2: The Periodicity of Element Properties

Use and modify the section of code below to answer questions 6 & 7 in your coding booklet.
Can you write Python code for it?

ax = data.plot('AtomicNumber', 'BoilingPoint', title="Looking for Periodicity of Properties", legend=False)
ax.set(xlabel="Atomic Number", ylabel="Comparison Factor")


I am working on the file "Periodic_Trends.ipynb".
The first task is:
PART 3: Unstructured Coding

Use and modify the section of code below to answer questions 8-11 in your coding booklet.
Can you write Python code for it?

data.Radioactive.count()

# Set variables for scatter plot
x = data.Group
y = data.NumberofValence

plt.scatter(x,y)
plt.title('Looking For Patterns')
plt.xlabel('x-axis')
plt.ylabel('y-axis')

#this sets the interval on the x-axis
plt.xticks(np.arange(min(x), max(x)+1, 1.0))

# This actually shows the plot
plt.show()

data[['AtomicNumber', 'Element', 'Type']].sort_values(by='AtomicNumber')


I am working on the file "Permutations.ipynb".
The first task is:
Before diving deeper into Permutations, a couple helper functions will come in handy, that of gcd (greatest common divisor) and lcm (lowest common multiple).  These two are related.  Furthermore, we would like versions which tackle any number of integer inputs, which we'll call GCD and LCD respectively.
Can you write Python code for it?

def gcd(a, b):
    while b:
        b, a = a % b, b
    return a

def lcm(a, b):
    return int((a * b)/gcd(a, b))

def GCD(*terms):
    return reduce(gcd, terms)

def LCM(*terms):
    return reduce(lcm, terms)

GCD(25, 50, 15)

LCM(25, 50, 15)

class P:
    """
    A Permutation

    self._code: a dict, is a mapping of iterable elements
    to themselves in any order.

    start out with Identity, or directly inject the mapping as
    a dict or use an inversions table to construct the permutation
    """

    def __init__(self,
        the_code = None,   # direct inject
        inv_table = None,  # construct
        iterable = ascii_lowercase + ' '): # default domain

        if the_code:
            self._code = the_code

        elif inv_table:
            values = []
            for key in sorted(inv_table, reverse=True):
                if inv_table[key] >= len(values):
                    values.append(key)
                else:
                    values.insert(inv_table[key], key)
            self._code = dict(zip(sorted(inv_table), values))

        elif iterable:
            try:
              # create two iterators for zipping together
              iter1 = iter(iterable)
              iter2 = iter(iterable)
            except:
                raise TypeError

            self._code = dict(zip(iter1, iter2))

    def shuffle(self):
        """
        return a random permutation of this permutation
        """
        # use shuffle
        # something like
        the_keys = list(self._code.keys()) # grab keys
        shuffle(the_keys)  # shuffles copied one
        newP = P()
        # old keys point to new ones
        newP._code = dict(zip(self._code.keys(), the_keys))
        return newP

    def encrypt(self, plain):
        """
        turn plaintext into cyphertext using self._code
        """
        output = ""  # empty string
        for c in plain:
            output = output + self._code.get(c, c)
        return...


I am working on the file "Permutations.ipynb".
The first task is:
Finally, we should talk about the order and the parity of a permutation.
Permutations define subgroups with a specific number of unique elements, which we may define as the order of that permutation.  A permutation raised to its own order, as a power, cycles around to where it started, meaning the resulting permutation is the identity permutation.
The sign of a permutation depends on the parity (even or odd) of the number of transpositions it would take, to express the same permutation.  An even number of transpositions or no transpositions at all, have a sign of positive 1.  Odd numbers of transpositions define permutations of negative sign which is likewise expressed as -1.
The identity permutation is even (no transpositions, or an even number that end up self canceling) and only the set of even permutations defines a subgroup of the total space.  The odd permutations define a coset in the total space.
Can you write Python code for it?

p = P().shuffle()
order = len(p)
sign = p.sgn()
print("Perm:", p._code)
print("Order:", order)
print("Sign:", sign)
try:
    inv_p = ~p
    assert p.sgn() == inv_p.sgn()
    assert P().sgn() == 1 # identity is even
    print("Seventh Test Succeeds")
except AssertionError:
    print("Seventh Test Fails")


I am working on the file "Permutations.ipynb".
The first task is:
In this first test, lets make sure that any random permutation, times its inverse, gives the Identity permutation...
Can you write Python code for it?

p = P() # identity permutation
print(dir(p))
p.shuffle()
#new_p = p.shuffle()
#inv_p = ~new_p
try:
    #assert p == inv_p * new_p   # should be True
    print("First Test Succeeds")
except AssertionError:
    print("First Test Fails")


I am working on the file "Permutations.ipynb".
The first task is:
Inversion Table Notation
"The inversion table B1 B2 ... Bn of the permutation A1, A2 ... An is obtained by letting Bj be the number of elements to the left of j that are greater than j." -- Donald E. Knuth
Can you write Python code for it?

knuth = {1:5, 2:9, 3:1, 4:8, 5:2, 6:6, 7:4, 8:7, 9:3} # vol 3 pg. 12
expected = {1:2, 2:3, 3:6, 4:4, 5:0, 6:2, 7:2, 8:1, 9:0} # Ibid
k = P(the_code=knuth)
try:
    assert k.inversion_table() == expected
    print("Fourth Test Succeeds")
except AssertionError:
    print("Fourth Test Fails")


I am working on the file "Permutations.ipynb".
The first task is:
Lets make sure that, given the derived inversion table for the Permutation named knuth, we're able to regenerate knuth upon initialization...
Can you write Python code for it?

p = P(inv_table = expected)
try:
    assert p == k
    print("Fifth Test Succeeds")
except AssertionError:
    print("Fifth Test Fails")


I am working on the file "Permutations.ipynb".
The first task is:
Next, given any random permutation, lets make sure we're able to reconstruct it from its inversion table.  The only difference from the test above is we're using a random permutation versus the one given on page 12 in volume 3 of The Art of Computer Programming.
Can you write Python code for it?

p = P().shuffle()
inv = p.inversion_table()
print("Perm:", p._code)
print("Inv table:", inv)
new_p = P(inv_table = inv)
try:
    assert p == new_p
    print("Sixth Test Succeeds")
except AssertionError:
    print("Sixth Test Fails")


I am working on the file "Permutations.ipynb".
The first task is:
Since it makes sense to both power, and invert, we may raise permutations to a negative power. (p ** -1) is another way of saying "give me the inverse"...
Can you write Python code for it?

p = P().shuffle()
try:
    assert p ** -1 == ~p
    assert p ** -2 == ~(p * p)
    assert p ** -2 == (~p * ~p)
    print("Second Test Succeeds")
except AssertionError:
    print("Second Test Fails")


I am working on the file "Permutations.ipynb".
The first task is:
That's why multiplying a permutation with its own inverse is the same as "doing nothing" just as the identity permutation does.  Any permutation multiplied by I (the identity) results in the same thing.
Can you write Python code for it?

I = P()  # without shuffling, P() returns I
assert I == ~I
q = P().shuffle() # any perm
assert I * q == q == q * I


I am working on the file "Permutations.ipynb".
The first task is:
What you see above is operator overloading.  Even if you're maybe glazing over as to what it means to invert, or to multiply, what's clear is these to operators have meaning, are being used.
The inverted permutation is a reverse dictionary, undoing the swap with a path back to the start.  'a' maps to 's' but then, in the inverse code, 's' maps right back to 'a'.
To multiply is like to compose two functions, in that for A times B, each key in A points ends up pointing to the corresponding B[A[key]].
Can you write Python code for it?


I am working on the file "Person marking.ipynb".
The first task is:
1 entity referent

self ("me")
addressee ("you here")
other ("somebody else")

2+ entity referent

self, addressee ("me and you here" / inclusive we)
self, other ("me and somebody else" / exclusive we)
addressee, addressee ("the two or more of you here")
addressee, other ("one of you here and somebody else")
other, other ("the two or more of them")

3+ entity referent

self, addressee, addressee ("me and the two or more of you here")
self, addressee, other ("me, one of you here, and somebody else")
self, other, other ("me and two or more other people")
addressee, addressee, other ("the two or more of you and somebody else")
addressee, other, other ("one of you and two or more other people")

4+ entity referent

self, addressee, addressee, other ("me, the two or more of you here, and somebody else")
self, addressee, other, other ("me, one of you here, and two or more other people")
addressee, addressee, other, other ("the two or more of you here and two or more other people")

5+ entity referent

self, addressee, addressee, other, other ("me, the two or more of you here, and two or more other people")

There are 17 possible markers if there's no distinction between 2 entities of the same type and 3+ entities of the same type.
* a dual or trial entity number could be added to have a 3-way distinction between e.g. [other, other] and [other, other, other]
* another entity category besides self, addressee, and other could be added (invisible/divine entities)
* multiple self referents could be included (choral we)
Also, what about the issue of mis-identifying the cue as "self" rather than "addressee" (kids calling themselves "you")?
Can you write Python code for it?

from itertools import combinations, combinations_with_replacement

referents = []

for i in xrange(1, len(entities) * 2):
    for combo in combinations_with_replacement(entities, i):

        # choral we is impossible
        if combo.count('self') > 1:
            continue

        # only singular vs plural
        if combo.count('addressee') > 2:
            continue

        if combo.count('other') > 2:
            continue

        # compound cues
        referent = list(combo)

        for j in xrange(2, len(combo) + 1):
            for compound in combinations(combo, j):

                if compound not in referent:
                    referent.append(compound)

        referents.append(referent)

len(referents)

referents


I am working on the file "Person marking.ipynb".
The first task is:
Assume that the distribution of referent sets is uniform, which is probably not true.
Can you write Python code for it?

import numpy

def sampler(p):

    def uniform():
        return numpy.random.choice(p)

    return uniform

referent_sampler = sampler(len(data))

import ndl

def activation(W):
    return pandas.DataFrame([ndl.activation(c, W) for c in data.Cues], index=data.index)

W = ndl.rw(data, M=100, distribution=referent_sampler)
A = activation(W)
A

pandas.DataFrame([data['Outcomes'], A.idxmax(1), A.idxmax(1) == data['Outcomes']],
                 index = ['Truth', 'Prediction', 'Accurate?']).T


I am working on the file "Person marking.ipynb".
The first task is:
Spoken English collapses these to 6 possibilities: I, you, s/he, we, you guys, they
Can you write Python code for it?

def english(referents):
    # first-person
    if 'self' in referents:

        if 'addressee' in referents: # inclusive we
            # doesn't matter who else is being referred to
            return 'we'

        if 'other' in referents: # exclusive we
            # doesn't matter who else is being referred to
            return 'we'

        return 'I'

    # second-person, if the speaker isn't included
    elif 'addressee' in referents:

        if referents.count('addressee') > 1: # inclusive you
            return 'you guys'

        if 'other' in referents: # exclusive you
            return 'you guys'

        return 'you'

    # third-person, if the addressee isn't included either
    elif 'other' in referents:

        if referents.count('other') > 1:
            return 'they'

        return 's/he'

english(['self', 'addressee'])

english(['self', 'other'])

english(['addressee', 'other'])

english(['addressee', 'addressee']) # also ('addressee', 'addressee') compound

import pandas

data = pandas.DataFrame()

data['Cues'] = referents
data['Outcomes'] = [english(referent) for referent in referents]
data


I am working on the file "Person marking.ipynb".
The first task is:
With 100 trials, the learner is getting a lot of them right, but just by predicting 'you guys' or 'we' (if self is a referent) all of the time, since those cover most of the referent sets.
Can you write Python code for it?

import sim

english_learning = sim.Simulation(english, data, referent_sampler, 2000)

import matplotlib.pyplot as plt
%matplotlib inline

trajectory = [english_learning.accuracy(i) for i in xrange(1, english_learning.MAX_M)]

plt.plot(range(1, len(trajectory) + 1), trajectory, '-')
plt.xlabel('Trial Number')

%load_ext rpy2.ipython

%Rpush trajectory

%%R

trajectory = data.frame(trial=1:length(trajectory), learned=trajectory)

library('ggplot2')

ggplot(trajectory, aes(trial, learned)) +
    geom_point(alpha=0.25) +
    stat_smooth() +
    coord_cartesian(ylim=c(0,1))


I am working on the file "Perturbation methods.ipynb".
The first task is:
<div class="alert alert-warning">

**Note:** This notebook requires SymPy 1.5 to work.
</div>

Consider the following system
$$\ddot{u} + 4 u + \varepsilon u^2 \ddot{u} = 0 \enspace .$$
This system can be rewritten as
$$(1 + \varepsilon u^2)\ddot u + 4u = 0 \enspace ,$$
or
$$\ddot u + \frac{4u}{1 + \varepsilon u^2} = 0 \enspace .$$
As a first order system it reads
$$\begin{pmatrix}
\dot u \
\dot v
\end{pmatrix} = \begin{pmatrix}
v \
-\frac{4u}{1 + \varepsilon u^2}
\end{pmatrix} \enspace ,$$
with Jacobian matrix
$$J(u,v) = \begin{bmatrix}
0 & 1 \
-\frac{4(1 - \varepsilon u^2)}{(1+ \varepsilon u^2)^2} & 0
\end{bmatrix} \enspace .$$
This system has a fixed point in $(0,0)$, with eigenvalues
$$\lambda_1 = -2i,\quad \lambda_2 = 2i \enspace ,$$
and we can conclude that this fixed point is a center.
Can you write Python code for it?

eq = (1 + eps*u(t)**2)*diff(u(t), t, 2) + omega**2*u(t)
eq

ode_order(eq, u)


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
2D Poisson Problem
We solve the following problem:
\begin{equation}
\frac{\partial u^2}{\partial^2 x} + \frac{\partial u^2}{\partial^2 y} = -8\pi^2\cos{(2\pi x)}\cos{(2\pi y)}
\end{equation}
with boundary conditions:
\begin{equation}
\left.\frac{\partial u}{\partial x}\right|{x=0}=\left.\frac{\partial u}{\partial x}\right|{x=1}=\left.\frac{\partial u}{\partial y}\right|{y=0}=\left.\frac{\partial u}{\partial y}\right|{y=1}=0
\end{equation}
The exact solution is
\begin{equation}
u(x, y) = \cos{(2\pi x)}\cos{(2\pi y)}
\end{equation}
Test set 1:
Number of GPU: 1 (K40)
Machine: Theo
Top solver:

Preconditioned CG (PCG)
Flexible Preconditioned CG (PCGF)
Preconditioned Stable BiCG (PBICGSTAB)

Tolerance: absolute residual reach $10^{-12}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: AHAT
Smoother: Block Jocabi
Coarsest solver: Block Jacobi
Cycle: V
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 1
Maximum size of coarsest grid: 100
Relaxation factor of the block Jacobi: from 0.1 to 1
Grid size: 3750 $\times$ 3750
Can you write Python code for it?

omg=numpy.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1])
tPCG = numpy.array([5.72, 4.54, 3.78, 3.14, 2.71, 2.38, 2.06, 1.95, 2.49, 10.15])
tPCGF = numpy.array([2.48, 2.14, 2.03, 2.6, 10.7])
tPBICGSTAB = numpy.array([2.79, 2.58, 2.48, 3, 12.1])

pyplot.plot(omg, tPCG, label="PCG")
pyplot.plot(omg[5:], tPCGF, label="PCGF")
pyplot.plot(omg[5:], tPBICGSTAB, label="PBICGSTAB")
pyplot.xlabel("Relaxation factor")
pyplot.ylabel("Time for solve")
pyplot.legend(loc=0);


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
3D Poisson Problem
Weak Scaling Test
Number of GPU: 1, 2, 4, 8 (K20m)
Machine: ivygpu-noecc
Top solver: Preconditioned CG (PCG)
Tolerance: absolute residual reach $10^{-8}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: AHAT
Smoother: Block Jocabi
Coarsest solver: Gauss-Seidel
Cycle: V
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 4
Maximum size of coarsest grid: 2
Relaxation factor of the block Jacobi: 0.8
Grid size: 10x10x10, 20x20x20, 40x40x40, 80x80x80, 160x160x160
Can you write Python code for it?


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
Strong Scaling Test
Number of GPU: 1, 2, 4, 8, 16, 32 (K20m)
Machine: ivygpu-noecc
Top solver: Preconditioned CG (PCG)
Tolerance: absolute residual reach $10^{-8}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: AHAT
Smoother: Block Jocabi
Coarsest solver: Gauss-Seidel
Cycle: V
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 4
Maximum size of coarsest grid: 2
Relaxation factor of the block Jacobi: 0.8
Grid size: 200x200x100, 200x200x200
Can you write Python code for it?


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
Test set 2:
Number of GPU: 12, 24 (K20)
Machine: ivygpu-noecc
Top solver: Preconditioned CG (PCG)
Tolerance: absolute residual reach $10^{-10}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: AHAT
Smoother: Block Jocabi
Coarsest solver: Block Jacobi
Cycle: V, W, F
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 1, 2, 3
Maximum size of coarsest grid: 10
Relaxation factor of the block Jacobi: 0.8
Grid size: 9000 $\times$ 9000 (matrix size: 81M $\times$ 81M)
Can you write Python code for it?

cycle = ["V", "W", "F"]
sweep = numpy.array([1, 2, 3])

time24 = {"V": numpy.array([5.152, 3.822, 4.314]),
        "W": numpy.array([5.568, 5.89, 6.39]),
        "F": numpy.array([5.886, 4.232, 4.53])}
errL24 = {"V": numpy.array([0.052, 0.152, 2.004]),
        "W": numpy.array([0.008, 0.03, 0.01]),
        "F": numpy.array([2.766, 0.002, 0.23])}
errU24 = {"V": numpy.array([0.018, 0.078, 1.986]),
        "W": numpy.array([0.012, 0.04, 0.02]),
        "F": numpy.array([3.174, 0.008, 0.89])}
err24 = {cyc: numpy.array(numpy.vstack((errL24[cyc], errU24[cyc]))) for cyc in cycle}

time12 = {"V": numpy.array([6.248, 5.238, 4.15]),
        "W": numpy.array([7.382, 5.53, 7.456]),
        "F": numpy.array([5.672, 5.58, 4.24])}
errL12 = {"V": numpy.array([0.008, 0.368, 0]),
        "W": numpy.array([0.002, 0, 1.656]),
        "F": numpy.array([0.992, 1.22, 0])}
errU12 = {"V": numpy.array([0.002, 1.472, 0]),
        "W": numpy.array([0.008, 0, 0.424]),
        "F": numpy.array([0.658, 1.83, 0])}
err12 = {cyc: numpy.array(numpy.vstack((errL12[cyc], errU12[cyc]))) for cyc in cycle}

pyplot.figure(figsize=(16, 4))
pyplot.subplot(1, 2, 1)
pyplot.title("12 Devices")
pyplot.errorbar(sweep, time12["V"], yerr = err12["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time12["W"], yerr = err12["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time12["F"], yerr = err12["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)
pyplot.subplot(1, 2, 2)
pyplot.title("24 Devices")
pyplot.errorbar(sweep, time24["V"], yerr = err24["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time24["W"], yerr = err24["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time24["F"], yerr = err24["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
Test set 3:
Number of GPU: 12, 24 (K20)
Machine: ivygpu-noecc
Top solver: Preconditioned CG (PCG)
Tolerance: absolute residual reach $10^{-8}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: AHAT
Smoother: Block Jocabi
Coarsest solver: Block Jacobi
Cycle: V, W, F
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 1, 2, 3
Maximum size of coarsest grid: 10
Relaxation factor of the block Jacobi: 0.8
Grid size: 9000 $\times$ 9000 (matrix size: 81M $\times$ 81M)
Can you write Python code for it?

cycle = ["V", "W", "F"]
sweep = numpy.array([1, 2, 3])

time24 = {"V": numpy.array([3.102, 2.444, 2.166]),
          "W": numpy.array([3.716, 4.376, 5.4]),
          "F": numpy.array([2.872, 3.31, 3.78])}
errL24 = {"V": numpy.array([0.032, 0.044, 0.006]),
          "W": numpy.array([0.066, 0.316, 0.99]),
          "F": numpy.array([0.012, 0.49, 0.88])}
errU24 = {"V": numpy.array([0.058, 0.016, 0.004]),
          "W": numpy.array([0.074, 1.214, 0.67]),
          "F": numpy.array([0.008, 0.74, 0.23])}
err24 = {cyc: numpy.array(numpy.vstack((errL24[cyc], errU24[cyc]))) for cyc in cycle}

time12 = {"V": numpy.array([6.238, 4.42, 3.7]),
          "W": numpy.array([5.272, 5.174, 5.396]),
          "F": numpy.array([4.23, 3.974, 4.58])}
errL12 = {"V": numpy.array([0.608, 0, 0]),
          "W": numpy.array([0.402, 0.004, 0.156]),
          "F": numpy.array([0.05, 0.004, 0.61])}
errU12 = {"V": numpy.array([2.422, 0, 0]),
          "W": numpy.array([1.608, 0.006, 0.044]),
          "F": numpy.array([0.08, 0.016, 0.92])}
err12 = {cyc: numpy.array(numpy.vstack((errL12[cyc], errU12[cyc]))) for cyc in cycle}

pyplot.figure(figsize=(16, 4))
pyplot.subplot(1, 2, 1)
pyplot.title("12 Devices")
pyplot.errorbar(sweep, time12["V"], yerr = err12["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time12["W"], yerr = err12["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time12["F"], yerr = err12["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)
pyplot.subplot(1, 2, 2)
pyplot.title("24 Devices")
pyplot.errorbar(sweep, time24["V"], yerr = err24["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time24["W"], yerr = err24["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time24["F"], yerr = err24["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
Test set 4:
Number of GPU: 12, 24 (K20)
Machine: ivygpu-noecc
Top solver: Preconditioned CG (PCG)
Tolerance: absolute residual reach $10^{-8}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: ALL
Smoother: Block Jocabi
Coarsest solver: Block Jacobi
Cycle: V, W, F
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 1, 2, 3
Maximum size of coarsest grid: 10
Relaxation factor of the block Jacobi: 0.8
Grid size: 9000 $\times$ 9000 (matrix size: 81M $\times$ 81M)
Can you write Python code for it?

cycle = ["V", "W", "F"]
sweep = numpy.array([1, 2, 3])

time24 = {"V": numpy.array([3.06, 2.422, 2.39]),
          "W": numpy.array([3.802, 4.376, 5.406]),
          "F": numpy.array([2.878, 3.382, 5.568])}
errL24 = {"V": numpy.array([0.05, 0.022, 0.23]),
          "W": numpy.array([0.002, 0.306, 1.006]),
          "F": numpy.array([0.008, 0.552, 0.668])}
errU24 = {"V": numpy.array([0.02, 0.038, 0.91]),
          "W": numpy.array([0.008, 1.214, 0.674]),
          "F": numpy.array([0.012, 0.988, 0.452])}
err24 = {cyc: numpy.array(numpy.vstack((errL24[cyc], errU24[cyc]))) for cyc in cycle}

time12 = {"V": numpy.array([6.23, 4.376, 3.702]),
          "W": numpy.array([5.266, 5.174, 5.43]),
          "F": numpy.array([4.208, 4.288, 4.572])}
errL12 = {"V": numpy.array([0.65, 0.126, 0.002]),
          "W": numpy.array([0.406, 0.004, 0.01]),
          "F": numpy.array([0.028, 0.318, 0.602])}
errU12 = {"V": numpy.array([2.42, 0.044, 0.008]),
          "W": numpy.array([1.614, 0.006, 0.01]),
          "F": numpy.array([0.112, 1.272, 0.908])}
err12 = {cyc: numpy.array(numpy.vstack((errL12[cyc], errU12[cyc]))) for cyc in cycle}

pyplot.figure(figsize=(16, 4))
pyplot.subplot(1, 2, 1)
pyplot.title("12 Devices")
pyplot.errorbar(sweep, time12["V"], yerr = err12["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time12["W"], yerr = err12["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time12["F"], yerr = err12["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)
pyplot.subplot(1, 2, 2)
pyplot.title("24 Devices")
pyplot.errorbar(sweep, time24["V"], yerr = err24["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time24["W"], yerr = err24["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time24["F"], yerr = err24["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)


I am working on the file "PetAmgXTest/Report.ipynb".
The first task is:
Test set 5:
Number of GPU: 12, 24 (K20)
Machine: ivygpu-noecc
Top solver: Preconditioned CG (PCG)
Tolerance: absolute residual reach $10^{-10}$
Preconditioner: AMG
AMG algorithm: Classical
Selector: PMIS
Interpolation: D2
Strength: AHAT
Smoother: Block Jocabi
Coarsest solver: LU Decomposition
Cycle: V, W, F
Pre-sweep: 1
Post-Sweep: 1
Coarsest sweep: 1, 2, 3
Maximum size of coarsest grid: 10
Relaxation factor of the block Jacobi: 0.8
Grid size: 9000 $\times$ 9000 (matrix size: 81M $\times$ 81M)
Can you write Python code for it?

cycle = ["V", "W", "F"]
sweep = numpy.array([1, 2, 3])

time24 = {"V": numpy.array([4.512, 4.508, 5.152]),
          "W": numpy.array([5.626, 5.63, 5.622]),
          "F": numpy.array([5.268, 5.286, 6.822])}
errL24 = {"V": numpy.array([0.332, 0.338, 0.962]),
          "W": numpy.array([0.026, 0.02, 0.022]),
          "F": numpy.array([1.088, 1.026, 0.012])}
errU24 = {"V": numpy.array([1.278, 1.292, 0.638]),
          "W": numpy.array([0.034, 0.02, 0.028]),
          "F": numpy.array([1.562, 1.534, 0.008])}
err24 = {cyc: numpy.array(numpy.vstack((errL24[cyc], errU24[cyc]))) for cyc in cycle}

time12 = {"V": numpy.array([4.81, 5.212, 6.016]),
          "W": numpy.array([7.402, 7.406, 7.4]),
          "F": numpy.array([5.584, 5.53, 5.554])}
errL12 = {"V": numpy.array([0, 0.402, 1.206]),
          "W": numpy.array([0.002, 0.006, 0]),
          "F": numpy.array([0.084, 0.03, 0.054])}
errU12 = {"V": numpy.array([0, 1.608, 0.804]),
          "W": numpy.array([0.008, 0.014, 0]),
          "F": numpy.array([0.056, 0.1, 0.086])}
err12 = {cyc: numpy.array(numpy.vstack((errL12[cyc], errU12[cyc]))) for cyc in cycle}

pyplot.figure(figsize=(16, 4))
pyplot.subplot(1, 2, 1)
pyplot.title("12 Devices")
pyplot.errorbar(sweep, time12["V"], yerr = err12["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time12["W"], yerr = err12["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time12["F"], yerr = err12["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)
pyplot.subplot(1, 2, 2)
pyplot.title("24 Devices")
pyplot.errorbar(sweep, time24["V"], yerr = err24["V"], fmt='ks-', label="V")
pyplot.errorbar(sweep, time24["W"], yerr = err24["W"], fmt='r^-', label="W")
pyplot.errorbar(sweep, time24["F"], yerr = err24["F"], fmt='bo-', label="F")
pyplot.xlabel("Number of iteration on coarsest grid")
pyplot.xlim(0, 4)
pyplot.ylabel("Time for solve (sec)")
pyplot.ylim(2, 10)
pyplot.legend(loc=0)


I am working on the file "PetFriendly.ipynb".
The first task is:
A quick glance at the whole dataset reveals that countries with few listings do not tell much about their attitude towards pets. Let's look at the top-5 friendliest countries.
Can you write Python code for it?

# Ratio of PFL
compare = df['apt_pets']/df['apt_total']
compare.sort(ascending=False)
print "The friendliest countries:"
compare[:5]


I am working on the file "PetFriendly.ipynb".
The first task is:
But, again, these are raw numbers. I'll show the top 5 most friendly countries in each region.
Can you write Python code for it?

for region in regions:
    print ("Top-5 friendliest countries in {}:"
           .format(region))
    display(df_cut[df_cut['region'] == region].sort('apt_ratio', ascending=False)[['apt_ratio']][:5])


I am working on the file "PetFriendly.ipynb".
The first task is:
I collected data on total number of listings located in each country ('apt_total') and number of pet friendly listings ('apt_pets'), or in short PFL.
I also parsed the average price per night in both categories ('apt_total_price' & 'apt_pets_price').
Prices are in USD.
World listings overview
Let's explore the world!
Can you write Python code for it?

# Number of listings.
apt_sum = df['apt_total'].sum()

# Number of PFL.
pets_sum = df['apt_pets'].sum()

print ("There are {0} listings all over the world on Airbnb."
       .format(humanize.intword(apt_sum)))
print ("Pets would be welcomed only in {0:.0%} of all listings."
       .format(pets_sum / apt_sum))

# Average price per night.
price_mean = round(df['apt_total_price'].mean())
pet_mean = round(df['apt_pets_price'].mean())

# Difference in price.
diff = (price_mean - pet_mean) / price_mean

print ("The average price for a listing is ${0} per night."
       .format(int(price_mean)))
print ("A PFL would cost {0:.0%} less, or ${1}."
       .format(diff, int(pet_mean)))


I am working on the file "PetFriendly.ipynb".
The first task is:
I'll map the ratio of PFL. First, I'll split the data into clusters using k-means. I have a module to choose the number of k.
Here I plot the sum of squared errors (between each point and the mean of its cluster) as a function of k and look at where the graph “bends”.
Can you write Python code for it?

import choosing_k as chk
data = df_cut['apt_ratio'].values.tolist()
chk.plot_errors(data)


I am working on the file "PetFriendly.ipynb".
The first task is:
I'll plot countries that account for more than 1% of PFL.
Can you write Python code for it?

# Add country codes to the DF.
pet_share_df = pd.concat(
                    [pet_share, df['country_code']],
                    axis=1).set_index('country_code').sort('apt_pets',
                                                        ascending=False)
# Make a bar chart
matplotlib.style.use('ggplot')
plt.figure();
share_plot = pet_share_df[pet_share_df['apt_pets']>1].plot(kind='bar',
                                                        legend=False,
                                                        rot=45,
                                                        title='Countries with share of all PFL > 1%\n')

share_plot.set_ylabel('Share, %')
share_plot.set_xlabel('')
plt.show()


I am working on the file "PetFriendly.ipynb".
The first task is:
I'll skip the data gathering step. I used a CSV file with the list of countries and regions to parse Airbnb.com website and saved the data as a sqlite dictionary. That is why I use the sqlitedict library here.
Can you write Python code for it?

from sqlitedict import SqliteDict
db = SqliteDict('./pet_friendly.sqlite')


I am working on the file "PetFriendly.ipynb".
The first task is:
Is Antarctica in the top-5 chart? Do penguins rent out their nests?
Apparently, there's some mistake on Airbnb. 17 listings located in different parts of the globe were mistakenly marked off as based in Antarctica.
I'll fix the dataset a bit for further analysis. I'll add a threshold of 20 listings and remove Antarctica.
Can you write Python code for it?

# Add a threshold of 20 listings.
df_cut = df[df['apt_total'] > 20]

# Remove Antarctica.
df_cut = df_cut[df_cut.index != 'Antarctica']

print ("These changes leave us {0} countries to examine."
       .format(len(df_cut)))

# Calculate ratio of PFL and add to the DataFrame.
df_cut['apt_ratio'] = (df_cut['apt_pets'] / df_cut['apt_total'])

print "\nTop-5 countries with the highest ratio of PFL:"
df_cut.sort('apt_ratio', ascending=False)[['apt_ratio', 'apt_total', 'apt_pets', 'region']][:5]


I am working on the file "PetFriendly.ipynb".
The first task is:
It seems, Airbnb is very popular in Europe. I'll plot the data.
Can you write Python code for it?

# Make a bar chart
matplotlib.style.use('bmh')

plt.figure();
region_listings.plot(kind='bar', rot=0, figsize=(8, 6))
plt.xlabel('')
plt.ylabel('Percentage')
plt.title('The distribution of accommodation '
          'listings among regions \n')
plt.show()


I am working on the file "PetFriendly.ipynb".
The first task is:
The box-and-whisker plot shows the distribution of data based on the five number summary: minimum, first quartile, median, third quartile, and maximum. This plot does not include outliers.
North America stands out as the most expensive region. Prices in this region are spread out over a wide range of values from \$40 to almost \$300.
Prices for a PFL in 50% of countries in North America (above median) would be higher than in the most expensive countries in Africa, Europe and Oceania.
Africa and Oceania are the most homogeneous regions. Prices do not differ greatly within each region.
Other observation is that in South America 50% of countries below the median are very uniform in listing prices and other half above the median, on the contrary, is widely distributed.
The mentioned wide distribution in the top (or most expensive) part of the box (above the third quartile) is true for all regions except Oceania. Oceania is uniform when it comes to high prices.
Country overview
Ok, so now we know that Europe is a region to go with a pet. It provides the biggest number of listings for a reasonable price, quarter of which accept pets, though price would be higher for a pet friendly place.
But let's examine the country level and find leaders in every region.
Can you write Python code for it?

# Share of each country in total listings
apt_share = (df_cut['apt_total'] / df_cut['apt_total'].sum() * 100).round(2)
apt_share.sort(ascending=False)

# Count 80% of countries
quantile_count = int(len(df_cut) * 0.80)

print ("80%, or {0} countries in the dataset "
       "account for {1}% of all listings available."
       .format(quantile_count, apt_share[-1*quantile_count: ].sum()))

print ("These 5 countries account for {0}% "
       "of all listings published on Airbnb: "
        .format(apt_share[:5].sum()))
apt_share[:5]


I am working on the file "PetFriendly.ipynb".
The first task is:
The cheapest region in terms of accommodation is South America. On average, day rent would cost \$66. Though the cheapest stay with a pet would be possible in Africa, just \$67 a night.
In Asia, price for a listing is rather low and equals \$78 a night, however pet allowance adds 41% to the average price and reaches \$110. That's more expensive than a PFL in Europe.
In Europe, stay with a pet would cost \$100, or 15% more than the average price per night. But in any case this price is lower than in Oceania and North America.
North America is the most expensive region to rent an apartment. The average price reaches \$268.5. Luxury apartments in this region close doors to pet owners. That lowers the average price for a PFL by 36% but it's still the highest comparing to other regions.
Can you write Python code for it?

# Make a boxplot
color = dict(boxes='DarkGreen', whiskers='DarkOrange',
             medians='DarkBlue', caps='Gray')
plt.figure();

# Split the data on 'apt_pets_price' into regional groups
region_groups = df_cut.groupby(['region'])[['apt_pets_price']]

# List of regions
regions = df_cut['region'].unique()
data = []

# Add the data from the DF
for item in regions:
        a = region_groups.get_group(item)
        data.append(a)

# Make a new DF for regions
box_df = pd.concat(data, ignore_index=True, axis=1)
box_df.columns = regions

# Plot the data
box_df.plot(kind='box', sym='r+', color=color,
            figsize=(8, 6)).set_ylim([0,300])
plt.ylabel('Average price per night, USD')
plt.title('The distribution of PFL prices in regions')
plt.show()


I am working on the file "PetFriendly.ipynb".
The first task is:
The same calculations for PFL reveal even lower numbers.
Can you write Python code for it?

# Share of each country in PFL
pet_share = (df_cut['apt_pets'] /
             df_cut['apt_pets'].sum() * 100).round(2)

pet_share.sort(ascending=False)

print ("80%, or {0} countries represent just "
       "{1}% of PFL.".format(quantile_count,
                           pet_share[-1 * quantile_count: ].sum()))

print ("These 5 countries account for {0}% "
       "of all listings published on Airbnb: "
        .format(pet_share[:5].sum()))
pet_share[:5]


I am working on the file "PetFriendly.ipynb".
The first task is:
This chart tells us that 100% of listings in Western Sahara and Wallis and Futuna are pet friendly. But that's only because there is only one listing in the country, and pets are allowed there.
Can you write Python code for it?

df.loc[['Western Sahara', 'Wallis and Futuna']][['apt_total', 'apt_pets']]


I am working on the file "PetFriendly.ipynb".
The first task is:
To map the data I'll use the folium library.
Can you write Python code for it?

import json
import folium
from folium.utilities import split_six

columns = ['country_code', 'apt_ratio']
color_data = df_cut.set_index(columns[0])[columns[1]].to_dict()

geo_json_data = json.load(open('countries.geo.json'))

from folium.utilities import split_six, color_brewer
from folium.features import ColorScale

m = folium.Map([32, -45], tiles='Mapbox',
               API_key='wrobstory.map-12345678',
               zoom_start=2)

# Pass own threshold scale
color_domain = threshold_scale

# Choose color
fill_color = 'YlGnBu'
color_range = color_brewer(fill_color, n=len(color_domain))
key_on = 'id'

# I made a fix for the folium library to colour with white countries with no data.
def get_by_key(obj, key):
    if len(key.split('.')) <= 1:
        return obj.get(key, None)

def color_scale_fun(x):
    try:
        r = [u for u in color_domain if u <= color_data[get_by_key(x, key_on)]]
        return color_range[len(r)]
    except KeyError:
        return '#FFFFFF'

# Make a map
folium.GeoJson(geo_json_data,
              style_function=lambda feature: {
                  'fillColor': color_scale_fun(feature),
                  'color': 'black',
                  'weight': 1,
                    'legend_name' : 'Unemployment Rate (%)',
                  'fillOpacity': 0.8
              }).add_to(m)

color_scale = ColorScale(color_domain, fill_color, caption="Ratio of pet friendly listings")
m.add_children(color_scale)
print "Ratio of pet friendly listings to all listings"
m


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Applying the classification model to new data
Now that we have a trained facies classification model we can use it to identify facies in wells that do not have core data.  In this case, we will apply the classifier to two wells, but we could use it on any number of wells for which we have the same set of well logs for input.
This dataset is similar to the training data except it does not have facies labels.  It is loaded into a dataframe called test_data.
Can you write Python code for it?

well_data = pd.read_csv('validation_data_nofacies.csv')
well_data['Well Name'] = well_data['Well Name'].astype('category')
well_features = well_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
As noted above, the boundaries between the facies classes are not all sharp, and some of them blend into one another.  The error within these 'adjacent facies' can also be calculated.  We define an array to represent the facies adjacent to each other.  For facies label i, adjacent_facies[i] is an array of the adjacent facies labels.
Can you write Python code for it?

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

print('Facies classification accuracy = %f' % accuracy(conf))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf, adjacent_facies))


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Before we plot the well data, let's define a color map so the facies are represented by consistent color in all the plots in this tutorial. We also create the abbreviated facies labels, and add those to the facies_vectors dataframe.
Can you write Python code for it?

# 1=sandstone  2=c_siltstone   3=f_siltstone
# 4=marine_silt_shale 5=mudstone 6=wackestone 7=dolomite
# 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00',
       '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Conditioning the data set
Now we extract just the feature variables we need to perform the classification.  The predictor variables are the five wireline values and two geologic constraining variables. We also get a vector of the facies labels that correspond to each feature vector.
Can you write Python code for it?

correct_facies_labels = training_data['Facies'].values

# dropping certain labels and only keeping the geophysical log values to train on
feature_vectors = training_data.drop(['Formation', 'Well Name', 'Depth','Facies','FaciesLabels'], axis=1)
feature_vectors.describe()


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Considering adjacent facies, the F1 scores for all facies types are above 0.9, except when classifying SiSh or marine siltstone and shale.  The classifier often misclassifies this facies (recall of 0.66), most often as wackestone.
These results are comparable to those reported in Dubois et al. (2007).
Applying the classification model to the blind data
We held a well back from the training, and stored it in a dataframe called blind:
Can you write Python code for it?


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Finally we predict facies labels for the unknown data, and store the results in a Facies column of the test_data dataframe.
Can you write Python code for it?

#predict facies of unclassified data
y_unknown = clf.predict(X_unknown)
well_data['Facies'] = y_unknown
well_data

well_data['Well Name'].unique()


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Let's see how we did with the confusion matrix:
Can you write Python code for it?

cv_conf = confusion_matrix(y_blind, y_pred)

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf, adjacent_facies))


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Let's take a look at the data from individual wells in a more familiar log plot form. We will create plots for the five well log variables, as well as a log for facies labels. The plots are based on the those described in Alessandro Amato del Monte's excellent tutorial.
Can you write Python code for it?

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')

    ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([])
    ax[4].set_yticklabels([]); ax[5].set_yticklabels([])
    ax[5].set_xticklabels([])
    f.suptitle('Well: %s'%logs.iloc[0]['Well Name'],...


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Now it's a simple matter of making a prediction and storing it back in the dataframe:
Can you write Python code for it?

y_pred = clf.predict(X_blind)
blind['Prediction'] = y_pred


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Now that the model has been trained on our data, we can use it to predict the facies of the feature vectors in the test set.  Because we know the true facies labels of the vectors in the test set, we can use the results to evaluate the accuracy of the classifier.
Can you write Python code for it?


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Now we can transform this with the scaler we made before:
Can you write Python code for it?


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Scikit also includes a handy function to randomly split the training data into training and test sets.  The test set contains a small subset of feature vectors that are not used to train the network.  Because we know the true facies labels for these examples, we can compare the results of the classifier to the actual facies and determine the accuracy of the model.  Let's use 20% of the data for the test set.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
        scaled_features, correct_facies_labels, test_size=0.1, random_state=42)


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Scikit includes a preprocessing module that can 'standardize' the data (giving each variable zero mean and unit variance, also called whitening).  Many machine learning algorithms assume features will be standard normally distributed data (ie: Gaussian with zero mean and unit variance).  The factors used to standardize the training set must be applied to any subsequent feature set that will be input to the classifier.  The StandardScalar class can be fit to the training set, and later used to standardize any training data.
Can you write Python code for it?

from sklearn import preprocessing

scaler = preprocessing.StandardScaler().fit(feature_vectors)
scaled_features = scaler.transform(feature_vectors)

feature_vectors

feature_vectors.describe()


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
The best accuracy on the cross validation error curve was achieved for gamma = 1, and C = 10. We can now create and train an optimized classifier based on these parameters:
Can you write Python code for it?

clf = svm.SVC(C=10, gamma=1)
clf.fit(X_train, y_train)

cv_conf = confusion_matrix(y_test, clf.predict(X_test))

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf, adjacent_facies))


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables. Looking at the count values, there are 3232 feature vectors in the training set.
Remove a single well to use as a blind test later.
Can you write Python code for it?

blind = training_data[training_data['Well Name'] == 'SHANKLE']
training_data = training_data[training_data['Well Name'] != 'SHANKLE']


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
Training the SVM classifier
Now we use the cleaned and conditioned training set to create a facies classifier.  As mentioned above, we will use a type of machine learning model known as a support vector machine.  The SVM is a map of the feature vectors as points in a multi dimensional space, mapped so that examples from different facies are divided by a clear gap that is as wide as possible.
The SVM implementation in scikit-learn takes a number of important parameters.  First we create a classifier using the default settings.
Can you write Python code for it?

from sklearn import svm

clf = svm.SVC()


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
We can use the well log plot to view the classification results along with the well logs.
Can you write Python code for it?

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'STUART'],
    facies_colors=facies_colors)

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'CRAWFORD'],
    facies_colors=facies_colors)

well_data.to_csv('well_data_with_facies.csv')


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
We need some metrics to evaluate how good our classifier is doing.  A confusion matrix is a table that can be used to describe the performance of a classification model.  Scikit-learn allows us to easily create a confusion matrix by supplying the actual and predicted facies labels.
The confusion matrix is simply a 2D array.  The entries of confusion matrix C[i][j] are equal to the number of observations predicted to have facies j, but are known to have facies i.
To simplify reading the confusion matrix, a function has been written to display the matrix along with facies labels and various error metrics.  See the file classification_utilities.py in this repo for the display_cm() function.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
from classification_utilities import display_cm, display_adj_cm

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, hide_zeros=True)

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
editing the well viewer code in an attempt to understand it and potentially not show everything
Can you write Python code for it?

def make_faciesOnly_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)
    f, ax = plt.subplots(nrows=1, ncols=2, figsize=(3, 9))
#     f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
#     ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[0].plot(logs.ILD_log10, logs.Depth, '-')
#     ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
#     ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
#     ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[1].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[1])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

#     ax[0].set_xlabel("GR")
#     ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[0].set_xlabel("ILD_log10")
    ax[0].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
#     ax[2].set_xlabel("DeltaPHI")
#     ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
#     ax[3].set_xlabel("PHIND")
#     ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
#     ax[4].set_xlabel("PE")
#     ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[1].set_xlabel('Facies')

#     ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([])
#     ax[4].set_yticklabels([]);
    ax[1].set_yticklabels([])
   ...


I am working on the file "Pet_Stromatolite/Facies_Classification_Draft2.ipynb".
The first task is:
looking at several wells at once
Can you write Python code for it?

# make_faciesOnly_log_plot(
#     training_data[training_data['Well Name'] == 'SHRIMPLIN'],
#     facies_colors)

for i in range(len(well_name_list)-1):
#     well_name_list[i]
    make_faciesOnly_log_plot(
        training_data[training_data['Well Name'] == well_name_list[i]],
        facies_colors)


I am working on the file "Phase-diagrams.ipynb".
The first task is:
Calculate full phase diagram
Make sure tempdata exists in the current folder.
Set full_phase_diagram to False if you only want the band gap in the non-trivial region or True if you want it in the whole Bs, mu_mesh range.
Can you write Python code for it?


I am working on the file "Phase-diagrams.ipynb".
The first task is:
Check how the phase diagram looks
This will show all data.
Can you write Python code for it?

%%output size=200
%%opts Image [colorbar=False] {+axiswise} (clims=(0, 0.1))
phase_diagram = create_holoviews(fname)

(phase_diagram.Phase_diagram.Band_gap.hist()
 + phase_diagram.Phase_diagram.Inverse_decay_length
 + phase_diagram.Sphere.I).cols(2)

%%opts Image [colorbar=True]
phase_diagram.Phase_diagram.Band_gap

phase_diagram.cdims


I am working on the file "Phase-diagrams.ipynb".
The first task is:
Make sure to add the correct path like:
sys.path.append("/path/where/to/ipynb/runs")
Can you write Python code for it?

%%px --local
import sys
import os
# CHANGE THE LINE BELOW INTO THE CORRECT FOLDER!
sys.path.append(os.path.join(os.path.expanduser('~'), 'orbitalfield'))
import kwant
import numpy as np
from fun import *

def gap_and_decay(lead, p, val, tol=1e-4):
    gap = find_gap(lead, p, val, tol)
    decay_length = find_decay_length(lead, p, val)
    return gap, decay_length

import holoviews as hv
import holoviews_rc
hv.notebook_extension()


I am working on the file "Phase-diagrams.ipynb".
The first task is:
Save
Run this function to save the data to hdf5 format, it will include all data and parameters that are used in the simulation.
Can you write Python code for it?

fname = 'data/test.h5'
save_data(fname, Bs, thetas, phis, mu_mesh, mus_output, gaps_output, decay_length_output, p, constants)


I am working on the file "Phase-diagrams.ipynb".
The first task is:
Uncomment the lines for the wire that you want to use.
Can you write Python code for it?

%%px --local
# angle = 0 # WIRE WITH SC ON TOP

angle = 45 # WIRE WITH SC ON SIDE
p = make_params(t_interface=7/8*constants.t, Delta=68.4, r1=50, r2=70,
                orbital=True, angle=angle, A_correction=True, alpha=100) #r2=70

p.V = lambda x, y, z: 2 / 50 * z
lead = make_3d_wire_external_sc(a=constants.a, r1=p.r1, r2=p.r2, angle=p.angle)

# WIRE WITH CONSTANT GAP
# lead = make_3d_wire()
# p = make_params(V=lambda x, y, z: 0, orbital=True)


I am working on the file "Phi_identity.ipynb".
The first task is:
Lets evaluate the individual terms on either side of the + symbol.  They're not both equal to 1 are they?  That would mean we're just writing 1 + 1 == 2 every time.  But we already know either term may be 0, so clearly we can't expect the same terms in each row.
Can you write Python code for it?

term1    = "{coeff0:>3} * Ø**({a:>3})"
term2    = "{coeff1:>3} * Ø**({b:>3})"
terms    = "{t1:>12.9f} + {t2:>12.9f}"
args     = {}
coeff0 = fibo()
coeff1 = fibo(2, -1)

for i in range(10):
    args["a"] = 3 - i
    args["b"] = -i
    args["coeff0"] = next(coeff0)
    args["coeff1"] = next(coeff1)
    t1 = eval(term1.format(**args))
    t2 = eval(term2.format(**args))
    print(terms.format(t1=t1, t2=t2), end=" --> ")
    print("{:>12.9f}".format(eval(expr.format(**args))))

def sequence(n):
    coeff0 = fibo()
    coeff1 = fibo(2, -1)
    total = 0
    for i in range(n+1):
        args["a"] = 3 - i
        args["b"] = -i
        args["coeff0"] = next(coeff0)
        args["coeff1"] = next(coeff1)
        total += eval(expr.format(**args))
    return total

sequence(0)

[sequence(n) for n in range(10)]


I am working on the file "Phi_identity.ipynb".
The first task is:
Lets generalize by rewinding further back in the Fibonacci Numbers, starting with coefficients 610 and -144 respectively, and going forward from there.  The printout below overlaps the one above, while showing that, indeed, we're free to press this identity in both directions.
Two terms with Fibonacci coefficients, three apart (the first ahead), and exponents of phi three apart (the first three higher), always add to give the number 2, starting with anchoring relationships such as:
$$
2 = fibo(i)\phi^{3-i} + fibo(i-3)\phi^{-i}
$$
We could rewrite the Sigma expression accordingly.
Can you write Python code for it?

fib_gen = fibo(-144, 89)
print("SEQ1:",[next(fib_gen) for _ in range(10)])

fib_gen = fibo(610, -377)
print("SEQ2:",[next(fib_gen) for _ in range(10)])

coeff0 = fibo(-144, 89)
coeff1 = fibo(610, -377)
for i in range(-12,8):
    args["a"] = 3 - i
    args["b"] = -i
    args["coeff0"] = next(coeff0)
    args["coeff1"] = next(coeff1)
    identity = template.format(**args)
    print(identity, end=" --> ")
    print("{:>20.18f}".format(eval(expr.format(**args))))

terms    = "{t1:>18.9f} + {t2:>18.9f}"
coeff0 = fibo(-144, 89)
coeff1 = fibo(610, -377)
for i in range(-12,8):
    args["a"] = 3 - i
    args["b"] = -i
    args["coeff0"] = next(coeff0)
    args["coeff1"] = next(coeff1)
    t1 = eval(term1.format(**args))
    t2 = eval(term2.format(**args))
    print(terms.format(t1=t1, t2=t2), end=" --> ")
    print("{:>12.9f}".format(eval(expr.format(**args))))

from IPython.display import YouTubeVideo
YouTubeVideo("kXn-cvynsoE")


I am working on the file "Phillies-2016.ipynb".
The first task is:
A downloadable CSV schedule is available from mlb.com.  Here is a direct link to the Phillies schedule.
The CSV schedule will be used to instantiate a Pandas DataFrame object.
Can you write Python code for it?


I am working on the file "Phillies-2016.ipynb".
The first task is:
Cleaning up the schedule
The DESCRIPTION column contains the broadcast information.  Less interesting columns can be removed.
Can you write Python code for it?

schedule.drop(["REMINDER OFF",
             "REMINDER ON",
             "START TIME ET",
             "END DATE",
             "END DATE ET",
             "END TIME",
             "END TIME ET",
             "REMINDER TIME",
             "REMINDER TIME ET",
             "SHOWTIMEAS FREE",
             "SHOWTIMEAS BUSY",
             "REMINDER DATE"], axis=1, inplace=True)
schedule.head()


I am working on the file "Phillies-2016.ipynb".
The first task is:
Parse television station broadcast channels from DESCRIPTION
Thankfully, the DESCRIPTION column data is parseable.  Getting a list of television broadcast stations for each game is not too difficult.
Can you write Python code for it?

description = schedule.DESCRIPTION[6]
print description


I am working on the file "Phillies-2016.ipynb".
The first task is:
Test it out on all of the descriptions.
Can you write Python code for it?

tv_stations = set()
for d in schedule.DESCRIPTION:
    tv_stations |= set(tv_stations_from_description(d))
tv_stations


I am working on the file "Phillies-2016.ipynb".
The first task is:
The 190 Phillies games are broadcast on 6 television channels.  Unfortunately only 1 of those 6 stations are available without a cable television subscription.  This means that I can only watch games on NBC.
The Phillies national television broadcast schedule
Filtering the DESCRIPTION column to national television broadcast stations yields only the games which I can watch over the air with my HD antenna.
Can you write Python code for it?

national_broadcast_schedule = schedule[schedule.DESCRIPTION.str.contains("NBC 10") == True]
national_broadcast_schedule

national_broadcast_schedule.describe()


I am working on the file "Phillies-2016.ipynb".
The first task is:
What does the schedule metadata look like?
Can you write Python code for it?


I am working on the file "PhillyCrime.ipynb".
The first task is:
Above: blue dots are original data, green line is the fit from the simple regression.
Remove Center City and redo the analysis
Center City is the one observation with an extremely high crime rate, yet house prices are not very low.  This point does not follow the trend of the rest of the data very well.  A question is how much including Center City is influencing our fit on the other datapoints.  Let's remove this datapoint and see what happens.
Can you write Python code for it?

sales_noCC = sales[sales['MilesPhila'] != 0.0]

sales_noCC.show(view="Scatter Plot", x="CrimeRate", y="HousePrice")


I am working on the file "PhillyCrime.ipynb".
The first task is:
Compare coefficients for full-data fit versus no-Center-City fit
Visually, the fit seems different, but let's quantify this by examining the estimated coefficients of our original fit and that of the modified dataset with Center City removed.
Can you write Python code for it?


I am working on the file "PhillyCrime.ipynb".
The first task is:
Do the coefficients change much?
Can you write Python code for it?


I am working on the file "PhillyCrime.ipynb".
The first task is:
Fit the regression model using crime as the feature
Can you write Python code for it?

crime_model = graphlab.linear_regression.create(sales, target='HousePrice', features=['CrimeRate'],validation_set=None,verbose=False)


I am working on the file "PhillyCrime.ipynb".
The first task is:
Let's see what our fit looks like
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(sales['CrimeRate'],sales['HousePrice'],'.',
        sales['CrimeRate'],crime_model.predict(sales),'-')


I am working on the file "PhillyCrime.ipynb".
The first task is:
Look at the fit:
Can you write Python code for it?

plt.plot(sales_noCC['CrimeRate'],sales_noCC['HousePrice'],'.',
        sales_noCC['CrimeRate'],crime_model.predict(sales_noCC),'-')


I am working on the file "PhillyCrime.ipynb".
The first task is:
Refit our simple regression model on this modified dataset:
Can you write Python code for it?

crime_model_noCC = graphlab.linear_regression.create(sales_noCC, target='HousePrice', features=['CrimeRate'],validation_set=None, verbose=False)


I am working on the file "Photon_Diffusion/Photon_Diffusion.ipynb".
The first task is:

Calculate number of photons absorbed as a function of d

A function is defined to calculate the total number of photons absorbed at $z=d$ after all time as a function of $d$. The results are then plotted and found to be linear. The rather unpleasant expression defined above evidently can be approximated as (or is exactly equal to)
$$N_{abs}(d) = N_0 (1 - d/L) $$

Can you write Python code for it?

FractionAbsArrayAnalytic = []
distancearray = []

#Find the fraction of photons absorbed at z=d for various values of d ranging from 0 to L - 1 um (to avoid division by zero errors)
for distance in linspace(0,L-10**-6,100):
    Integrationtime = 10**-12
    TargetError = 10**-3
    Error = 1.0
    FractionAbsAnalytic=0
    while Error>TargetError:
        Error = abs(FractionAbsAnalytic-F(Integrationtime,100,distance))/F(Integrationtime,100,distance)
        FractionAbsAnalytic = F(Integrationtime,100,distance)
        Integrationtime *= 2
    FractionAbsArrayAnalytic.append(FractionAbsAnalytic)
    distancearray.append(distance*10**6)

#Update the matplotlib configuration parameters
mpl.rcParams.update({'font.size': 18, 'font.family': 'serif'})

#Adjust figure size
plt.subplots(figsize=(12,6))

plt.plot(distancearray,FractionAbsArrayAnalytic,linewidth=2)
#plt.xlim(np.min(tarray),np.max(tarray))
#plt.ylim(0,1.0)
#plt.xlim(0,50)
plt.xlabel('Segment Distance (um)')
plt.ylabel('Fraction Absorbed by CCD')
#plt.semilogy()