I am working on the file "JHU/information_retrieval/prog_4/prog_4.ipynb".
The first task is:
Title + Abstract + Keywords
Can you write Python code for it?

train_data = [concat(*item) for item in zip(train['title'], train['abstract'], train['keywords'])]

vectorizer = CountVectorizer()
train_counts = vectorizer.fit_transform(train_data)

clf = BernoulliNB(alpha=.001).fit(train_counts, train['assessment'])

train_data = [concat(*item) for item in zip(dev['title'] + dev['abstract'] + dev['keywords'])]
dev_text = vectorizer.transform(train_data)
preds = clf.predict(dev_text)

recall(preds, dev['assessment'])
precision(preds, dev['assessment'])
f1(preds, dev['assessment'])


I am working on the file "JLOWE/JLowe_NN.ipynb".
The first task is:
Cross Validation using a 'Blind Well Test'. Code adapted from PA_Team submission
Can you write Python code for it?

logo = LeaveOneGroupOut()
t0 = tm.time()
f1s_ls = []
acc_ls = []
adj_ls = []

for train, test in logo.split(X_scaled, y, groups=wells):
    well_name = wells[test[0]]
    X_tr = X_scaled[train]
    X_te = X_scaled[test]

    #convert y array into categories matrix
    classes = 9
    y_tr = np_utils.to_categorical(y[train], classes)

    # Method initialization
    NN = DNN()

    # Training
    NN.fit(X_tr, y_tr, nb_epoch=15, batch_size=5, verbose=0)

    # Predict
    y_hat = NN.predict_classes(X_te, verbose=0)
    y_hat = medfilt(y_hat, kernel_size=7)

    try:
        f1s = f1_score(y[test], y_hat, average="weighted", labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
    except:
        f1s = 0

    try:
        conf = confusion_matrix(y[test], y_hat, labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
        acc = accuracy(conf) # similar to f1 micro
    except:
        acc = 0

    try:
        acc_adj = accuracy_adjacent(conf, adjacent_facies)
    except:
        acc_adj = 0

    f1s_ls += [f1s]
    acc_ls += [acc]
    adj_ls += [acc_adj]
    print("{:>20s} f1_weigthted:{:.3f} | acc:{:.3f} | acc_adj:{:.3f}".format(well_name, f1s, acc, acc_adj))

t1 = tm.time()
print("Avg F1", np.average(f1s_ls)*100, "Avg Acc", np.average(acc_ls)*100, "Avg Adj", np.average(adj_ls)*100)
print("Blind Well Test Run Time:",'{:f}'.format((t1-t0)), "seconds")


I am working on the file "JLOWE/JLowe_NN.ipynb".
The first task is:
Defining the neural network model
Can you write Python code for it?

def DNN():
    # Model
    model = Sequential()
    model.add(Dense(205, input_dim=8, activation='relu',W_constraint=maxnorm(5)))
    model.add(Dropout(0.1))
    model.add(Dense(69, activation='relu',W_constraint=maxnorm(5)))
    model.add(Dropout(0.1))
    model.add(Dense(69, activation='relu'))
    model.add(Dense(9, activation='softmax'))
    # Compilation
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


I am working on the file "JLOWE/JLowe_NN.ipynb".
The first task is:
Final Model which uses all the training data
By using all the training data I may be potentially increasing the variance of the model but I believe it’s best to use all the data in the model as the data available is limited.
Can you write Python code for it?

NN = DNN()
NN.fit(X_train, Y_train, nb_epoch=15, batch_size=5, verbose=0)

y_predicted = NN.predict_classes(X_train, verbose=0)
y_predicted = medfilt(y_predicted, kernel_size=7)

f1s = f1_score(y, y_predicted, average="weighted")
Avgf1s = np.average(f1s_ls)*100
print ("f1 training error: ", '{:f}'.format(f1s))
print ("f1 test error: ", '{:f}'.format(Avgf1s))


I am working on the file "JLOWE/JLowe_NN.ipynb".
The first task is:
My variance is high and my bias is too low.
I haven’t found the optimum bias-variance trade off. --> Back to the drawing board.
Predicting the lithologies in the unknown test wells
Can you write Python code for it?

x_blind = scaler.transform(X_blind)
y_blind = NN.predict_classes(x_blind, verbose=0)
y_blind = medfilt(y_blind, kernel_size=7)
blind_data["Facies"] = y_blind + 1  # return the original value (1-9)

blind_data.to_csv("J_Lowe_Submission.csv")


I am working on the file "JLOWE/JLowe_NN.ipynb".
The first task is:
Sorting the data and dropping unwanted columns from the training and test data
Leave the depth in as a predictor - can the NN recognise depth trends? - Other teams gone much further and have taken into account a predictors relationship/change with depth.
Can you write Python code for it?

X = training_data.drop(['Formation', 'Well Name', 'Facies', 'FaciesLabels'], axis=1).values
y = training_data['Facies'].values - 1
X_blind = blind_data.drop(['Formation', 'Well Name'], axis=1).values
wells = training_data["Well Name"].values


I am working on the file "JSD_Example.ipynb".
The first task is:
and returns zero when operating on the same image...
Can you write Python code for it?

hist_loss(john, john)

hist_loss(luke, luke)

hist_loss(np.zeros(255),np.ones(255))

JSD(np.zeros(255),np.ones(255)/255)


I am working on the file "JSI-release.ipynb".
The first task is:
Validation and Visualization
Can you write Python code for it?

# Validation and Visualization of the clusters
clusters = Util.removeContained(clusters)
data_array2 = Util.remove_small_activities(data_array, dataAll[:, [3]], minDur)
dataAll2 = np.column_stack((data_array2[:, [3, 5]], data_array2[:, 0], data_array2[:, -1]))

activity_means = Util.get_activity_means(np.column_stack((data_array2[:, features_list], data_array2[:, [0, -1]])))
activities_set = list(set(dataAll2[:, [3]].T[0]))
dict_activity_index_colour = dict(zip(activities_set, np.arange(len(activities_set))))  # {1:0, 2:1, 6:2, 32:3}

# find the closest activity to each cluster and assign color
cluster_segments, cluster_segments_complex, cluster_colors_set, cluster_array, ratios = \
    Util.findClosestActivity(clusters, activity_means, dict_activity_index_colour)

#Validate and visualize
confusion_matrix_detailed, hungarian_matrix, result = \
    Util.validation(cluster_colors_set, dataAll, dict_activity_index_colour, activities_set,
                    cluster_segments_complex, True, [], cluster_array, [], n_clusters_,
                    cluster_segments, minDur, True)


I am working on the file "JSONSchemaNotes.ipynb".
The first task is:
OK, now let's write a traitlets class that does the same thing:
Can you write Python code for it?

import traitlets as T

class SimpleInstance(T.HasTraits):
    foo = T.Unicode()
    bar = T.Float()

# Should succeed

SimpleInstance(**good_instance)

# Should fail

try:
    SimpleInstance(**bad_instance)
except T.TraitError as err:
    print(err)


I am working on the file "JSONSchemaNotes.ipynb".
The first task is:
Trying it out...
Can you write Python code for it?

code = JSONSchema(simple_schema).object_code()
print(code)


I am working on the file "James_Joyce_Ulysses.ipynb".
The first task is:
A better method in Python 3 is -
Can you write Python code for it?

sorted(dictionary.items(), key=lambda x: x[1])

#Much more interesting are the uncommon words

infreq_metric = []
for ordered_words,value_words in sorted_dict:
    if value_words == 1:
        infreq_metric.append((ordered_words,value_words))
for word in infreq_metric:
    print('"{}"'.format(word[0]))

#the very common words
freq_metric = []
for ordered_words,value_words in sorted_dict:
    if value_words > 944:
        freq_metric.append((ordered_words,value_words))
for pair in freq_metric:
    print('The word "{}" appears {} times'.format(pair[0],pair[1]))


I am working on the file "James_Joyce_Ulysses.ipynb".
The first task is:
These are the words that appear more than 200 times and I have excluded the really common words (greater than 944 times)
Can you write Python code for it?

freq_metric = []
for ordered_words,value_words in sorted_dict:
    if value_words >200 and value_words< 944:
        freq_metric.append(str(value_words))
        print(ordered_words,value_words)

%matplotlib inline
numbs = [int(x) for x in freq_metric]
plt.plot(numbs)

	l = [makeSortable(str(dictionary[k])) + " # " + k for k in dictionary.keys()]

	for w in sorted(l):
		print(w)
	count = {}

	for k in dictionary.keys():
		if dictionary[k] in count:
			tmp = count[dictionary[k]]
			count[dictionary[k]] = tmp + 1
		else:
			count[dictionary[k]] = 1
	for k in sorted(count.keys()):
		print(str(count[k]) + " words appear " + str(k) + " times")

# %load text_analysis.py

# this code is licenced under creative commons licence as long as you
# cite the author: Rene Pickhardt / www.rene-pickhardt.de

# adds leading zeros to a string so all result strings can be ordered
def makeSortable(w):
	l = len(w)
	tmp = ""
	for i in range(5-l):
		tmp = tmp + "0"
	tmp = tmp + w
	return tmp

#replaces all kind of structures passed in l in a text s with the 2nd argument
def removeDelimiter(s,new,l):
	for c in l:
		s = s.replace(c, new);
	return s;

def analyzeWords(s):
	s = removeDelimiter(s," ",[".",",",";","_","-",":","!","?","\"",")","("])
	wordlist = s.split()

	dictionary = {}
	for word in wordlist:
		if word in dictionary:
			tmp = dictionary[word]
			dictionary[word]=tmp+1
		else:
			dictionary[word]=1

	l = [makeSortable(str(dictionary[k])) + " # " + k for k in dictionary.keys()]

	for w in sorted(l):
		print(w)
	count = {}

	for k in dictionary.keys():
		if dictionary[k] in count:
			tmp = count[dictionary[k]]
			count[dictionary[k]] = tmp + 1
		else:
			count[dictionary[k]] = 1
	for k in sorted(count.keys()):
		print(str(count[k]) + " words appear " + str(k) + " times")

def differentWords(s):
	s = removeDelimiter(s," ",[".",",",";","_","-",":","!","?","\"",")","("])
	wordlist = s.split()
	count = 0
	dictionary = {}
	for word in wordlist:
		if word in dictionary:
			tmp =...


I am working on the file "Janacare_Habits_dataset_upto-7May2016.ipynb".
The first task is:
Remove duplicates
Can you write Python code for it?

user_data_to_deDuplicate = user_data_to_clean.copy()

user_data_deDuplicateD = user_data_to_deDuplicate.loc[~user_data_to_deDuplicate.email.str.strip().duplicated()]
len(user_data_deDuplicateD)

user_data_deDuplicateD.info()

# Now its time to convert the timedelta64 data type column named age_on_platform to seconds
def convert_timedelta64_to_sec(td64):
    ts = (td64 / np.timedelta64(1, 's'))
    return ts

user_data_deDuplicateD_timedelta64_converted = user_data_deDuplicateD.copy()
temp_copy = user_data_deDuplicateD.copy()
user_data_deDuplicateD_timedelta64_converted.drop("age_on_platform", 1)
user_data_deDuplicateD_timedelta64_converted['age_on_platform'] = temp_copy['age_on_platform'].apply(convert_timedelta64_to_sec)

user_data_deDuplicateD_timedelta64_converted.info()


I am working on the file "Janacare_Habits_dataset_upto-7May2016.ipynb".
The first task is:
Validate if email i'd is correctly formatted and the email i'd really exists
Can you write Python code for it?

from validate_email import validate_email

email_count_invalid = 0
for index, row in user_data_to_clean.iterrows():
        if not validate_email(row.email): # , verify=True)  for checking if email i'd actually exits
            user_data_to_clean.drop(index, inplace=True)
            email_count_invalid = email_count_invalid + 1

print "Number of email-id invalid: %d" % (email_count_invalid)

# Check the result of last operation
user_data_to_clean.info()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
But on eyeballing I noticed some, cells of column first_login have greater value than corresponding cell of last_activity. These cells need to be swapped, since its not possible to have first_login > last_activity
Can you write Python code for it?

# Run a loop through the data frame and check each row for this anamoly, if found swap
for index, row in user_data_to_clean.iterrows():
    if row.first_login > row.last_activity:
        temp_date_var = row.first_login
        user_data_to_clean.set_value(index, 'first_login', row.last_activity)
        user_data_to_clean.set_value(index, 'last_activity', temp_date_var)
        #print "\tSw\t" + "first\t" + row.first_login.isoformat() + "\tlast\t" + row.last_activity.isoformat()

# Create new column 'age_on_platform' which has the corresponding value in date type format
user_data_to_clean["age_on_platform"] = user_data_to_clean["last_activity"] - user_data_to_clean["first_login"]

# Check the result in first few rows
user_data_to_clean["age_on_platform"].head(5)

# Lets check the health of the data set
user_data_to_clean.info()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
Data Clean up
In the last section of looking around, I saw that a lot of rows do not have any values or have garbage values(see first row of the table above).
This can cause errors when computing anything using the values in these rows, hence a clean up is required.
We will clean up only those columns, that are being used for features.

num_modules_consumed
num_glucose_tracked
num_of_days_food_tracked
watching_videos

The next two colums will not be cleaned, as they contain time data which in my opinion should not be imputed
* first_login
* last_activity
Can you write Python code for it?

# Lets check the health of the data set
user_data_to_clean.info()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
Finally the columns last_activity, age_on_platform have missing values, as evident from above table. Since this is time data, that in my opinion should not be imputed, we will drop/delete the columns.
Can you write Python code for it?

# Since only these two columns are having null values, we can run the function *dropna()* on the whole data frame
# All rows with missing data get dropped
user_data_imputed_data_frame.dropna(axis=0, inplace=True)
user_data_imputed_data_frame.info()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
Here we remove the datetime64 & timedelta64 columns too, the issue is Sci Kit learn methods can only deal with numerical and string features. I am trying to sort this issue
Can you write Python code for it?

# Now to create the other list of lists with features as elements
# before that we will have to remove the Labels column
user_data_imputed_data_frame_UNlabeled_training = user_data_imputed_data_frame_labeled_training.drop(['label'] ,1)
user_data_imputed_data_frame_UNlabeled_training.info()

# As you may notice, the data type of watching_videos is float, while it should be int
user_data_imputed_data_frame_UNlabeled_training['watching_videos'] = user_data_imputed_data_frame_UNlabeled_training['watching_videos'].apply(lambda x: int(x))
user_data_imputed_data_frame_UNlabeled_training.info()

# Finally lets create the list of list from the row contents
features_list_training = map(list, user_data_imputed_data_frame_UNlabeled_training.values)


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
Its time to train the model
Can you write Python code for it?

from sklearn import tree

classifier = tree.DecisionTreeClassifier() # We create an instance of the Decision tree object
classifier = classifier.fit(features_list_training, label_list_training) # Train the classifier

# Testing data is the first 90 rows
user_data_imputed_data_frame_labeled_testing = user_data_imputed_data_frame_labeled_datetime64_timedelta64_converted.ix[:90]

# take the labels in seprate list
label_list_test = user_data_imputed_data_frame_labeled_testing['label'].values.tolist()
label_list_test = map(int, label_list_test)

# Drop the time and Label columns
user_data_imputed_data_frame_UNlabeled_testing = user_data_imputed_data_frame_labeled_testing.drop(['label'] ,1)
# Check if every thing looks ok
user_data_imputed_data_frame_UNlabeled_testing.info()

# Finally lets create the list of list from the row contents for testing
features_list_test = map(list, user_data_imputed_data_frame_UNlabeled_testing.values)

len(features_list_test)

# the prediction results for first ten values of test data set
print list(classifier.predict(features_list_test[:20]))

# The labels for test data set as labeled by code
print label_list_test[:20]


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
Labelling the Raw data
Now comes the code that will based on the rules mentioned below label the provided data, so it can be used as trainning data for the classifer.
This tables defines the set of rules used to assign labels for Traning data
| label               | age_on_platform      | last_activity             | num_modules_comsumed        | num_of_days_food_tracked | num_glucose_tracked         | watching_videos  |
|---------------------|----------------------|---------------------------|-----------------------------|--------------------------|-----------------------------|------------------|
| Generic (ignore)    | Converted to days    | to be Measured from 16Apr | Good >= 3/week Bad < 3/week | Good >= 30 Bad < 30      | Good >= 4/week Bad < 4/week | Good = 1 Bad = 0 |
| good_new_user = 1       | >= 30 days && < 180  | <= 2 days                 | >= 12                       | >= 20                    | >= 16                       | Good = 1         |
| bad_new_user = 2    | >= 30 days && < 180  | > 2 days                  | < 12                        | < 20                     | < 16                        | Bad = 0          |
| good_mid_term_user = 3  | >= 180 days && < 360 | <= 7 days                 | >= 48                       | >= 30                    | >= 96                       | Good = 1         |
| bad_mid_term_user = 4   | >= 180 days && <360  | > 7 days                  | < 48                        | < 30                     | < 96                        | Bad = 0          |
| good_long_term_user = 5 | >= 360 days          | <= 14 days                | >= 48                       | >= 30                    | >= 192                      | Good = 1         |
| bad_long_term_user = 6 | >= 360 days          | > 14 days                 | < 48                        | < 30                     | < 192                       | Bad = 0          |
Can you write Python code for it?

# This if else section will bin the rows based on the critiria for labels mentioned in the table above

user_data_imputed_data_frame_labeled = user_data_imputed_data_frame

for index, row in user_data_imputed_data_frame.iterrows():

    if row["age_on_platform"] >= np.timedelta64(30, 'D') and row["age_on_platform"] < np.timedelta64(180, 'D'):
        if row['last_activity'] <= np.datetime64(2, 'D') and\
            row['num_modules_consumed'] >= 12 and\
            row['num_of_days_food_tracked'] >= 20 and\
            row['num_glucose_tracked'] >= 16 and\
            row['watching_videos'] == 1:
            user_data_imputed_data_frame_labeled.set_value(index, 'label', 1)
        else:
            user_data_imputed_data_frame_labeled.set_value(index, 'label', 2)

    elif row["age_on_platform"] >= np.timedelta64(180, 'D') and row["age_on_platform"] < np.timedelta64(360, 'D'):
        if row['last_activity'] <= np.datetime64(7, 'D') and\
            row['num_modules_consumed'] >= 48 and\
            row['num_of_days_food_tracked'] >= 30 and\
            row['num_glucose_tracked'] >= 96 and\
            row['watching_videos'] == 1:
            user_data_imputed_data_frame_labeled.set_value(index, 'label', 3)
        else:
            user_data_imputed_data_frame_labeled.set_value(index, 'label', 4)

    elif row["age_on_platform"] >= np.timedelta64(360, 'D'):
        if row['last_activity'] <= np.datetime64(14, 'D') and\
            row['num_modules_consumed'] >= 48 and\
            row['num_of_days_food_tracked'] >= 30 and\
            row['num_glucose_tracked'] >= 192 and\
            row['watching_videos'] == 1:
            user_data_imputed_data_frame_labeled.set_value(index, 'label', 5)
        else:
            user_data_imputed_data_frame_labeled.set_value(index, 'label', 6)
    else:
        user_data_imputed_data_frame_labeled.set_value(index, 'label', 0)

user_data_imputed_data_frame_labeled['label'].unique()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
Now lets add back the useful colums that we had removed from data set, these are

last_activity
age_on_platform
watching_videos
Can you write Python code for it?

# using the Series contructor from Pandas
user_data_imputed_data_frame['last_activity'] = pd.Series(user_data_to_clean['last_activity'])
user_data_imputed_data_frame['age_on_platform'] = pd.Series(user_data_to_clean['age_on_platform'])
# Check if every thing is Ok
user_data_imputed_data_frame.info()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
One major limitation with Sci Kit Learn is with the datatypes it can deal with for features
the data type of last_activity is datetime64 and of age_on_platform is timedelta64
These we need to convert to a numerical type.
Can you write Python code for it?

# Lets start with the column last_activity
# ts = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
# This function takes a datetime64 value and converts it into float value that represents time from epoch
def convert_datetime64_to_from_epoch(dt64):
    ts = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
    return ts
# Lets apply this function on last_activity column
user_data_imputed_data_frame_labeled_datetime64_converted = user_data_imputed_data_frame_labeled
user_data_imputed_data_frame_labeled_datetime64_converted['last_activity'] = user_data_imputed_data_frame_labeled['last_activity'].apply(convert_datetime64_to_from_epoch)
user_data_imputed_data_frame_labeled_datetime64_converted.info()

# Now its time to convert the timedelta64 column named age_on_platform
def convert_timedelta64_to_sec(td64):
    ts = (td64 / np.timedelta64(1, 's'))
    return ts
user_data_imputed_data_frame_labeled_datetime64_timedelta64_converted = user_data_imputed_data_frame_labeled_datetime64_converted
user_data_imputed_data_frame_labeled_datetime64_timedelta64_converted['age_on_platform'] = user_data_imputed_data_frame_labeled_datetime64_converted['age_on_platform'].apply(convert_timedelta64_to_sec)
user_data_imputed_data_frame_labeled_datetime64_timedelta64_converted.info()

user_data_imputed_data_frame_labeled_datetime64_timedelta64_converted.describe()

# Save the labeled data frame as excel file
from pandas import options
options.io.excel.xlsx.writer = 'xlsxwriter'
user_data_imputed_data_frame_labeled_datetime64_timedelta64_converted.to_excel('user_data_imputed_data_frame_labeled.xlsx')


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
The above 3 steps are for KNN based Imputation, did not work well. As visible 804 items could not be imputed for and get replaced with zero
Lets use simpler method that is provided by Scikit Learn itself
Can you write Python code for it?

# Lets use simpler method that is provided by Scikit Learn itself
# import the function
from sklearn.preprocessing import Imputer

# Create an object of class Imputer, with the relvant parameters
imputer_object = Imputer(missing_values='NaN', strategy='mean', axis=0, copy=False)

# Impute the data and save the generated Numpy array
user_data_imputed_np_array = imputer_object.fit_transform(user_data_to_impute)


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
The column name watching_videos (binary - 1 for yes, blank/0 for no) is too long and has special chars, lets change it to watching_videos
Can you write Python code for it?

user_data_to_clean = user_data_raw.rename(columns = {'watching_videos (binary - 1 for yes, blank/0 for no)':'watching_videos'})

# Some basic statistical information on the data
user_data_to_clean.describe()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
The next 3 cells describes the steps to Impute data using KNN strategy, sadly this is not working well for our data set! One possible reason could be that the column is too sparse to find a neighbourer !
In future this method could be combined with the mean imputation method, so the values not covered by KNN get replaced with mean values.
Github repo and Documentation for fancyimpute
Can you write Python code for it?

# Import Imputation method KNN
##from fancyimpute import KNN

# First lets convert the Pandas Dataframe into a Numpy array. We do this since the data frame needs to be transposed,
# which is only possible if the format is an Numpy array.
##user_data_to_impute_np_array = user_data_to_impute.as_matrix()
# Lets Transpose it
##user_data_to_impute_np_array_transposed = user_data_to_impute_np_array.T

# Run the KNN method on the data.   function usage X_filled_knn = KNN(k=3).complete(X_incomplete)
##user_data_imputed_knn_np_array = KNN(k=5).complete(user_data_to_impute_np_array_transposed)


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
The output above for the array says only 2,4,6,0 were selected as labels. Which means there are no good users in all three new, mid, long - term categories.
Consequently either I change the label selection model or get better data (which has good users) :P
Can you write Python code for it?

# Look at basic info for this Labeled data frame
user_data_imputed_data_frame_labeled.info()


I am working on the file "Janacare_User-Segmentation_dataset_Aug2014-Apr2016.ipynb".
The first task is:
the user_data_imputed_np_array is a NumPy array, we need to convert it back to Pandas data frame
Can you write Python code for it?

# create a list of tuples, with the column name and data type for all existing columns in the Numpy array.
# exact order of columns has to be maintained
column_names_of_imputed_np_array = ['num_modules_consumed', 'num_glucose_tracked', 'num_of_days_food_tracked']
# create the Pandas data frame from the Numpy array
user_data_imputed_data_frame = pd.DataFrame(user_data_imputed_np_array, columns=column_names_of_imputed_np_array)
# Check if the data frame created now is proper
user_data_imputed_data_frame.info()


I am working on the file "Jeffreys prior/Jeffreys prior examples.ipynb".
The first task is:
Guassian mean
$f(x\mid\mu) = \frac{e^{-(x - \mu)^2 / 2\sigma^2}}{\sqrt{2 \pi \sigma^2}}$
with $\sigma$ fixed, the Jeffreys prior for the mean $\mu$ is
: $\begin{align} p(\mu) & \propto \sqrt{I(\mu)}
= \sqrt{\operatorname{E}!\left[ \left( \frac{d}{d\mu} \log f(x\mid\mu) \right)^2\right]}
= \sqrt{\operatorname{E}!\left[ \left( \frac{x - \mu}{\sigma^2} \right)^2 \right]} \\
& = \sqrt{\int_{-\infty}^{+\infty} f(x\mid\mu) \left(\frac{x-\mu}{\sigma^2}\right)^2 dx}
=  \sqrt{1/\sigma^2} \propto 1.\end{align}
Gaussian sd
$f(x\mid\sigma) = \frac{e^{-(x - \mu)^2 / 2 \sigma^2}}{\sqrt{2 \pi \sigma^2}}$,
with $\mu$ fixed, the Jeffreys prior for the standard deviation $\sigma > 0$ is
: $\begin{align}p(\sigma) & \propto \sqrt{I(\sigma)}
= \sqrt{\operatorname{E}!\left[ \left( \frac{d}{d\sigma} \log f(x\mid\sigma) \right)^2\right]}
= \sqrt{\operatorname{E}!\left[ \left( \frac{(x - \mu)^2-\sigma^2}{\sigma^3} \right)^2 \right]} \\
& = \sqrt{\int_{-\infty}^{+\infty} f(x\mid\sigma)\left(\frac{(x-\mu)^2-\sigma^2}{\sigma^3}\right)^2 dx}
= \sqrt{\frac{2}{\sigma^2}}
\propto \frac{1}{\sigma}.
\end{align}
Example model
Can you write Python code for it?

data = np.random.normal(10, 2, size=20)
sns.distplot(data)

def jeffreys(value):
    return -theano.tensor.log(value)

with pm.Model() as model:
    sigma = pm.DensityDist('sigma', jeffreys, testval=1)
    mu = pm.DensityDist('mu', jeffreys, testval=1)


I am working on the file "Jeffreys prior/Jeffreys prior examples.ipynb".
The first task is:
Poisson
Can you write Python code for it?

data = np.random.poisson(20, size=10)
sns.distplot(data)

def jeffreys(value):
    return -theano.tensor.log(value)

with pm.Model() as model:
    mean = pm.DensityDist('mean', jeffreys, testval=1)
    poi = pm.Poisson('Poisson', mean, observed=data)

    mean_flat = pm.Uniform('mean_flat', 0, 100)
    poi_flat = pm.Poisson('Poisson_flat', mean_flat, observed=data)

    trace = pm.sample(10000, tune=1000)

pm.summary(trace)

pm.traceplot(trace)

pm.plot_posterior(trace)

# Make custom Jeffreys prior, returns log prior
def jeffreys(value):
    return -theano.tensor.log(value)

# Instantiate model
model = pm.Model()
with model:
    # Priors for unknown model parameters
    sigma = pm.DensityDist('sigma', jeffreys, testval=1)
    slope = pm.Uniform('slope', lower=-1, upper=1, testval=0.5)
    intercept = pm.Uniform('intercept', lower=-10, upper=10, testval=0)

    # Expected value of outcome
    mu = slope * df.parent_bd + intercept

    # Likelihood (sampling distribution) of observations
    y_obs = pm.Normal('y_obs', mu=mu, sd=sigma, observed=df.offspring_bd)

    # draw 10000 posterior samples for burn-in
    trace = pm.sample(10000)

pm.summary(trace, ['slope', 'intercept', 'sigma'])

pm.traceplot(trace, ['slope', 'intercept', 'sigma']);


I am working on the file "Jerk_jounce_etc.ipynb".
The first task is:
Adjust the last entry, to avoid a very long interval.
Can you write Python code for it?

elapsed_time[-1] = 2 * elapsed_time[-2] - elapsed_time[-3]

t = np.linspace(0, elapsed_time[-1], 1000)

v_t = np.interp(t, elapsed_time, v_x)

plt.plot(t, v_t)
plt.show()


I am working on the file "Jerk_jounce_etc.ipynb".
The first task is:
Compute integrals
Use trapezoidal integral approximation, https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.integrate.cumtrapz.html
Can you write Python code for it?

import scipy.integrate

# Displacement, d
d = scipy.integrate.cumtrapz(v_t, t, initial=0)

plt.plot(t, d)
plt.show()

# Absement
abt = scipy.integrate.cumtrapz(d, t, initial=0)

# Absity
aby = scipy.integrate.cumtrapz(abt, t, initial=0)

# Abseleration
abn = scipy.integrate.cumtrapz(aby, t, initial=0)

plt.plot(abn)
plt.show()


I am working on the file "Jerk_jounce_etc.ipynb".
The first task is:
Convert x to m and v to m/s, per the instructions in the blog post about the dataset (modified for metric units).
Can you write Python code for it?

x = (data[:, 0] + 3) * 2.05404
x = x - np.min(x)
v_x = np.mean(data[:, 1:], axis=1) * 0.0380610

plt.plot(x, v_x)
plt.xlabel('Displacement [m]')
plt.ylabel('Velocity [m/s]')
plt.show()


I am working on the file "Jerk_jounce_etc.ipynb".
The first task is:
Note that the sampling was done per unit of displacement; we'd really prefer time. Let's convert it!
Time conversion
Convert to the time domain, since we want derivatives with respect to time, not distance.
Can you write Python code for it?


I am working on the file "Jerk_jounce_etc.ipynb".
The first task is:
Plot everything!
Can you write Python code for it?

plots = {
    'Abseleration': abn,
    'Absity': aby,
    'Absement': abt,
    'Displacement': d,
    'Velocity': v_t,
    'Acceleration': a,
    'Jerk': j,
    'Jounce': s,
#    'Crackle': c,
#    'Pop': p,
}

colors = ['C0', 'C0', 'C0', 'C1', 'C2', 'C2', 'C2', 'C2']

fig, axs = plt.subplots(figsize=(15,15), nrows=len(plots))
pos = 0.01, 0.8
params = dict(fontsize=13)

for i, (k, v) in enumerate(plots.items()):
    ax = axs[i]
    ax.plot(t, v, lw=2, color=colors[i])
    ax.text(*pos, k, transform=ax.transAxes, **params)
#     if np.min(v) < 0:
#         ax.axhline(color='k', lw=0.5, zorder=0)
    if i < len(plots)-1:
        ax.set_xticklabels([])

plt.show()


I am working on the file "Jerk_jounce_etc.ipynb".
The first task is:
That's a boring graph!
Check that derivative of displacement gives back velocity
Use Savitsky-Golay filter for differentiation with some smoothing: https://en.wikipedia.org/wiki/Savitzky%E2%80%93Golay_filter
Can you write Python code for it?

import scipy.signal

dt = t[1] - t[0]

# Check that Savitsky-Golay filter gives velocity from d/dt displacement.
v_ = scipy.signal.savgol_filter(d, delta=dt, window_length=3, polyorder=2, deriv=1)

plt.figure(figsize=(15, 3))
plt.plot(t, v_, lw=3)
plt.plot(t, v_t, '--', lw=3)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Applying the classification model to new data
Now that we have a trained facies classification model we can use it to identify facies in wells that do not have core data.  In this case, we will apply the classifier to two wells, but we could use it on any number of wells for which we have the same set of well logs for input.
This dataset is similar to the training data except it does not have facies labels.  It is loaded into a dataframe called test_data.
Can you write Python code for it?

well_data = pd.read_csv('../validation_data_nofacies.csv')
well_data['Well Name'] = well_data['Well Name'].astype('category')
well_data0 = well_data.query('NM_M==1')
well_data1 = well_data.query('NM_M==2')
well_features0 = well_data0.drop(['Formation', 'Well Name','Depth','NM_M'], axis=1)
well_features1 = well_data1.drop(['Formation', 'Well Name','Depth','NM_M'], axis=1)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
As noted above, the boundaries between the facies classes are not all sharp, and some of them blend into one another.  The error within these 'adjacent facies' can also be calculated.  We define an array to represent the facies adjacent to each other.  For facies label i, adjacent_facies[i] is an array of the adjacent facies labels.
Can you write Python code for it?

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies,offset):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j-offset]
    return total_correct / sum(sum(conf))

print('Facies classification accuracy = %f' % accuracy(conf))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf, adjacent_facies,0))

print('Facies classification accuracy = %f' % accuracy(conf0))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf0, adjacent_facies[0:3],0))

print('Facies classification accuracy = %f' % accuracy(conf1))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf1, adjacent_facies[3:],3))

print('Facies classification accuracy = %f' % accuracy(conf_comb))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf_comb, adjacent_facies,0))


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
At this point I woult like to take a look at the separated main classes.
Can you write Python code for it?

sns.pairplot(plotdata.query('NM_M==1').drop(['Facies','NM_M'],axis=1),
             hue='FaciesLabels', palette=facies_color_map,
             hue_order=list(reversed(facies_labels)))

facies_counts0.plot(kind='bar',color=facies.Colors,
                   title='Distribution of Training Data by Facies')
pd.DataFrame(facies_counts0).T


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Feature Engineering
Can you write Python code for it?

correct_facies_labels  = training_data['Facies'].values
correct_facies_labels0 = training_data0['Facies'].values
#facies_labels=['Sand','Carbs']
correct_facies_labels1 = training_data1['Facies'].values

feature_vectors = training_data.drop(['Formation', 'Well Name','Depth','Facies','NM_M','FaciesLabels'], axis=1)
feature_vectors0 = training_data0.drop(['Formation', 'Well Name','Depth','Facies','NM_M','FaciesLabels'], axis=1)
feature_vectors1 = training_data1.drop(['Formation', 'Well Name','Depth','Facies','NM_M','FaciesLabels'], axis=1)

feature_vectors.describe()

feature_vectors0.describe()

feature_vectors1.describe()


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Finally we predict facies labels for the unknown data, and store the results in a Facies column of the test_data dataframe.
Can you write Python code for it?

#predict facies of unclassified data
y_unknown0 = clf0.predict(X_unknown0)
y_unknown1 = clf1.predict(X_unknown1)
well_data0['Facies'] = y_unknown0
well_data1['Facies'] = y_unknown1
well_data=pd.concat([well_data0,well_data1], axis=0)
well_data

well_data['Well Name'].unique()


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Here we can see aparent outliers in GR, PhiND, PE and DeltaPhi
* GR > 350
* DeltaPhi < -20
* PhiND > 45
* PE < 2
I have decided to keep GR as it is a general trend in the data (and not unheard of in that rock). DeltaPhi also fits a general trend and seems to be a feature. PhiND and PE do not lie in a trend or cluster and will therefore be removed.
Can you write Python code for it?

processed_data0 = processed_data.query('NM_M==1 and not (PHIND > 60 or PE < 1 or GR > 150) and Facies < 3')
processed_data1 = processed_data.query('NM_M==2 and (PHIND < 45 or PE > 2) and Facies > 2')

processed_data = pd.concat([processed_data0,processed_data1])
print("Dropped {0} rows due to outliers and misclassification.".format(rows-len(processed_data.index),))

sns.set()
plotdata=processed_data.drop(['Well Name','Facies','Formation','Depth','NM_M','RELPOS'],axis=1).dropna()

sns.pairplot(plotdata,
             hue='FaciesLabels', palette=facies_color_map,
             hue_order=list(reversed(facies_labels)))

#switch back to default matplotlib plot style
mpl.rcParams.update(inline_rc)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Here's the SHRIMPLIN well for your vieling pleasure.
Can you write Python code for it?

make_facies_log_plot(
    processed_data[processed_data['Well Name'] == 'SHRIMPLIN'],
    facies.Colors)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
In addition to individual wells, we can look at how the various facies are represented by the entire training set.  Let's plot a histogram of the number of training examples for each facies class.
Can you write Python code for it?

#count the number of unique entries for each facies, sort them by
#facies number (instead of by number of entries)
facies_counts = processed_data['Facies'].value_counts().sort_index()
tmp = processed_data.query('NM_M==1')
facies_counts0 = tmp['Facies'].value_counts().sort_index()

tmp = processed_data.query('NM_M==2')
facies_counts1 = tmp['Facies'].value_counts().sort_index()

#use facies labels to index each count
facies_counts.index = facies_labels

facies_counts.plot(kind='bar',color=facies.Colors,
                   title='Distribution of Training Data by Facies')
pd.DataFrame(facies_counts).T


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
In this view it becomes obvious that GR, PHIND and PE have some outliers at:

GR > 150
PhiND > 60
PE < 1
Can you write Python code for it?

sns.pairplot(plotdata.query('NM_M==2').drop(['Facies','NM_M'],axis=1),
             hue='FaciesLabels', palette=facies_color_map,
             hue_order=list(reversed(facies_labels)))

facies_counts1.plot(kind='bar',color=facies.Colors,
                   title='Distribution of Training Data by Facies')
pd.DataFrame(facies_counts1).T


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Model parameter selection
The classifier so far has been built with the default parameters.  However, we may be able to get improved classification results with optimal parameter choices.
We will consider two parameters.  The parameter C is a regularization factor, and tells the classifier how much we want to avoid misclassifying training examples.  A large value of C will try to correctly classify more examples from the training set, but if C is too large  it may 'overfit' the data and fail to generalize when classifying new data. If C is too small then the model will not be good at fitting outliers and will have a large error on the training set.
The SVM learning algorithm uses a kernel function to compute the distance between feature vectors.  Many kernel functions exist, but in this case we are using the radial basis function rbf kernel (the default).  The gamma parameter describes the size of the radial basis functions, which is how far away two vectors in the feature space need to be to be considered close.
We will train a series of classifiers with different values for C and gamma.  Two nested loops are used to train a classifier for every possible combination of values in the ranges specified.  The classification accuracy is recorded for each combination of parameter values.  The results are shown in a series of plots, so the parameter values that give the best classification accuracy on the test set can be selected.
This process is also known as 'cross validation'.  Often a separate 'cross validation' dataset will be created in addition to the training and test sets to do model selection.  For this tutorial we will just use the test set to choose model parameters.
Can you write Python code for it?

from sklearn.model_selection import RandomizedSearchCV,LeaveOneGroupOut
from scipy.stats import uniform as sp_uniform

Fscorer = make_scorer(f1_score, average = 'micro')
Ascorer = make_scorer(accuracy_score)

clf = svm.SVC(cache_size = 800, random_state=1)
clf0 = svm.SVC(cache_size = 800, random_state=1)
clf1 = svm.SVC(cache_size = 800, random_state=1)

#parm_grid={'kernel': ['linear', 'rbf'],
#            'C': [0.5, 1, 10, 50],
#            'gamma':[0.0001, 0.01, 1, 10]}

param_dist = {"kernel": ['rbf'],
              "C": sp_uniform(0.0001,10),
              "gamma": sp_uniform(0.0001, 10)}

n_iter_search = 20

random_search0 = RandomizedSearchCV(clf0, param_distributions=param_dist,scoring = Fscorer,n_iter=n_iter_search)
random_search0.fit(scaled_features0,correct_facies_labels0)
print('Best score for NM: {}'.format(random_search0.best_score_))
print('Best parameters for NM: {}'.format(random_search0.best_params_))
clf0 = random_search0.best_estimator_
random_search0.best_estimator_

random_search1 = RandomizedSearchCV(clf1, param_distributions=param_dist,scoring = Fscorer,n_iter=n_iter_search)
random_search1.fit(scaled_features1,correct_facies_labels1)
print('Best score for random: {}'.format(random_search1.best_score_))
print('Best parameters for random: {}'.format(random_search1.best_params_))
clf1 = random_search1.best_estimator_
random_search1.best_estimator_

random_search =  (clf, param_distributions=param_dist,scoring = Fscorer,n_iter=n_iter_search)
random_search.fit(scaled_features,correct_facies_labels)
print('Best score for combo: {}'.format(random_search.best_score_))
print('Best parameters for combo: {}'.format(random_search.best_params_))
clf = random_search.best_estimator_
random_search.best_estimator_


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Okay, take it up a notch.
Can you write Python code for it?

clf0.fit(scaled_features0, correct_facies_labels0)

clf1.fit(scaled_features1, correct_facies_labels1)

clf.fit(scaled_features, correct_facies_labels)

cv_conf = confusion_matrix(np.concatenate([correct_facies_labels0,correct_facies_labels1]), np.concatenate([clf0.predict(scaled_features0),clf1.predict(scaled_features1)]))
cv_conf_comb = confusion_matrix(correct_facies_labels, clf.predict(scaled_features))

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf, adjacent_facies,0))

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf_comb))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf_comb, adjacent_facies,0))


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Scikit includes a preprocessing module that can 'standardize' the data (giving each variable zero mean and unit variance, also called whitening).  Many machine learning algorithms assume features will be standard normally distributed data (ie: Gaussian with zero mean and unit variance).  The factors used to standardize the training set must be applied to any subsequent feature set that will be input to the classifier.  The StandardScalar class can be fit to the training set, and later used to standardize any training data.
Can you write Python code for it?

from sklearn import preprocessing

scaler = preprocessing.StandardScaler().fit(feature_vectors)
scaled_features = scaler.transform(feature_vectors)

scaler0 = preprocessing.StandardScaler().fit(feature_vectors0)
scaled_features0 = scaler0.transform(feature_vectors0)

scaler1 = preprocessing.StandardScaler().fit(feature_vectors1)
scaled_features1 = scaler1.transform(feature_vectors1)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Shrimplin:
* Longitude: -100.98329
* Latitude: 37.98126
ALEXANDER D
* Longitude: -95.3534525
* Latitude: 37.6607787
SHANKLE
* Longitude: -101.3891894
* Latitude: 38.0633727
LUKE G U
* Longitude: -101.6073725
* Latitude: 37.4537739
KIMZEY A
* Longitude: -101.39697
* Latitude: 37.12289
CROSS H CATTLE
* Longitude: -101.6464517
* Latitude: 37.9105826
NOLAN
* Longitude: -101.0451641
* Latitude: 37.7866294
Recruit F9
NEWBY
* Longitude: -101.5847635
* Latitude: 37.5406739
CHURCHMAN BIBLE
* Longitude: -101.1060761
* Latitude: 37.3497658
STUART
* Longitude: -101.1391063
* Latitude: 37.4857262
CRAWFORD
* Longitude: -101.1494994
* Latitude: 37.1893654
Can you write Python code for it?


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
The data needs to be scaled using the same constants we used for the training data.
Can you write Python code for it?

X_unknown0 = scaler0.transform(well_features0)
X_unknown1 = scaler1.transform(well_features1)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
These facies gradually blend into one another. We can see a marine non-marine neighbourhood already.
Can you write Python code for it?

pd.DataFrame({'Neighbours' : [facies.ShortName[x] for x in facies.Neighbours]},index=facies_labels)


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
Verbatim from Brendon Hall:

This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.

The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
Can you write Python code for it?


I am working on the file "JesperDramsch/Facies_classification_NMM_Split-Jesper.ipynb".
The first task is:
We can use the well log plot to view the classification results along with the well logs.
Can you write Python code for it?

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'STUART'],
    facies_colors=facies.Colors)

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'CRAWFORD'],
    facies_colors=facies.Colors)


I am working on the file "Joe #1-Implementation of Easy21/#1-Implementation of Easy21.ipynb".
The first task is:
Each draw from the deck results in a colour of red (probability 1/3) or black (probability 2/3).
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib notebook
plt.figure(2)

values = []
for i in xrange(0,100000):
    if (Card().is_black):
        values.append(0.6666666)
    else:
        values.append(0.3333333)

plt.title('Test; red (probability 1/3) or black (probability 2/3)')
plt.hist(values)

         # , c='g', s=20, alpha=0.25, label='true positive')
plt.show()
plt.savefig("#1-test2.png")


I am working on the file "Joe #1-Implementation of Easy21/#1-Implementation of Easy21.ipynb".
The first task is:
Test: If the player sticks then the dealer starts taking turns. The dealer always sticks on any sum of 17 or greater, and hits otherwise. If the dealer goes bust, then the player wins; otherwise, the outcome – win (reward +1), lose (reward -1), or draw (reward 0) – is the player with the largest sum.
Can you write Python code for it?

def play_test_player_stick():
    s = State(Card(True),Card(True))
    a = Actions.hit

    a = Actions.stick
    while not s.term:
        s, r = step(s, a)
        # print ("state = %s, %s, %s" % (s.player, s.dealer, s.term))
    return s, r

import matplotlib.pyplot as plt
%matplotlib notebook
plt.figure(4)

values = []
for i in xrange(0,100000):
    s, r = play_test_player_stick()
    if s.dealer > 21 or s.dealer < 1:
        if r == 1:
            values.append(1)
        else:
            print "error, player should have won"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-1)
    elif s.player == s.dealer:
        if r == 0:
            values.append(1)
        else:
            print "error, player should have drawn"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-2)
    elif s.player > s.dealer:
        if r == 1:
            values.append(1)
        else:
            print "error, player should have won"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-3)
    elif s.player < s.dealer:
        if r == -1:
            values.append(1)
        else:
            print "error, player should have lost"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-4)
    else:
        print "all cases should have been dealt with"
        print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
        values.append(-5)

plt.title('Test; player sticks')
plt.hist(values)
plt.show()
plt.savefig("#1-test4.png")


I am working on the file "Joe #2 Monte-Carlo Control in Easy21/easy21 tests.ipynb".
The first task is:
Test: If the player sticks then the dealer starts taking turns. The dealer always sticks on any sum of 17 or greater, and hits otherwise. If the dealer goes bust, then the player wins; otherwise, the outcome – win (reward +1), lose (reward -1), or draw (reward 0) – is the player with the largest sum.
Can you write Python code for it?

def play_test_player_stick():
    s = State(Card(True),Card(True))
    a = Actions.hit
    e = Environment()

    a = Actions.stick
    while not s.term:
        s, r = e.step(s, a)
        # print ("state = %s, %s, %s" % (s.player, s.dealer, s.term))
    return s, r

import matplotlib.pyplot as plt
%matplotlib notebook
plt.figure(4)

values = []
for i in xrange(0,100000):
    s, r = play_test_player_stick()
    if s.dealer > 21 or s.dealer < 1:
        if r == 1:
            values.append(1)
        else:
            print "error, player should have won"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-1)
    elif s.player == s.dealer:
        if r == 0:
            values.append(1)
        else:
            print "error, player should have drawn"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-2)
    elif s.player > s.dealer:
        if r == 1:
            values.append(1)
        else:
            print "error, player should have won"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-3)
    elif s.player < s.dealer:
        if r == -1:
            values.append(1)
        else:
            print "error, player should have lost"
            print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
            values.append(-4)
    else:
        print "all cases should have been dealt with"
        print ("state = %s, %s, %s. result = %s" % (s.player, s.dealer, s.term, r))
        values.append(-5)

plt.title('Test; player sticks')
plt.hist(values)
plt.show()
plt.savefig("#1-test4.png")


I am working on the file "Joe #2 Monte-Carlo Control in Easy21/easy21 tests.ipynb".
The first task is:
Test: If the player’s sum exceeds 21, or becomes less than 1, then she “goes bust” and loses the game (reward -1)
Can you write Python code for it?

def play_test_player_bust():
    s = State(Card(True),Card(True))
    a = Actions.hit
    e = Environment()

    while not s.term:
        s, r = e.step(s, a)
        # print ("state = %s, %s, %s" % (s.player, s.dealer, s.term))
    return s, r

import matplotlib.pyplot as plt
%matplotlib notebook
plt.figure(3)

values = []
for i in xrange(0,100000):
    s, r = play_test_player_bust()
    if s.player > 21:
        values.append(1)
    elif s.player < 1:
        values.append(1)
    else:
        values.append(-1)
        print "error!!!!"

plt.title('Test; player busts > 21 or <1')
plt.hist(values)

         # , c='g', s=20, alpha=0.25, label='true positive')
plt.show()
plt.savefig("#1-test3.png")


I am working on the file "Joe #3 TD Learning in Easy21/Joe #3 TD Learning in Easy21.ipynb".
The first task is:
For λ = 0 and λ = 1 only, plot the learning curve of mean-squared error against episode number.
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib.animation as animation
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
%matplotlib inline

fig = plt.figure("N100")
surf = plt.plot(lambdas[1:10], sme_list[1:10])

plt.show()

N0 = 100
l = 0.0
learning_rate = []
learning_rate_i = []
n_elements = len(mc_agent.Q)
agent = Sarsa_Agent(Environment(), N0, l)
for i in xrange(1000):
    learning_rate_i.append(i)
    agent.train(1)
    sme = np.sum(np.square(agent.Q-mc_agent.Q))/float(1000)

    learning_rate.append(sme)
fig = plt.figure("0.0")
surf = plt.plot(learning_rate_i, learning_rate)
plt.show()

N0 = 100
l = 1.0
learning_rate = []
learning_rate_i = []
agent = Sarsa_Agent(Environment(), N0, l)
for i in xrange(1000):
    learning_rate_i.append(i)
    agent.train(1)
    sme = np.sum(np.square(agent.Q-mc_agent.Q))/float(1000)

    learning_rate.append(sme)
fig = plt.figure("0.0")
surf = plt.plot(learning_rate_i, learning_rate)
plt.show()


I am working on the file "Joe #3 TD Learning in Easy21/Joe #3 TD Learning in Easy21.ipynb".
The first task is:
Implement Sarsa(λ) in 21s.

[x] Initialise the value function to zero.
[x] Use the same step-size and exploration schedules as in the previous section.
[x] Run the algorithm with parameter values λ ∈ {0, 0.1, 0.2, ..., 1}.
[x] Stop each run after 1000 episodes
    and report the mean-squared error over all states and actions,
    comparing the true values Q∗(s,a) computed in the previous section with the estimated values Q(s, a) computed by Sarsa.
[x] Plot the mean- squared error against λ.
[x] For λ = 0 and λ = 1 only, plot the learning curve of mean-squared error against episode number.
Can you write Python code for it?

class Sarsa_Agent:
    def __init__(self, environment, n0, mlambda):
        self.n0 = float(n0)
        self.env = environment
        self.mlambda = mlambda

        # N(s) is the number of times that state s has been visited
        # N(s,a) is the number of times that action a has been selected from state s.
        self.N = np.zeros((self.env.dealer_values_count,
                           self.env.player_values_count,
                           self.env.actions_count))

        self.Q = np.zeros((self.env.dealer_values_count,
                           self.env.player_values_count,
                           self.env.actions_count))
        self.E = np.zeros((self.env.dealer_values_count, self.env.player_values_count, self.env.actions_count))

        # Initialise the value function to zero.
        self.V = np.zeros((self.env.dealer_values_count, self.env.player_values_count))

        self.count_wins = 0
        self.iterations = 0

        # get optimal action, with epsilon exploration (epsilon dependent on number of visits to the state)
    # ε-greedy exploration strategy with εt = N0/(N0 + N(st)),
    def train_get_action(self, state):
        dealer_idx = state.dealer-1
        player_idx = state.player-1
        try:
            n_visits = sum(self.N[dealer_idx, player_idx, :])
        except:
            n_visits = 0

        # epsilon = N0/(N0 + N(st)
        curr_epsilon = self.n0 / (self.n0 + n_visits)

        # epsilon greedy policy
        if random.random() < curr_epsilon:
            r_action = Actions.hit if random.random()<0.5 else Actions.stick
#             if (dealer_idx == 0 and player_idx == 0):
#                 print ("epsilon:%s, random:%s " % (curr_epsilon, r_action))
            return r_action
        else:
            action = Actions.to_action(np.argmax(self.Q[dealer_idx, player_idx, :]))
#             if (dealer_idx == 0 and player_idx == 0):
#                 print ("epsilon:%s Qvals:%s Q:%s" % (curr_epsilon, self.Q[dealer_idx,...


I am working on the file "Joe #3 TD Learning in Easy21/easy21 tests.ipynb".
The first task is:
Each draw from the deck results in a colour of red (probability 1/3) or black (probability 2/3).
Can you write Python code for it?


I am working on the file "Joe #3 TD Learning in Easy21/easy21 tests.ipynb".
The first task is:
Test: If the player’s sum exceeds 21, or becomes less than 1, then she “goes bust” and loses the game (reward -1)
Can you write Python code for it?


I am working on the file "Joe #3 TD Learning in Easy21/easy21 tests.ipynb".
The first task is:
from 1st module
Test; Each draw from the deck results in a value between 1 and 10 (uniformly distributed)
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib notebook
plt.figure(1)

values = []
for i in xrange(0,100000):
    values.append(Card().absolute_value)
    # values.append(random.randint(1,10))

plt.title('Test; Each draw from the deck results in a value between 1 and 10 (uniformly distributed)')
plt.hist(values)

         # , c='g', s=20, alpha=0.25, label='true positive')
plt.show()
plt.savefig("#1-test1.png")


I am working on the file "John Doty's Global Calibration of the Housekeeping Data Collection.ipynb".
The first task is:
Once again, we run the calculation; this should be $\approx 1$
Can you write Python code for it?


I am working on the file "John Doty's Global Calibration of the Housekeeping Data Collection.ipynb".
The first task is:
We assume that there is a slight global error in the housekeeping that may be corrected by a linear transformation:
$$ f(x) := m \cdot x + c $$
To calculus $c$, we can average the biases in the housekeeping, but we must first convert them to ADUs from uA by using the unscale_value function.
Note that because of statistical variation, we collect 50 samples:
Can you write Python code for it?

def estimate_c_param(fpe,samples=50):
    from tessfpe.dhu.house_keeping import unscale_value
    from tessfpe.data.housekeeping_channels import housekeeping_channels
    sample_data = []
    for _ in range(samples):
        analogue_house_keeping = fpe.house_keeping["analogue"]
        biases = [unscale_value(analogue_house_keeping[k],
                                16,
                                housekeeping_channels[k]["low"],
                                housekeeping_channels[k]["high"])
                  for k in fpe1.house_keeping["analogue"]
                  if 'bias' in k]
        for b in biases:
            sample_data.append(b)

    return sum(sample_data) / len(sample_data)


I am working on the file "John/numba.ipynb".
The first task is:
After JIT compilation, function execution speed is about the same as FORTRAN.
But remember, JIT compilation for Python is still limited --- see here
If these limitations frustrate you, then try Julia.
Problem 2: Brute Force Optimization
The problem is to maximize the function
$$ f(x, y) = \frac{\cos \left(x^2 + y^2 \right)}{1 + x^2 + y^2} + 1$$
using brute force --- searching over a grid of $(x, y)$ pairs.
Can you write Python code for it?

def f(x, y):
    return np.cos(x**2 + y**2) / (1 + x**2 + y**2) + 1

from mpl_toolkits.mplot3d.axes3d import Axes3D
from matplotlib import cm

gridsize = 50
gmin, gmax = -3, 3
xgrid = np.linspace(gmin, gmax, gridsize)
ygrid = xgrid
x, y = np.meshgrid(xgrid, ygrid)

# === plot value function === #
fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x,
                y,
                f(x, y),
                rstride=2, cstride=2,
                cmap=cm.jet,
                alpha=0.4,
                linewidth=0.05)

ax.scatter(x, y, c='k', s=0.6)

ax.scatter(x, y, f(x, y), c='k', s=0.6)

ax.view_init(25, -57)
ax.set_zlim(-0, 2.0)
ax.set_xlim(gmin, gmax)
ax.set_ylim(gmin, gmax)

plt.show()


I am working on the file "John/numba.ipynb".
The first task is:
Here's a function that simulates for n periods, starting from x0, and returns only the final value:
Can you write Python code for it?

def quad(x0, n):
    x = x0
    for i in range(1, n):
        x = α * x * (1 - x)
    return x


I am working on the file "John/numba.ipynb".
The first task is:
Here's a typical time series:
Can you write Python code for it?

n = 200
x =  np.empty(n)
x[0] = 0.2
for t in range(n-1):
    x[t+1] = α * x[t] * (1 - x[t])

plt.plot(x)
plt.show()


I am working on the file "John/numba.ipynb".
The first task is:
Let's see how fast this runs:
Can you write Python code for it?


I am working on the file "John/numba.ipynb".
The first task is:
Now let's do the same thing in Python using Numba's JIT compilation:
Can you write Python code for it?

quad_jitted = jit(quad)

tic()
x = quad_jitted(0.2, n)
toc()

tic()
x = quad_jitted(0.2, n)
toc()


I am working on the file "John/numba.ipynb".
The first task is:
Now let's try this in FORTRAN.
Note --- this step is intended to be a demo and will only execute if

you have the file fastquad.f90 in your pwd
you have a FORTRAN compiler installed and modify the compilation code below appropriately
Can you write Python code for it?

!cat fastquad.f90

!gfortran -O3 fastquad.f90

!./a.out


I am working on the file "John/numba.ipynb".
The first task is:
Numba for vectorization with automatic parallelization - even faster:
Can you write Python code for it?

@vectorize('float64(float64, float64)', target='parallel')
def f_par(x, y):
    return np.cos(x**2 + y**2) / (1 + x**2 + y**2) + 1

x, y = np.meshgrid(grid, grid)

np.max(f_par(x, y))

tic()
np.max(f_par(x, y))
toc()


I am working on the file "John/numba.ipynb".
The first task is:
Vectorized code
Can you write Python code for it?

grid = np.linspace(-3, 3, 10000)

x, y = np.meshgrid(grid, grid)

tic()
np.max(f(x, y))
toc()


I am working on the file "John/plots.ipynb".
The first task is:
Exercise 1
Plot the tent map
$$ f(x) = \mu \min{x, 1 - x} $$
on the interval $[0, 1]$ when $\mu = 1.8$
Can you write Python code for it?

for i in range(15):
    print("solution below")

μ = 1.8
x = np.linspace(0, 1, 200)
y = μ * np.minimum(x, 1 - x)

fig, ax = plt.subplots()
ax.plot(x, y)
plt.show()


I am working on the file "John/plots.ipynb".
The first task is:
Exercise 3
The next code shows how to build a histogram.
Can you write Python code for it?

z = np.random.randn(1000)
fig, ax = plt.subplots()
ax.hist(z, bins=40)
plt.show()


I am working on the file "John/plots.ipynb".
The first task is:
Here's a plot with two curves.
Can you write Python code for it?

y1 = np.minimum(np.sin(x), np.cos(x))
y2 = np.maximum(np.sin(x), np.cos(x))

fig, ax = plt.subplots()
ax.plot(x, y1, label='y1')
ax.plot(x, y2, label='y2')
ax.legend()

plt.show()


I am working on the file "John/plots.ipynb".
The first task is:
Now compute the time series of length 500 given by
$$ x_{t+1} = f(x_t) $$
where $f$ is as given above and $x_0 = 0.2$.  Plot $x_t$ against $t$.
Can you write Python code for it?

for i in range(15):
    print("solution below")

n = 500
x = np.empty(n)
x[0] = 0.2
for t in range(n-1):
    x[t+1] = μ * min(x[t], 1 - x[t])

fig, ax = plt.subplots()
ax.plot(range(n), x)
plt.show()


I am working on the file "John/plots.ipynb".
The first task is:
The next code shows what the range() function does:
Can you write Python code for it?


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
(1) Create a 10x10 grid to visualize 10 examples of each digit. Python hints:

plt.rc() for setting the colormap, for example to black and white
plt.subplot() for creating subplots
plt.imshow() for rendering a matrix
np.array.reshape() for reshaping a 1D feature vector into a 2D matrix (for rendering)
Can you write Python code for it?

#def P1(num_examples=10):

### STUDENT START ###
# Credit where due... some inspiration drawn from:
# https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/fig/mnist.py

# example_as_pixel_matrix():
#   transforms a 784 element pixel into a 28 x 28 pixel matrix
def example_as_pixel_matrix(example):
    return np.reshape(example, (-1, 28))

# add_example_to_figure():
#   given an existing figure, number of rows, columns, and position,
#   adds a subplot with the example to the figure
def add_example_to_figure(example,
                     figure,
                     subplot_rows,
                     subplot_cols,
                     subplot_number):
    matrix = example_as_pixel_matrix(example)

    subplot = figure.add_subplot(subplot_rows, subplot_cols, subplot_number)
    subplot.imshow(matrix, cmap='Greys', interpolation='Nearest')
    # disable tick marks
    subplot.set_xticks(np.array([]))
    subplot.set_yticks(np.array([]))

# plot_examples():
#   given a matrix of examples (digit, example#) => example,
#   plots it with digits as rows and examples as columns
def plot_examples(examples):

    figure = plt.figure()

    shape = np.shape(examples)
    rows = shape[0]
    columns = shape[1]

    subplot_index = 1

    for digit, examples_for_digit in enumerate(examples):
        for example_index, example in enumerate(examples_for_digit):
            add_example_to_figure(example,
                                  figure,
                                  rows,
                                  columns,
                                  subplot_index
                                 )
            subplot_index = subplot_index + 1

    figure.tight_layout()
    plt.show()

# plot_one_example():
#   given an example, plots only that example, typically
#   for debugging or diagnostics
def plot_one_example(example):
    examples = [ [ example ] ]
    plot_examples(examples)

# select_indices_of_digit():
#   given an array of digit lables, selects the...


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
(4) Fit a regression model that predicts accuracy from training size. What does it predict for n=60000? What's wrong with using regression here? Can you apply a transformation that makes the predictions more reasonable?

Remember that the sklearn fit() functions take an input matrix X and output vector Y. So each input example in X is a vector, even if it contains only a single value.
Can you write Python code for it?

#def P4():

### STUDENT START ###

from sklearn.linear_model import LogisticRegression

# fit_linear_regression():
#   given arrays of training data sizes and corresponding accuracies,
#   train and return a linear regression model for predicting accuracies
def fit_linear_regression(train_sizes, accuracies):
    train_sizes_matrix = [ [ train_size ] for train_size in train_sizes ]

    linear = LinearRegression()
    linear.fit(train_sizes_matrix, accuracies)

    return linear

# fit_logistic_regression():
#   given arrays of training data sizes and corresponding accuracies,
#   train and return a logistic regression model for predicting accuracies
def fit_logistic_regression(train_sizes, accuracies):
    train_sizes_matrix = [ [ train_size ] for train_size in train_sizes ]

    logistic = LogisticRegression()
    logistic.fit(train_sizes_matrix, accuracies)

    return logistic

def P4():
    full_training_size = 60000

    linear = fit_linear_regression(train_sizes, accuracies)
    linear_prediction = linear.predict(full_training_size)
    print('Linear model prediction for '
          + str(full_training_size) + ' : ' + str(linear_prediction[0]))

    logistic = fit_logistic_regression(train_sizes, accuracies)
    logistic_prediction = logistic.predict(full_training_size)
    print('Logistic model prediction for '
          + str(full_training_size) + ' : ' + str(logistic_prediction[0]))

P4()

### STUDENT END ###

#P4()


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
(6) A common image processing technique is to smooth an image by blurring. The idea is that the value of a particular pixel is estimated as the weighted combination of the original value and the values around it. Typically, the blurring is Gaussian -- that is, the weight of a pixel's influence is determined by a Gaussian function over the distance to the relevant pixel.
Implement a simplified Gaussian blur by just using the 8 neighboring pixels: the smoothed value of a pixel is a weighted combination of the original value and the 8 neighboring values. Try applying your blur filter in 3 ways:
- preprocess the training data but not the dev data
- preprocess the dev data but not the training data
- preprocess both training and dev data
Note that there are Guassian blur filters available, for example in scipy.ndimage.filters. You're welcome to experiment with those, but you are likely to get the best results with the simplified version I described above.
Can you write Python code for it?

import itertools

# blur():
#   blurs an image by averaging adjacent pixels
def blur(image):
    pixel_matrix = example_as_pixel_matrix(image)
    blurred_image = []
    rows, columns = np.shape(pixel_matrix)

    for row in range(rows):
        for column in range(columns):
            # take the mean of the 9-pixel neighborhood (in clause)
            # but guard against running off the edges of the matrix (if clause)
            value = np.mean(list(
                pixel_matrix[i][j]
                for i, j
                in itertools.product(
                    range(row - 1, row + 2),
                    range(column - 1, column + 2)
                )
                if (i >= 0) and (j >= 0) and (i < rows) and (j < columns)
            ))

            blurred_image.append(value)

    return blurred_image

# blur_images():
#   blurs a collection of images
def blur_images(images):
    blurred = [ blur(image) for image in images ]
    return blurred

# Do this in batches since iPythonNB seems to hang on large batches
train_data_0k = train_data[:10000]
blurred_train_data_0k = blur_images(train_data_0k)

train_data_1k = train_data[10000:20000]
blurred_train_data_1k = blur_images(train_data_1k)

train_data_2k = train_data[20000:30000]
blurred_train_data_2k = blur_images(train_data_2k)

train_data_3k = train_data[30000:40000]
blurred_train_data_3k = blur_images(train_data_3k)

train_data_4k = train_data[40000:50000]
blurred_train_data_4k = blur_images(train_data_4k)

train_data_5k = train_data[50000:60000]
blurred_train_data_5k = blur_images(train_data_5k)

blurred_dev_data = blur_images(dev_data)

blurred_train_data = (
    blurred_train_data_0k
    + blurred_train_data_1k
    + blurred_train_data_2k
    + blurred_train_data_3k
    + blurred_train_data_4k
    + blurred_train_data_5k
)

#def P6():

### STUDENT START ###

def P6():
    k = 1
    neigh_blurred_train = train_k_nearest_neighbors(k, blurred_train_data, train_labels)
    neigh_unblurred_train =...


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER:
(10) Because Naive Bayes is a generative model, we can use the trained model to generate digits. Train a BernoulliNB model and then generate a 10x20 grid with 20 examples of each digit. Because you're using a Bernoulli model, each pixel output will be either 0 or 1. How do the generated digits compare to the training digits?

You can use np.random.rand() to generate random numbers from a uniform distribution
The estimated probability of each pixel is stored in feature_log_prob_. You'll need to use np.exp() to convert a log probability back to a probability.
Can you write Python code for it?

#def P10(num_examples):

### STUDENT START ###

def generate_example(log_probabilities):
    pixels = [
        1.0 if np.random.rand() <= np.exp( log_probability ) else 0.0
        for log_probability in log_probabilities
    ]

    return pixels

# more than 10 x 10 gets scaled too small
def plot_10_examples(binary_naive_bayes):
    per_digit_log_probabilities = binary_naive_bayes.feature_log_prob_

    examples = [
        [
            generate_example(per_digit_log_probabilities[digit])
            for example in range(10)
        ]
        for digit in range(10)
    ]

    plot_examples(examples)

def P10(num_examples):
    binarized_train_data = binarize_examples(train_data)
    binary_naive_bayes = BernoulliNB().fit(binarized_train_data, train_labels)

    page = 0

    while page < num_examples:
        plot_10_examples(binary_naive_bayes)
        page = page + 10

P10(20)

### STUDENT END ###

#P10(20)


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER:
(8) Use GridSearchCV to perform a search over values of alpha (the Laplace smoothing parameter) in a Bernoulli NB model. What is the best value for alpha? What is the accuracy when alpha=0? Is this what you'd expect?

Note that GridSearchCV partitions the training data so the results will be a bit different than if you used the dev data for evaluation.
Can you write Python code for it?

#def P8(alphas):

### STUDENT START ###

def P8(alphas):
    binarized_train_data = binarize_examples(train_data)

    bernoulli_naive_bayes = BernoulliNB()

    grid_search = GridSearchCV(bernoulli_naive_bayes, alphas, verbose = 3)
    grid_search.fit(binarized_train_data, train_labels)

    return grid_search

alphas = {'alpha': [0.0, 0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 10.0]}
nb = P8(alphas)
print nb.best_params_

### STUDENT END ###

#alphas = {'alpha': [0.0, 0.0001, 0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 10.0]}
#nb = P8(alphas)

#print nb.best_params_


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER: Blurring the training but not the development data
(7) Fit a Naive Bayes classifier and report accuracy on the dev data. Remember that Naive Bayes estimates P(feature|label). While sklearn can handle real-valued features, let's start by mapping the pixel values to either 0 or 1. You can do this as a preprocessing step, or with the binarize argument. With binary-valued features, you can use BernoulliNB. Next try mapping the pixel values to 0, 1, or 2, representing white, grey, or black. This mapping requires MultinomialNB. Does the multi-class version improve the results? Why or why not?
Can you write Python code for it?

#def P7():

### STUDENT START ###

from sklearn.metrics import accuracy_score

# binarize_example():
#   Turn all pixels below 0.5 (or threshold) -> 0, greater -> 1
def binarize_example(example, threshold = 0.5):
    binarized = [ 1 if value > threshold else 0 for value in example ]
    return binarized

# binarize_examples():
#   Apply binarization to a set of example
def binarize_examples(examples, threshold = 0.5):
    binarized = [ binarize_example(example, threshold) for example in examples ]
    return binarized

# ternarize_example():
#   Turn all pixels below 1/3 (or threshold) -> 0, 1/3 through 2/3 -> 1, greater -> 2
def ternarize_example(example, threshold_low = 0.33333333, threshold_high = 0.66666666):
    ternarized = [
        0 if value < threshold_low else 1 if value < threshold_high else 2
        for value in example
    ]
    return ternarized

# ternarize_examples():
#   Apply ternarization to a set of example
def ternarize_examples(examples, threshold_low = 0.33333333, threshold_high = 0.66666666):
    ternarized = [
        ternarize_example(example, threshold_low, threshold_high)
        for example in examples
    ]
    return ternarized

def P7():
    binarized_train_data = binarize_examples(train_data)

    binary_naive_bayes = BernoulliNB()
    binary_naive_bayes.fit(binarized_train_data, train_labels)

    binarized_dev_data = binarize_examples(dev_data)
    binary_naive_bayes_predicted = binary_naive_bayes.predict(binarized_dev_data)

    target_names = [ str(i) for i in range(10) ]

    print '============ Classification report for binarized ============'
    print ''
    print(classification_report(
            dev_labels,
            binary_naive_bayes_predicted,
            target_names = target_names))
    print ' Accuracy score: '
    print(accuracy_score(dev_labels, binary_naive_bayes_predicted, normalize = True))

    ternarized_train_data = ternarize_examples(train_data)

    ternary_naive_bayes = MultinomialNB()
   ...


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER: Many of the generated digits are recognizable. However, they lack the connected lines of handdrawn digits because each pixel is sampled independently.
(11) Remember that a strongly calibrated classifier is rougly 90% accurate when the posterior probability of the predicted class is 0.9. A weakly calibrated classifier is more accurate when the posterior is 90% than when it is 80%. A poorly calibrated classifier has no positive correlation between posterior and accuracy.
Train a BernoulliNB model with a reasonable alpha value. For each posterior bucket (think of a bin in a histogram), you want to estimate the classifier's accuracy. So for each prediction, find the bucket the maximum posterior belongs to and update the "correct" and "total" counters.
How would you characterize the calibration for the Naive Bayes model?
Can you write Python code for it?

#def P11(buckets, correct, total):

### STUDENT START ###

buckets = [0.5, 0.9, 0.999, 0.99999, 0.9999999, 0.999999999, 0.99999999999, 0.9999999999999, 1.0]
correct = [0 for i in buckets]
total = [0 for i in buckets]

def train_binarized_bernoulli(training_data, training_labels, alpha = 0.0001):
    binarized_train_data = binarize_examples(training_data)
    binary_naive_bayes = BernoulliNB(alpha = alpha)
    binary_naive_bayes.fit(binarized_train_data, training_labels)

    return binary_naive_bayes

def find_bucket_index(buckets, posterior):
    index = None

    for i in range(len(buckets)):
        if index == None and posterior <= buckets[i]:
            index = i

    return index

def score_by_posterior_buckets(
    binary_naive_bayes, test_data, test_labels,
    buckets, correct, total):

    predictions = binary_naive_bayes.predict(test_data)
    posteriors = binary_naive_bayes.predict_proba(test_data)
    confidences = [
        posteriors[index][predictions[index]]
        for index in range(len(predictions))
    ]

    for index, confidence in enumerate(confidences):
        bucket_index = find_bucket_index(buckets, confidence)

        total[bucket_index] = total[bucket_index] + 1

        if predictions[index] == test_labels[index]:
            correct[bucket_index] = correct[bucket_index] + 1

def P11(buckets, correct, total):
    binary_naive_bayes = train_binarized_bernoulli(
        train_data, train_labels)

    binarized_dev_data = binarize_examples(dev_data)
    score_by_posterior_buckets(binary_naive_bayes, binarized_dev_data, dev_labels,
                               buckets, correct, total)

P11(buckets, correct, total)

for i in range(len(buckets)):
    if (total[i] > 0): accuracy = float(correct[i]) / float(total[i])
    print 'p(pred) <= %.13f    total = %3d    accuracy = %.3f' %(buckets[i], total[i], accuracy)

### STUDENT END ###

#buckets = [0.5, 0.9, 0.999, 0.99999, 0.9999999, 0.999999999, 0.99999999999,...


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER: OLS/Linear models aren't designed to respect probibility range (0,1) and can produce probabilities > 1 or < 0 (e.g. 1.24). A Logistic Regression model is a great straightforward fix as it produces predictions in valid probability range (0.0 - 1.0) by design.
Fit a 1-NN and output a confusion matrix for the dev data. Use the confusion matrix to identify the most confused pair of digits, and display a few example mistakes.

confusion_matrix() produces a confusion matrix
Can you write Python code for it?

#def P5():

### STUDENT START ###

# train_k_nearest_neighbors():
#   given the parameter k, training data and labels, and development data and labels,
#   fit a k nearest neighbors classifier using the training data
def train_k_nearest_neighbors(k,
                              training_data,
                              training_labels):

    neigh = KNeighborsClassifier(n_neighbors = k)
    neigh.fit(training_data, training_labels)

    return neigh

# most_confused():
#   given a confusion matrix
#   returns a sequence that comprises the two most confused digits, and errors between them
def most_confused(confusion):
    rows, columns = np.shape(confusion)
    worst_row, worst_column, worst_errors = 0, 1, 0

    # iterate through the upper triangle, ignoring the diagonals
    # confused is the sum for each pair of indices
    for row in range(rows):
        for column in range(row + 1, columns):
            errors = confusion[row][column] + confusion[column][row]
            if errors > worst_errors:
                worst_row, worst_column, worst_errors = row, column, errors

    return ( worst_row, worst_column, worst_errors )

# select_pairwire_error_indices():
#   given a predictions vector, actual label vector, and the digits of interest
#   returns an array of indices where the digits were confused
def select_pairwire_error_indices(predictions, labels, confused_low, confused_high):
    error_indices = [ ]
    for i, prediction in enumerate(predictions):
        label = labels[i]
        if ((prediction == confused_low and label == confused_high) or
            (prediction == confused_high and label == confused_low)):

            error_indices.append(i)

    return error_indices

def P5():
    k = 1
    neigh = train_k_nearest_neighbors(k, train_data, train_labels)
    development_predicted = neigh.predict(dev_data)

    confusion = confusion_matrix(dev_labels, development_predicted)

    confused_low, confused_high, confusion_errors =...


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER: The model is poorly calibrated - all probably buckets are over-confident, many drastically so.
(12) EXTRA CREDIT
Try designing extra features to see if you can improve the performance of Naive Bayes on the dev set. Here are a few ideas to get you started:
- Try summing the pixel values in each row and each column.
- Try counting the number of enclosed regions; 8 usually has 2 enclosed regions, 9 usually has 1, and 7 usually has 0.
Make sure you comment your code well!
Can you write Python code for it?

#def P12():

### STUDENT START ###

### STUDENT END ###

#P12()


I am working on the file "John_Bocharov_p1.ipynb".
The first task is:
ANSWER: The most difficult digit is 9, as measured by f1-score
(3) Using k=1, report dev set accuracy for the training set sizes below. Also, measure the amount of time needed for prediction with each training size.

time.time() gives a wall clock value you can use for timing operations
Can you write Python code for it?

#def P3(train_sizes, accuracies):

### STUDENT START ###
# k_nearest_neighbors_timed_accuracy():
#   given the parameter k, training data and labels, and development data and labels,
#   fit a k nearest neighbors classifier using the training data,
#   test using development data, and return the number of examples, prediction time,
#   and accuracy as a Python dictionary
def k_nearest_neighbors_timed_accuracy(k,
                                       training_data,
                                       training_labels,
                                       development_data,
                                       development_labels):

    neigh = KNeighborsClassifier(n_neighbors = k)
    neigh.fit(training_data, training_labels)

    start = time.time()
    predicted_labels = neigh.predict(development_data)
    end = time.time()

    examples, dimensions = np.shape(training_data)

    accuracy = accuracy_score(development_labels, predicted_labels, normalize = True)

    return { 'examples' : examples, 'time' : end-start, 'accuracy' : accuracy }

def P3(train_sizes, accuracies):
    k = 1
    for train_size in train_sizes:
        # sample train_size examples from the training set
        current_train_data, current_train_labels = X[:train_size], Y[:train_size]

        results = k_nearest_neighbors_timed_accuracy(k,
                                                     current_train_data,
                                                     current_train_labels,
                                                     dev_data,
                                                     dev_labels)
        print(results)
        accuracies.append(results['accuracy'])

train_sizes = [100, 200, 400, 800, 1600, 3200, 6400, 12800, 25000]
accuracies = [ ]
P3(train_sizes, accuracies)
### STUDENT END ###

#train_sizes = [100, 200, 400, 800, 1600, 3200, 6400, 12800, 25000]
#accuracies = []
#P3(train_sizes, accuracies)


I am working on the file "John_Bocharov_p2.ipynb".
The first task is:
(1) For each of the first 5 training examples, print the text of the message along with the label.
Can you write Python code for it?

#def P1(num_examples=5):
### STUDENT START ###

### STUDENT END ###
#P1()


I am working on the file "John_Bocharov_p2.ipynb".
The first task is:
(2) Use CountVectorizer to turn the raw training text into feature vectors. You should use the fit_transform function, which makes 2 passes through the data: first it computes the vocabulary ("fit"), second it converts the raw text into feature vectors using the vocabulary ("transform").
The vectorizer has a lot of options. To get familiar with some of them, write code to answer these questions:
a. The output of the transform (also of fit_transform) is a sparse matrix: http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sparse.csr_matrix.html. What is the size of the vocabulary? What is the average number of non-zero features per example? What fraction of the entries in the matrix are non-zero? Hint: use "nnz" and "shape" attributes.
b. What are the 0th and last feature strings (in alphabetical order)? Hint: use the vectorizer's get_feature_names function.
c. Specify your own vocabulary with 4 words: ["atheism", "graphics", "space", "religion"]. Confirm the training vectors are appropriately shaped. Now what's the average number of non-zero features per example?
d. Instead of extracting unigram word features, use "analyzer" and "ngram_range" to extract bigram and trigram character features. What size vocabulary does this yield?
e. Use the "min_df" argument to prune words that appear in fewer than 10 documents. What size vocabulary does this yield?
f. Using the standard CountVectorizer, what fraction of the words in the dev data are missing from the vocabulary? Hint: build a vocabulary for both train and dev and look at the size of the difference.
Can you write Python code for it?

#def P2():
### STUDENT START ###

### STUDENT END ###
#P2()


I am working on the file "John_Bocharov_p2.ipynb".
The first task is:
(3) Use the default CountVectorizer options and report the f1 score (use metrics.f1_score) for a k nearest neighbors classifier; find the optimal value for k. Also fit a Multinomial Naive Bayes model and find the optimal value for alpha. Finally, fit a logistic regression model and find the optimal value for the regularization strength C using l2 regularization. A few questions:
a. Why doesn't nearest neighbors work well for this problem?
b. Any ideas why logistic regression doesn't work as well as Naive Bayes?
c. Logistic regression estimates a weight vector for each class, which you can access with the coef_ attribute. Output the sum of the squared weight values for each class for each setting of the C parameter. Briefly explain the relationship between the sum and the value of C.
Can you write Python code for it?

#def P3():
### STUDENT START ###

### STUDENT END ###
#P3()


I am working on the file "John_Bocharov_p2.ipynb".
The first task is:
(7) Use the TfidfVectorizer -- how is this different from the CountVectorizer? Train a logistic regression model with C=100.
Make predictions on the dev data and show the top 3 documents where the ratio R is largest, where R is:
maximum predicted probability / predicted probability of the correct label
What kinds of mistakes is the model making? Suggest a way to address one particular issue that you see.
Can you write Python code for it?

#def P7():
### STUDENT START ###

## STUDENT END ###
#P7()


I am working on the file "John_Bocharov_p2.ipynb".
The first task is:
ANSWER:
(4) Train a logistic regression model. Find the 5 features with the largest weights for each label -- 20 features in total. Create a table with 20 rows and 4 columns that shows the weight for each of these features for each of the labels. Create the table again with bigram features. Any surprising features in this table?
Can you write Python code for it?

#def P4():
### STUDENT START ###

### STUDENT END ###
#P4()


I am working on the file "John_Bocharov_p2.ipynb".
The first task is:
ANSWER:
(5) Try to improve the logistic regression classifier by passing a custom preprocessor to CountVectorizer. The preprocessing function runs on the raw text, before it is split into words by the tokenizer. Your preprocessor should try to normalize the input in various ways to improve generalization. For example, try lowercasing everything, replacing sequences of numbers with a single token, removing various other non-letter characters, and shortening long words. If you're not already familiar with regular expressions for manipulating strings, see https://docs.python.org/2/library/re.html, and re.sub() in particular. With your new preprocessor, how much did you reduce the size of the dictionary?
For reference, I was able to improve dev F1 by 2 points.
Can you write Python code for it?

#def empty_preprocessor(s):
#    return s

#def better_preprocessor(s):
### STUDENT START ###

### STUDENT END ###

#def P5():
### STUDENT START ###

### STUDENT END ###
#P5()


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Final join between joined lunches1314 and lunches2015
Can you write Python code for it?

lunches = lunches1314.join(cleanLunches2015, 'CODISTSCH13', 'CODISTSCH15', inner=True )


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Find those top free, reduced lunch percent change
Can you write Python code for it?

sorted_change = rounded_change.order_by('pc_freelunch', reverse=True)


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Join the cleanLunches2013 & 2014 on district codes
Can you write Python code for it?

lunches1314 = cleanLunches2013.join(cleanLunches2014, 'CODISTSCH13', 'CODISTSCH14', inner=True )


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Make those percent changes to one decimal
Can you write Python code for it?

from decimal import Decimal
def round_freelunch_change(row):
    return row['freelunch_change'].quantize(Decimal('0.1'))
def round_enrollment_change(row):
    return row['enrollment_change'].quantize(Decimal('0.1'))

rounded_change = change.compute([
    ('pc_freelunch', agate.Formula(agate.Number(), round_freelunch_change)),
    ('pc_enrollment', agate.Formula(agate.Number(), round_enrollment_change)),
])


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Only need certain columns to print out
Can you write Python code for it?

for_printing = sorted_change.select(['SCHOOLNAME13', 'pc_freelunch', 'pc_enrollment'])


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Pull in the 2014 lunch information csv, clean out any empty values, print table columns and lengths
Can you write Python code for it?

lunches2014 = agate.Table.from_csv('frl14.csv')
cleanLunches2014 = lunches2014.where(lambda row: row['FREEREDUCED14'] is not None)
print(cleanLunches2014)
print(len(cleanLunches2014.rows))


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
Pull in the 2015 lunch information csv, clean out any empty values, print table columns and lengths
Can you write Python code for it?

lunches2015 = agate.Table.from_csv('frl15.csv')
cleanLunches2015 = lunches2015.where(lambda row: row['FREEREDUCED15'] is not None)
print(cleanLunches2015)
print(len(cleanLunches2015.rows))


I am working on the file "JoinHomework/JoinHomework.ipynb".
The first task is:
View the final joined table
Can you write Python code for it?


I am working on the file "Joint-Space-Trajectories.ipynb".
The first task is:
Cubic Polynomial
Can you write Python code for it?

def quintic_trajectory(current_position, target_position, current_velocity,
                       target_velocity, current_acceleration,
                       target_acceleration, duration_in_seconds):
    trajectories = []
    t = duration_in_seconds
    xs = linspace(0, t)
    for qi, qf, dqi, dqf, ddqi, ddqf in zip(current_position, target_position,
                                            current_velocity, target_velocity,
                                            current_acceleration, target_acceleration):
        A = np.array(
            [[0.0, 0.0, 0.0, 0.0, 0.0, 1.0],
             [t**5, t**4, t**3, t**2, t, 1.0],
             [0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
             [5. * t**4, 4. * t**3, 3. * t**2, 2. * t, 1., 0.0],
             [0.0, 0.0, 0.0, 2.0, 0.0, 0.0],
             [20. * t**3, 12. * t**2, 6. * t, 2., 0.0, 0.0]])

        b = np.array([qi, qf, dqi, dqf, ddqi, ddqf])
        x = np.linalg.solve(A, b)

        qs = np.polyval(x, xs)
        dqs = np.polyval([5. * x[0], 4. * x[1], 3. * x[2], 2. * x[3], x[4]], xs)
        ddqs = np.polyval([20. * x[0], 12. * x[1], 6. * x[2], 2. * x[3]], xs)

        trajectories.append((qs, dqs, ddqs))
    return trajectories

qi = [0,np.pi]
qf = [np.pi,0]
dqi = [0,0]
dqf = [0,0]
ddqi = [0,0]
ddqf = [0,0]
trajectories = quintic_trajectory(qi, qf, dqi, dqf, ddqi, ddqf, 50)

plot_trajectory(trajectories[0], iscubic=False)


I am working on the file "Joint-Space-Trajectories.ipynb".
The first task is:
Joint Space Trajectories
Point-to-Point Motion
Cubic Polynomial
\begin{align}
q(t) &= a_3 t^3 + a_2 t^2 + a_1 t + a_0 \
\dot{q}(t)  &= 3 a_3 t^2 + 2 a_2 t + a_1 \
\ddot{q}(t) &= 6 a_3 t + 2 a_2
\end{align}
\begin{align}
a_0 &= q_i \
a_1 &= \dot{q}_i \
a_3 t_f^3 + a_2 t_f^2 + a_1 t_f + a_0 &= q_f \
3 a_3 t_f^2 + 2 a_2 t_f + a_1 &= \dot{q}_f
\end{align}
\begin{align}
\begin{pmatrix} q_i \ q_f \ \dot q_i \ \dot q_f \end{pmatrix} =
\begin{pmatrix}
  0 & 0 & 0 & 1 \
  t^3 & t^2 & t & 1 \
  0 & 0 & 1 & 0 \
  3t^2 & 2t & 1 & 0
  \end{pmatrix}
  \begin{pmatrix} a_3 \ a_2 \ a_1 \ a_0 \end{pmatrix}
\end{align}
Can you write Python code for it?

def cubic_trajectory(current_position, target_position,
                     current_velocity, target_velocity,
                     duration_in_seconds):
    trajectories = []
    t = duration_in_seconds
    xs = linspace(0,t)
    for qi, qf, dqi, dqf in zip(current_position, target_position,
                                current_velocity, target_velocity):
        A = np.array([[0.0,0.0,0.0,1.0],
                      [t**3, t**2, t, 1.],
                      [0.0, 0.0, 1.0, 0.0],
                      [3.0 * t**2, 2*t, 1.0, 0.0]])

        b = np.array([qi, qf, dqi, dqf])
        x = np.linalg.solve(A,b)

        qs = np.polyval(x, xs)
        dqs = np.polyval([3. * x[0], 2. * x[1], x[2]], xs)
        ddqs = np.polyval([6. * x[0], 2. * x[1]], xs)

        trajectories.append((qs, dqs, ddqs))
    return trajectories

qi = [0,np.pi]
qf = [np.pi,0]
dqi = [0,0]
dqf = [0,0]
trajectories = cubic_trajectory(qi, qf, dqi, dqf, 50)

plot_trajectory(trajectories[0], iscubic=True)


I am working on the file "Jones Calculus for EIT Setup.ipynb".
The first task is:
As given by newport (https://www.newport.com/f/quartz-zero-order-waveplates), the wave error at 795 (vs 780) corresponds to a normalized wavelength of 1.02, giving wave error of -0.08 waves. That corresponds to a phase error of 2pi*(-0.08) = 0.5 radians.
We'll explore the effect of this phase error below:
Can you write Python code for it?

ivals = []
thetas = np.linspace(0,np.pi)
for theta in thetas:
    #        vpol        quarter             quarter                   half               input
    output = vpol*wp(np.pi/2-0.5,theta)*horiz
    ivals.append(intensity(output))

plt.plot(thetas,ivals)
plt.title("rotating qwp w/error between crossed pols")

ivals = []
thetas = np.linspace(0,np.pi/2)
for phi in [0.3,0.4,0.5,0.6,0.7]:  # try a range of phase errors to compare
    ivals = []
    for theta in thetas:
        #        vpol        quarter             quarter                   half               input
        output = vpol*wp(np.pi/2 - phi,-theta)*wp(np.pi/2 - phi,theta)*wp(np.pi - phi,np.pi/19)*horiz
        ivals.append(intensity(output))

    plt.plot(thetas,ivals,label=phi)

plt.legend()
plt.ylabel("I output")
plt.xlabel("qwp1 angle (rad)")

# Rotating 780 QWP between crossed pols
# at 795 nm
ivals = []
thetas = np.linspace(0,np.pi)
for phi in [0.0,0.3,0.4,0.5,0.6,0.7]:
    ivals = []
    for theta in thetas:
        output = vpol*wp(np.pi/2 - phi,theta)*horiz
        ivals.append(intensity(output))

    plt.plot(thetas,ivals,label=phi)

plt.legend()
plt.ylabel("I output")
plt.xlabel("qwp angle (rad)")
plt.title("QWP w/ phase error")