I am working on the file "Subreddit Analysis t-SNE.ipynb".
The first task is:
The final real test was simply print out the contents of the clusters and see if they made sense. For the most part they are pretty good, but they are less good than what LargeVis provided, with more clusters for which there aren't clear topics. Feel free to do exactly this for the LargeVis version and you'll see what I mean.
Can you write Python code for it?


I am working on the file "SubstateDecoding.ipynb".
The first task is:
Below we take a look at a particular placefield, and where the mean and mode for this place field would lie.
Can you write Python code for it?

# select state to consider:
cc = 24

x0=0; xl=100; num_pos_bins=50
xx_mid = np.linspace(x0,xl,num_pos_bins+1)[:-1]; xx_mid += (xx_mid[1]-xx_mid[0])/2
#plt.plot(xx_mid,state_pos[24,:])

tmp = state_pos[cc,:]; tmp = tmp/tmp.sum()

plt.plot(xx_mid,tmp, linewidth=2)

pfmean = (tmp*xx_mid).sum()
pfmode = xx_mid[tmp.argmax()]

plt.axvline(x=pfmode, ymin=0, ymax=1, linewidth=2, color = 'gray', linestyle='dotted', label='mode')
plt.axvline(x=pfmean, ymin=0, ymax=1, linewidth=2, color = 'gray', linestyle='dashed', label='mean')
plt.legend()
plt.xlabel('position')
plt.ylabel('p(pos | state = {})'.format(cc))
plt.title('position distribution (place field) for state {}'.format(cc), y=1.03)
#vlines(x, ymin, ymax, colors='k', linestyles='solid', label='', hold=None, data=None
#ax3.plot([7, 7], [0, num_states], color='gray', linestyle='dashed', linewidth=1)


I am working on the file "SubstateDecoding.ipynb".
The first task is:
Part 1: Symbol-by-symbol maximum a posteriori (MAP) state [position] decoding
Here we are primarily interested in the posterior state distributions, and we don't actually ever expect symbol-by-symbol MAP sequences to exist that are impossible in our model. To decode the states further to actual positions, we then associate with each state a position derived either from the mode of its place field, or from the mean of its place field. We refer to these decoding methods simply as 'mode' and 'mean' decoding.
First, let's look at some posterior state distributions for a few observation sequences:
Can you write Python code for it?

## order states by peak location on track

tmpseqbdries = [0]; tmpseqbdries.extend(np.cumsum(ts_b2.sequence_lengths).tolist());

fig, axes = plt.subplots(3, 7, figsize=(17, 8))
axes = [item for sublist in axes for item in sublist]

for ii, ax in enumerate(axes):
    obs = ts_b2.data[tmpseqbdries[ii]:tmpseqbdries[ii+1],:]
    ll, pp = myhmm2.score_samples(obs)
    ax.matshow(np.transpose(pp)[mode_order,:], interpolation='none', cmap='RdPu')
    #ax.axison = False
    ax.set_xticklabels([])
    ax.set_yticklabels([])
    ax.set_title('sequence ' + str(ii))
    ax.set_ylabel('states')


I am working on the file "SubstateDecoding.ipynb".
The first task is:
Remarks: Are the above sequences continuous? They look somewhat continuous, but not exactly. The problem is that state-adjecency is not exactly equivalent to positional continuity. Nevertheless, the jump discontinuous nature of the sequences stem from the fact that the rat is running faster than the positions encoded by adjacent states. That is, the rat is running at 3 states per time bin, or something like that.
So how can we get a better idea about whether or not the above sequences actually correspond to contunuous trajectories in the environment? We further decode them to their underlying positions, shown next.
Part 2: Decoding positions from state sequences
Here we decode state sequences (symbol-by-symbol MAP sequences) to their associated position sequences on the track.
Can you write Python code for it?

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(17, 3))

seq_id = 7

obs = ts_b2.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
ll, pp = myhmm2.score_samples(obs)
seq_len = len(obs)

mean_pos = pfmeans[pp.argmax(axis=1)]
mode_pos = pfmodes[pp.argmax(axis=1)]

# true position:
bb = ts_b2.boundaries[seq_id,:] # inclusive
true_pos = truepos_125[bb[0]:bb[1]+1]

ax1.plot(mean_pos, '.', markersize=7, label='mean decoded', linestyle='dashed', linewidth=1.5, color='k')
ax1.plot(mode_pos, '.', markersize=7, label='mode decoded', linestyle='dotted', linewidth=1.5, color='k')
ax1.plot(true_pos,'.',markersize=7, label='true position', linestyle='solid', color='darkorange', linewidth=1.5)

ax1.set_ylim([0,100])
ax1.legend()
ax1.set_ylabel('position')
ax1.set_xlabel('time bin')
ax1.set_title('sequence decoding, seq {}'.format(seq_id), y=1.03)

seq_id = 14

obs = ts_b2.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
ll, pp = myhmm2.score_samples(obs)
seq_len = len(obs)

mean_pos = pfmeans[pp.argmax(axis=1)]
mode_pos = pfmodes[pp.argmax(axis=1)]

# true position:
bb = ts_b2.boundaries[seq_id,:] # inclusive
true_pos = truepos_125[bb[0]:bb[1]+1]

ax2.plot(mean_pos, '.', markersize=7, label='mean decoded', linestyle='dashed', linewidth=1.5, color='k')
ax2.plot(mode_pos, '.', markersize=7, label='mode decoded', linestyle='dotted', linewidth=1.5, color='k')
ax2.plot(true_pos,'.',markersize=7, label='true position', linestyle='solid', color='darkorange', linewidth=1.5)

ax2.set_ylim([0,100])
ax2.legend(loc='upper left')
ax2.set_ylabel('position')
ax2.set_xlabel('time bin')
ax2.set_title('sequence decoding, seq {}'.format(seq_id), y=1.03)


I am working on the file "SubstateDecoding.ipynb".
The first task is:
Remarks: As argued before, since this place field is unimodal, assigning its position as the mean of the place field might be more appropriate than using its mode. However, if we first smoothed the place field, then most likely the mean and the mode would coincide.
Next we look at all the place fields learned by the HMM.
Can you write Python code for it?

x0=0; xl=100; num_pos_bins=50
xx_mid = np.linspace(x0,xl,num_pos_bins+1)[:-1]; xx_mid += (xx_mid[1]-xx_mid[0])/2

# normalize place fields:
placefields = state_pos.copy()
colsum = np.tile(placefields.sum(axis=1),(num_pos_bins,1)).T
placefields = placefields/colsum

# determine mean for each place field:
pfmeans = (np.tile(xx_mid,(num_states,1))*placefields).sum(axis=1)
pfmodes = xx_mid[placefields.argmax(axis=1)]

# order place fields by mean or mode:
mean_order = pfmeans.argsort()
mode_order = pfmodes.argsort()

fig, axes = plt.subplots(num_states, 2, figsize=(15, 15))
axes = [item for sublist in axes for item in sublist]

for ii, ax in enumerate(axes):
    if ii % 2 == 0:
        idx = int(ii/2)
        ax.plot(xx_mid,placefields[mode_order[idx],:])
        ax.fill_between(xx_mid, 0, placefields[mode_order[idx]], color='gray')
        ax.set_ylabel(mode_order[idx])
    else:
        idx = int((ii-1)/2)
        ax.plot(xx_mid,placefields[mean_order[idx],:])
        ax.fill_between(xx_mid, 0, placefields[mean_order[idx]], color='lightgray')
        ax.set_ylabel(mean_order[idx])
    ax.set_xticklabels([])
    ax.set_yticklabels([])

fig.suptitle('normalized place fields sorted by peak location (left) and mean location (right)', y=0.92, fontsize=14)


I am working on the file "SubstateDecoding.ipynb".
The first task is:
Remarks: Note that the decoded positions generally follow the true position fairly well, but that the decoded positions seem to be piece-wise constant. That is expected, because there are only a relatively small number of discrete states, each with its associated position.
Next, we look at substate decoding, where we decode the position as a weighted combination of the positions of all the participating states at any given time.
Can you write Python code for it?

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(17, 3))

seq_id = 7

obs = ts_b2.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
ll, pp = myhmm2.score_samples(obs)
seq_len = len(obs)

mean_pos = pfmeans[pp.argmax(axis=1)]
mode_pos = pfmodes[pp.argmax(axis=1)]
substate_pos = (pp*np.tile(pfmeans,(seq_len,1))).sum(axis=1)

# true position:
bb = ts_b2.boundaries[seq_id,:] # inclusive
true_pos = truepos_125[bb[0]:bb[1]+1]

ax1.plot(mean_pos, '.', markersize=7, label='mean decoded', linestyle='dashed', linewidth=1.5, color='k')
ax1.plot(mode_pos, '.', markersize=7, label='mode decoded', linestyle='dotted', linewidth=1.5, color='k')
ax1.plot(substate_pos,'.',markersize=7, label='substate decoded', linestyle='solid', color='deepskyblue', linewidth=1.5)
ax1.plot(true_pos,'.',markersize=7, label='true position', linestyle='solid', color='darkorange', linewidth=1.5)

ax1.set_ylim([0,100])
ax1.legend()
ax1.set_ylabel('position')
ax1.set_xlabel('time bin')
ax1.set_title('sequence decoding, seq {}'.format(seq_id), y=1.03)

seq_id = 14

obs = ts_b2.data[tmpseqbdries[seq_id]:tmpseqbdries[seq_id+1],:]
ll, pp = myhmm2.score_samples(obs)
seq_len = len(obs)

mean_pos = pfmeans[pp.argmax(axis=1)]
mode_pos = pfmodes[pp.argmax(axis=1)]
substate_pos = (pp*np.tile(pfmeans,(seq_len,1))).sum(axis=1)

# true position:
bb = ts_b2.boundaries[seq_id,:] # inclusive
true_pos = truepos_125[bb[0]:bb[1]+1]

ax2.plot(mean_pos, '.', markersize=7, label='mean decoded', linestyle='dashed', linewidth=1.5, color='k')
ax2.plot(mode_pos, '.', markersize=7, label='mode decoded', linestyle='dotted', linewidth=1.5, color='k')
ax2.plot(substate_pos,'.',markersize=7, label='substate decoded', linestyle='solid', color='deepskyblue', linewidth=1.5)
ax2.plot(true_pos,'.',markersize=7, label='true position', linestyle='solid', color='darkorange', linewidth=1.5)

ax2.set_ylim([0,100])
ax2.legend(loc='upper left')
ax2.set_ylabel('position')
ax2.set_xlabel('time bin')
ax2.set_title('sequence decoding, seq {}'.format(seq_id), y=1.03)

errors =...


I am working on the file "Subvocal.ipynb".
The first task is:
AF Extractor Models
These models will be optimized for extracting Articulatory Features from the data, before passing those AF's onto an MLPC for identifying the most likely phoneme.
Can you write Python code for it?

# Prepare lists of parameters for our GridSearch
# First, our layer sizes
layer_sizes = []
for i in range(2,5):
    for j in range(0,180,30):
        if j:
            tup = []
            for k in range(i):
                tup.append(j)
            layer_sizes.append(tuple(tup))
print('number layer sizes:',len(layer_sizes),'here be layer sizes',layer_sizes)

# Next, our alpha values
alphas = [0.0001,1,1000]


I am working on the file "Subvocal.ipynb".
The first task is:
Benchmark Optimization
A benchmark model is optimized and assessed, serving as a standard for the solution model to best.
Can you write Python code for it?

pho2_X_train, pho2_X_test, pho2_y_train, pho2_y_test = train_test_split(X,phoneme_labels, test_size=0.15, random_state=12)

benchmark_gs = GridSearchCV(pipeline, param_grid, n_jobs=-1)

benchmark_gs.fit(pho2_X_train, pho2_y_train)
pho2_score = benchmark_gs.score(pho2_X_test,pho2_y_test)
print(pho2_score)


I am working on the file "Subvocal.ipynb".
The first task is:
Labeling the Data
First, we generate the phoneme and articulatory feature labels from each word. We'll use these to process the data in each file based on the length of the file and how many phonemes it should contain. We perform CWT, the continuous wavelet transform on the EMG data to help differentiate MUAP activity. We scale the CWT windows relative to the length of time we expect an even distribution of phonemes across the file to have.
Can you write Python code for it?

from scipy import signal
import numpy as np

labels = {}
windows = {}
for word in singles_1:
    try:
        label = Output_Prep.transform(word)
        num_phonemes = label.shape[0]
        label = label.append(Output_Prep.transform(word))
        label = label.append(Output_Prep.transform(word))
        labels[word] = label
        widths = np.linspace(0.01,10,50)
        wt_out = signal.cwt(singles_1[word]['voltage'], signal.ricker, widths)
        wt_out = pandas.DataFrame(wt_out).T
        windows[word] = EMG_Prep.process(wt_out, num_phonemes, wavelets=True)
        wt_out_2 = signal.cwt(singles_2[word]['voltage'], signal.ricker, widths)
        wt_out_2 = pandas.DataFrame(wt_out_2).T
        windows[word] = windows[word].append(EMG_Prep.process(wt_out_2, num_phonemes, wavelets=True))
        wt_out_3 = signal.cwt(singles_3[word]['voltage'], signal.ricker, widths)
        wt_out_3 = pandas.DataFrame(wt_out_3).T
        windows[word] = windows[word].append(EMG_Prep.process(wt_out_3, num_phonemes, wavelets=True))
    except Exception as inst:
        print(inst)


I am working on the file "Subvocal.ipynb".
The first task is:
Preparing Input, Output 'Master' DataFrames
Various output and input dataframes need to be concatenated together for later usage with cross-validation.
Can you write Python code for it?

import pandas
%autoreload 2

y = pandas.DataFrame()
X = pandas.DataFrame()
for word in labels:
#     append labels to the master label dataframe
    label_frame = labels[word]
    y = y.append(label_frame)
#     Use phonemes to name each series in 'windows' for that word
    window_frame = windows[word]
    if len(label_frame.axes[0]):
        window_frame = window_frame.rename_axis(lambda x: label_frame.axes[0][x])
        X = X.append(window_frame)
    else:
        print('no labels for:',word)

print(y.head(),X.head())


I am working on the file "Subvocal.ipynb".
The first task is:
Preprocessing, continued
Now that wavelet transforms have been applied to the EMG data, further processing is done. Scaling columns helps center features, PCA reduces the total number of features to just 10 of the most important ones, and normalization helps balance individual samples.
Can you write Python code for it?

# print(X.head(18), y.head(18))
from sklearn.preprocessing import scale,normalize
from sklearn.decomposition import PCA

X_scaled = scale(X)
pca = PCA(n_components=10, random_state=9)
X_reduced = pca.fit_transform(X_scaled)
X_normalized = normalize(X_reduced)

X_normalized = pandas.DataFrame(X_normalized)
X_normalized = X_normalized.rename_axis(lambda x: 'pc-'+str(x), axis='columns')
print(X_normalized)


I am working on the file "Subvocal.ipynb".
The first task is:
Solution Model
Here's where we actually optimize and test our benchmark model to see if adding AF's in the form of predictions from specialized AFE's is better than the baseline architecture.
Can you write Python code for it?

pho_layer_sizes = []
for i in range(2,10):
    for j in range(60,120,30):
        if j:
            tup = []
            for k in range(i):
                tup.append(j)
            pho_layer_sizes.append(tuple(tup))
print('number layer sizes:',len(pho_layer_sizes),'here be layer sizes',pho_layer_sizes)

# Next, our alpha values
pho_alphas = [0.001,0.1,1,1000]

param_grid = {
#     'features__pca__n_components':[10,20,50],
    'model__solver':['adam'],
    'model__hidden_layer_sizes':pho_layer_sizes,
    'model__activation':['relu'],
    'model__alpha': pho_alphas,
    'model__max_iter':[300]
}

pho_model_grid_search = GridSearchCV(pipeline, param_grid, n_jobs=-1)
# The Solution Model
phoneme_classifier = pho_model_grid_search
phoneme_classifier.fit(pho_X_train, pho_y_train)
pho_train_f1 = phoneme_classifier.score(pho_X_train, pho_y_train)
print('phoneme classifier training score:',pho_train_f1)

pho_test_score = phoneme_classifier.score(pho_X_test, pho_y_test)
print('phoneme model test score:',pho_test_score)


I am working on the file "Subvocal.ipynb".
The first task is:
Training the Solution and Benchmark Models
Training the solution model requires predicted AF's, using the optimized AFE models gridsearched in the previous step. Here is where the predictions are made and transformed into input features for the solution model before CV train-test splitting is performed.
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder as LE
from sklearn.feature_extraction import DictVectorizer as DV
from sklearn.preprocessing import MultiLabelBinarizer as MLB
from sklearn.preprocessing import OneHotEncoder as OHE
from collections import Counter

manner_inputs = manner_classifier2.predict(X_normalized)
place_inputs = place_classifier2.predict(X_normalized)
height_inputs = height_classifier2.predict(X_normalized)
vowel_inputs = vowel_classifier2.predict(X_normalized)

# We need to account for each value that each category of label can take on
m_count = Counter()
p_count = Counter()
h_count = Counter()
v_count = Counter()

for row in range(y.shape[0]):
    m_count.update([y.iloc[row]['manner']])
    p_count.update([y.iloc[row]['place']])
    h_count.update([y.iloc[row]['height']])
    v_count.update([y.iloc[row]['vowel']])

counters = [m_count,p_count,h_count,v_count]

feature_dict = {}
for count in counters:
    current = 0
    for feature in count.keys():
        feature_dict[feature] = current
        current += 1

# Then, we transform the predicted labels with one-hot encoding after
# concatenating the AF outputs and Solution Model Inputs
raw_inputs = copy.deepcopy(y)
for row in range(len(raw_inputs)):

    raw_inputs.iloc[row]['manner'] = manner_inputs[row]
    raw_inputs.iloc[row]['place'] = place_inputs[row]
    raw_inputs.iloc[row]['height'] = height_inputs[row]
    raw_inputs.iloc[row]['vowel'] = vowel_inputs[row]

num_labels = copy.deepcopy(raw_inputs)
for row in range(raw_inputs.shape[0]):
    m_feat = raw_inputs.iloc[row]['manner']
    p_feat = raw_inputs.iloc[row]['place']
    h_feat = raw_inputs.iloc[row]['height']
    v_feat = raw_inputs.iloc[row]['vowel']
    num_labels.iloc[row]['manner'] = feature_dict[m_feat]
    num_labels.iloc[row]['place'] = feature_dict[p_feat]
    num_labels.iloc[row]['height'] = feature_dict[h_feat]
    num_labels.iloc[row]['vowel'] = feature_dict[v_feat]

encoder = OHE()
new_labels = encoder.fit_transform(num_labels)
enc_labels =...


I am working on the file "Subvocal.ipynb".
The first task is:
Various Stat Calculations
We calculate distributions of phonemes, various dataset statistics, detailed performance comparison between benchmark and solution, and inspection of the solution model.
Can you write Python code for it?

phonemes = Counter(phoneme_labels)
N = len(phonemes)

total = sum(phonemes.values())
for key in phonemes:
    phonemes[key] = phonemes[key] / total
    print(key, "represents", str(phonemes[key]*100)+"%","of all samples")
ind = np.arange(N)  # the x locations for the groups
width = .66       # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(ind, phonemes.values(), width, color='xkcd:purple')

# add some text for labels, title and axes ticks
ax.set_ylabel('Phonemes')
ax.set_title('Phoneme instances by type')
ax.set_xticks(ind)
ax.set_xticklabels(phonemes.keys(),size='xx-small')

ax.legend('Phonemes')

def autolabel(rects):
    """
    Attach a text label above each bar displaying its height
    """
    for rect in rects:
        height = rect.get_height()
        ax.text(rect.get_x() + rect.get_width()/2., 1.05*height,
                '%f' % int(height),
                ha='center', va='bottom')

autolabel(rects1)
plt.show()

num_phonemes = []
num_letters = []
for word in labels:
    label_length = len(labels[word].axes[0])
    phonemes = labels[word].axes[0].values[0:label_length/3]
    num_letters.append(len(word))
    num_phonemes.append(len(phonemes))
#     print(word, ",", labels[word].axes[0].values[0:label_length/3])
print('average word length:', np.mean(num_letters), '+/-',np.std(num_letters))
print('average num phonemes:', np.mean(num_phonemes), '+/-', np.std(num_phonemes))

import random

singles = [singles_1, singles_2, singles_3]
duration = 0
durations = []
dataframe_size = 0
dataframe_sizes = []
voltages = []
for single in singles:
    for word in single:
        length = len(single[word])
        dur = single[word]['time'][length-1]
        size = np.sum(single[word].memory_usage())
        duration += dur
        durations.append(dur)
        dataframe_size += size
        dataframe_sizes.append(size)
        avg_v = np.mean(single[word]['voltage'])
        voltages.append(avg_v)
print('total duration:', duration, "seconds")
print('standard deviation of duration:',...


I am working on the file "Summarize the reviews.ipynb".
The first task is:
Benchmark: predict by distribution
Can you write Python code for it?

accuracy = reviews_good.count() / float(train_reviews.count())
print('Always predicting 5 stars accuracy: {0}'.format(accuracy))


I am working on the file "Summarize the reviews.ipynb".
The first task is:
Splitting data and balancing skewness
Can you write Python code for it?

train, test = reviews.randomSplit([.8, .2], seed=5436L)

def multiply_dataset(dataset, n):
    return dataset if n <= 1 else dataset.union(multiply_dataset(dataset, n - 1))

reviews_good = train.filter('label == 1')
reviews_bad = train.filter('label == 0')

reviews_bad_multiplied = multiply_dataset(reviews_bad, reviews_good.count() / reviews_bad.count())

train_reviews = reviews_bad_multiplied.union(reviews_good)


I am working on the file "Summarize the reviews.ipynb".
The first task is:
Summarize single product - picks the best and worst
Can you write Python code for it?

full_model = pipeline.fit(reviews)

highly_reviewed_products = (all_reviews
    .groupBy('asin')
    .agg(F.count('asin').alias('count'), F.avg('overall').alias('avg_rating'))
    .filter('count > 25'))

best_product = highly_reviewed_products.sort('avg_rating', ascending=False).take(1)[0][0]

worst_product = highly_reviewed_products.sort('avg_rating').take(1)[0][0]

def most_contributing_summaries(product, total_reviews, ranking_model):
    reviews = total_reviews.filter(col('asin') == product).select('summary', 'overall')

    udf_max = udf(lambda p: max(p.tolist()), FloatType())

    summary_ranks = (ranking_model
        .transform(reviews)
        .select(
            'summary',
            second(col('probability')).alias('pos_prob')))

    pos_summaries = { row[0]: row[1] for row in summary_ranks.sort('pos_prob', ascending=False).take(10) }
    neg_summaries = { row[0]: row[1] for row in summary_ranks.sort('pos_prob').take(10) }

    return pos_summaries, neg_summaries

from wordcloud import WordCloud
import matplotlib.pyplot as plt

def present_product(product, total_reviews, ranking_model):
    pos_summaries, neg_summaries = most_contributing_summaries(product, total_reviews, ranking_model)

    pos_wordcloud = WordCloud(background_color='white', max_words=20).fit_words(pos_summaries)
    neg_wordcloud = WordCloud(background_color='white', max_words=20).fit_words(neg_summaries)

    fig = plt.figure(figsize=(15, 15))

    ax = fig.add_subplot(1,2,1)
    ax.set_title('Positive summaries')
    ax.imshow(pos_wordcloud, interpolation='bilinear')
    ax.axis('off')

    ax = fig.add_subplot(1,2,2)
    ax.set_title('Negative summaries')
    ax.imshow(neg_wordcloud, interpolation='bilinear')
    ax.axis('off')

    plt.show()

present_product(best_product, all_reviews, full_model)

present_product(worst_product, all_reviews, full_model)


I am working on the file "Summarize the reviews.ipynb".
The first task is:
Using model to extract the most predictive words
Can you write Python code for it?

from pyspark.sql.functions import explode
import pyspark.sql.functions as F
from pyspark.sql.types import FloatType

words = (tokenizer
    .transform(reviews)
    .select(explode(col('words')).alias('summary')))

predictors = (model
    .transform(words)
    .select(col('summary').alias('word'), 'probability'))

first = udf(lambda x: x[0].item(), FloatType())
second = udf(lambda x: x[1].item(), FloatType())

predictive_words = (predictors
   .select(
       'word',
       second(col('probability')).alias('positive'),
       first(col('probability')).alias('negative'))
   .groupBy('word')
   .agg(
       F.max('positive').alias('positive'),
       F.max('negative').alias('negative')))

positive_predictive_words = (predictive_words
    .select(col('word').alias('positive_word'), col('positive').alias('pos_prob'))
    .sort('pos_prob', ascending=False))

negative_predictive_words = (predictive_words
    .select(col('word').alias('negative_word'), col('negative').alias('neg_prob'))
    .sort('neg_prob', ascending=False))

import pandas as pd

pd.concat([
    positive_predictive_words.toPandas().head(n=20),
    negative_predictive_words.toPandas().head(n=20) ],
    axis=1)


I am working on the file "Summary and Results v.0.ipynb".
The first task is:
Discriminate Planarity with PCA
The first method uses eigenvectors for the curves along first two axes of maximum eigenvalues for projection. Then the curve is reconstructed in three dimensions by re-projecting the points along the original axes.
Can you write Python code for it?

# PCA analysis and plot
helper.plot_PCA_errors()


I am working on the file "Summary and Results v.0.ipynb".
The first task is:
Non-Planar Curve Generation
The process to create non-planar curves is less restrictive. We divide the cube into 8 sub-cubes and randomly choose points from each sub-cube. The chosen points are interpolated using spline to give us 300 3-dimensional points in the space contained within the unit cube.
Can you write Python code for it?


I am working on the file "Summary_of_the_2021_Lighthouse_Labs_21_Day_Data_Challenge.ipynb".
The first task is:
Day 3: List Slicing, Dictionaries, and Indexing
Link: https://data-challenge.lighthouselabs.ca/challenge/3
List Slicing
Revisiting lists, sometimes we need to extract more than one item in the list. Recall that lists are indexed starting at 0, as is shown in this previous example:
Can you write Python code for it?


I am working on the file "Summary_of_the_2021_Lighthouse_Labs_21_Day_Data_Challenge.ipynb".
The first task is:
If any of the results above look strange, it's because of the way computers do math. For further reading about this, see:
https://stackoverflow.com/questions/21895756/why-are-floating-point-numbers-inaccurate
Comments
Did you notice the comments in the code above? Anything after a "#" is a comment. Use lots of comments in your code!
round()
If we want to round any of the results, we can use the round() function:
Can you write Python code for it?

print(a / b)            # no rounding
print(round(a / b, 2))  # 2 decimals places


I am working on the file "Summary_of_the_2021_Lighthouse_Labs_21_Day_Data_Challenge.ipynb".
The first task is:
Note that you leave the first part of the argument (the part in the brackets) blank, the slice starts at [0]. If you leave the second part of the argument blank, it ends at the last item:
Can you write Python code for it?

print(fruits[:2]) # 'Apple', 'Pear', same as [0:2]
print(fruits[4:]) # 'Kiwi', 'Grapefruit', 'Kumquat'


I am working on the file "Summary_of_the_2021_Lighthouse_Labs_21_Day_Data_Challenge.ipynb".
The first task is:
Note the 'f' preceding the first quote of the string and that the variables are contained within braces.
For more information about f-strings, see https://zetcode.com/python/fstring/.
Day 2: Data Types, Lists, sum(), round(), Simple Math
Link: https://data-challenge.lighthouselabs.ca/challenge/2
Numeric Data Types
There are two types of numeric data types: integer (whole number) and float, like fractions or decimals.
Lists
Lists are a sequence of variables separated by commas and contained within brackets, such as:
Can you write Python code for it?

list_of_int = [1,2,3,4,5] #list of integers
list_of_string = ['string1','string2', 'string3'] #list of strings
list_of_anything = [1, 'string', 3.2] #list with an integer, string, and float.


I am working on the file "Summary_of_the_2021_Lighthouse_Labs_21_Day_Data_Challenge.ipynb".
The first task is:
Simple Math
Let's do some simple math:
Can you write Python code for it?

a = 34.2
b = 21.72

print(a + b) # addition
print(a - b) # subtraction
print(a * b) # multiplication
print(a / b) # division
print(2 ** 3) # exponentiation
print(5 // 2) # integer division
print(5 % 2) # remainder (modulus)


I am working on the file "Summary_of_the_2021_Lighthouse_Labs_21_Day_Data_Challenge.ipynb".
The first task is:
What if you want to print every 2nd item? Check this out:
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/CreateAHistogram_Solution.ipynb".
The first task is:
Time for styling the histogram and use jsroot
Can you write Python code for it?

%jsroot on
h.SetLineColor(ROOT.kBlue)
h.SetFillColor(ROOT.kBlue)
c = ROOT.TCanvas()
h.Draw()
c.Draw()


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/CreateAHistogram_Solution.ipynb".
The first task is:
We now create our histogram
Can you write Python code for it?

h = ROOT.TH1F("h", "My Notebook Histo;x;#", 64, -4, 4)


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/CreateAHistogram_Solution.ipynb".
The first task is:
We now import the gauss generation from the random module of Python and fill the histogram
Can you write Python code for it?

from random import gauss
numbers = [gauss(0., 1.) for _ in range(1000)]
numbers

for i in numbers: h.Fill(i)


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample.ipynb".
The first task is:
Construct the histogram containing the input data
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample.ipynb".
The first task is:
Create the function and try to fit it without setting any parameter. Activate the interactive visualisation with %jsroot on.
Remember that in order to embed a plot in a notebook, you'll need to create a TCanvas and draw it explicitly.
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample.ipynb".
The first task is:
Less than optimal. Set parameters and fit again, draw histogram with error bars
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample.ipynb".
The first task is:
Much better. Now time to beautify the plot. Construct a TF1 for the background and Lorentzian functions and draw them in the same canvas.
We save the fit results and set the parameters of the functions accordingly
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample.ipynb".
The first task is:
We can now add a legend
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample_Solution.ipynb".
The first task is:
Construct the histogram containing the input data
Can you write Python code for it?


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample_Solution.ipynb".
The first task is:
Create the function and try to fit it without setting any parameter
Can you write Python code for it?

%jsroot on
c = ROOT.TCanvas()
nparams = 6
fitFcn = ROOT.TF1('fitFcn', ROOT.fitFunction, xlow, xup, nparams)

histo.Fit(fitFcn)
c.Draw()


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/FittingExample_Solution.ipynb".
The first task is:
Less than optimal. Set parameters and fit again, draw histogram with error bars
Can you write Python code for it?

fitFcn.SetParameter(4, 0.2); # width
fitFcn.SetParameter(5, 1);   # peak
histo.Fit(fitFcn)
histo.Draw('E')
c.Draw()


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/graphDraw_Solution.ipynb".
The first task is:
Change marker style, colour as well as line colour and thickness. Make the plot interactive. Re-draw the plot and interact with it!
Can you write Python code for it?

%jsroot on
g.SetMarkerStyle(ROOT.kFullTriangleUp)
g.SetMarkerSize(3)
g.SetMarkerColor(ROOT.kAzure)
g.SetLineColor(ROOT.kRed - 2)
g.SetLineWidth(2)
g.SetLineStyle(3)
c.Draw()


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/graphDraw_Solution.ipynb".
The first task is:
Now we set the title and the grid on the canvas.
Can you write Python code for it?

g.SetTitle("My Graph;The X;My Y")
c.SetGrid()
c.Draw()


I am working on the file "SummerStudentCourse/2019/Exercises/ROOTBooks/graphDraw_Solution.ipynb".
The first task is:
We will now add the symbol of the Calcium isotope
Can you write Python code for it?

txt = "#color[804]{My text #mu {}^{40}_{20}Ca}"
l = ROOT.TLatex(.2, 10, txt)
l.Draw()
c.Draw()


I am working on the file "SummerStudentCourse/2019/Exercises/WorkingWithFiles/WritingOnFilesExercise.ipynb".
The first task is:
Inspect the content of the file: TXMLFile
ROOT provides a different kind of TFile, TXMLFile. It has the same interface and it's very useful to better understand how objects are written in files by ROOT.
Repeat the exercise above, either on Python or C++ - your choice, using a TXMLFILE rather than a TFile and then display its content with the cat command. Can you see how the content of the individual bins of the histograms is stored? And the colour of its markers?
Do you understand why the xml file is bigger than the root one even if they have the same content?
Can you write Python code for it?

f = ROOT.TXMLFile("histos.xml","RECREATE")

hg = ROOT.TH1F("gaus","Gaussian numbers", 64, -4, 4)
he = ROOT.TH1F("expo","Exponential numbers", 64, -4, 4)
hu = ROOT.TH1F("unif","Uniform numbers", 64, -4, 4)
for i in xrange(1024):
    hg.Fill(rndm.Gaus())
    # ... Same as above!

! ls -l histos.xml histos.root

! cat histos.xml


I am working on the file "SummerStudentCourse/2019/Exercises/WorkingWithFiles/WritingOnFilesExercise.ipynb".
The first task is:
Writing histograms
Create a TFile containing three histograms filled with random numbers distributed according to a Gaus, an exponential and a uniform distribution.
Close the file: you will reopen it later.
Can you write Python code for it?

rndm = ROOT.TRandom3(1)

filename = "histos.root"

# Here open a file and create three histograms

for i in xrange(1024):
    # Use the following lines to feed the Fill method of the histograms in order to fill
    rndm.Gaus()
    rndm.Exp(1)
    rndm.Uniform(-4,4)

# Here write the three histograms on the file and close the file


I am working on the file "SummerStudentCourse/2019/Exercises/WorkingWithFiles/WritingOnFilesExercise.ipynb".
The first task is:
You can now repeat the exercise above using C++. Transform the cell in a C++ cell using the %%cpp "magic".
Can you write Python code for it?

%%cpp
TFile f("histos.root");
TH1F *hg, *he, *hu;
f.GetObject("gaus", hg);
// ... read the histograms and draw them in each pad


I am working on the file "SummerStudentCourse/2019/Exercises/WorkingWithFiles/WritingOnFiles_Solution.ipynb".
The first task is:
Access the histograms and draw them in Python. Remember that you need to create a TCanvas before and draw it too in order to inline the plots in the notebooks.
You can switch to the interactive JavaScript visualisation using the %jsroot on "magic" command.
Can you write Python code for it?

%jsroot on
f = ROOT.TFile("histos.root")
c = ROOT.TCanvas()
c.Divide(2,2)
c.cd(1)
f.gaus.Draw()
c.cd(2)
f.expo.Draw()
c.cd(3)
f.unif.Draw()
c.Draw() # Draw the Canvas


I am working on the file "SummerStudentCourse/2019/Exercises/WorkingWithFiles/WritingOnFiles_Solution.ipynb".
The first task is:
Now, you can invoke the ls command from within the notebook to list the files in this directory. Check that the file is there. You can invoke the rootls command to see what's inside the file.
Can you write Python code for it?

! ls .
! echo Now listing the content of the file
! rootls -l ./histos.root


I am working on the file "Summer_15/Employment-Population-Ratio_DavidCai_Jul_15.ipynb".
The first task is:
Source: Figure created using data from the Bureau of Labor Statistics (BLS) accessed through the Federal Reserve Economic Data (FRED). This graph is adapted from Aaronson et al. (2014)’s Figure 9. Recession data is from NBER accessed through FRED.
Changes in the Age Distribution
As population aging is the largest contributor to the labor force participation decline, further analysis is necessary to understand its nature. Aaronson et al. (2014) observe that the proportion of the working age population reported as retired in the Current Population Survey (CPS) has increased by more than one percent in 2014 compared to 2007, accounting for the majority of the 1.3 percent effect of aging. The authors argue that this change is the result of a shift of the age distribution of the population, as the leading edge of the baby boom generation reaches age 62. However, on the contrary, within-age participation rates have increased since 2007, making a positive contribution to total labor force participation (Figure 3). Aaronson et al. (2014) make a similar finding, observing that within-age retirement rates have decreased, likely due to changes in social security and pensions, increased education levels, and longer life spans. These same factors can also explain the increase in the within-age participation rates among older cohorts. That said, the most important implication of Figure 3 is that labor force participation rates decrease with age. As the population age distribution shifts towards older ages, overall labor force participation can be expected to decrease.
Can you write Python code for it?

#file = '/Users/davidcai/lfpr.csv'
file = 'https://raw.githubusercontent.com/DaveBackus/Data_Bootcamp/master/Code/Projects/lfpr.csv'
df = pd.read_csv(file, index_col=0)

start, end = dt.datetime(1980, 1, 1), dt.datetime(2010, 1, 1)
data = web.DataReader('USREC', 'fred', start, end)
data.columns=['Recession']

# Take a simple averages of ratios for men and women
df["Age 62"] = df[["M62-64", "W62-64"]].mean(axis=1)
df["Age 65"] = df[["M65-69", "W65-69"]].mean(axis=1)
df["Age 70"] = df[["M70-74", "W70-74"]].mean(axis=1)
df["Age 75"] = df[["M75-79", "W75-79"]].mean(axis=1)

# Convert years into datetime series
df.index = df.index.astype(str) + "-1-1"
df.index = pd.to_datetime(df.index)
plt.figure(figsize=(plt.figaspect(0.5)))

df["Age 62"].plot()
df["Age 65"].plot()
df["Age 70"].plot()
df["Age 75"].plot()

plt.text(dt.datetime(2007, 1, 1), 42, 'Age 62', fontsize=11, weight='bold')
plt.text(dt.datetime(2007, 1, 1), 25, 'Age 65', fontsize=11, weight='bold')
plt.text(dt.datetime(2007, 1, 1), 15, 'Age 70', fontsize=11, weight='bold')
plt.text(dt.datetime(2007, 1, 1), 6, 'Age 75', fontsize=11, weight='bold')

shade_recession(get_recession_months())

plt.suptitle('Figure 3. Labor Force Participation Rates, By Age, 1980-2010', fontsize=12, fontweight='bold')
plt.show()


I am working on the file "Summer_15/Employment-Population-Ratio_DavidCai_Jul_15.ipynb".
The first task is:
Source: Figure created using data from the Bureau of Labor Statistics (BLS) accessed through the Federal Reserve Economic Data (FRED). This graph is updated from Moffitt (2012)’s Figure 2. Recession data is from NBER accessed through FRED.
Labor Force Participation
Since 1976, the labor force participation rate has trended upwards until hitting a peak around 2000 (Figure 2). Aaronson et al. (2006) note that this trend can be extended back to the early 1960s, with labor force participation rising from less than 60 percent its peak of 67.3 percent in 2000. After 2000, a reversal of the previous trend emerged, with a new trend of labor force decline until today. Aaronson et al. point out that a prolonged decline in labor force participation is unprecedented in the postwar era, thus leading observers to wonder if long-term structural changes in the labor market have occurred.
After the publication of the 2006 paper, the labor force participation rate has continued to fall. Revisiting this issue, Aaronson et al. (2014) examine the decline in labor force participation from 2007 onwards. They attempt to break down the factors contributing to this decline into structural and cyclical components. The authors find that 1.3 percent, or nearly one half, of the 2.8 percent decline in the aggregate participation rate can be attributable population aging. Moreover, they note the contributions of declines in specific age/sex categories, such as among youth and adult men. Finally, they discover the existence of a cyclical component; however, its magnitude is much more uncertain. Of these three components, population aging represents the largest contributor to the labor force participation decline.
Can you write Python code for it?

start, end = dt.datetime(1976, 1, 1), dt.datetime(2015, 3, 1)
data = web.DataReader(['CIVPART', 'USREC'], 'fred', start, end)
data.columns = ['LFPR', 'Recession']
plt.figure(figsize=plt.figaspect(0.5))
data['LFPR'].plot(color = 'k')
plt.xlabel('')
shade_recession(get_recession_months())
plt.suptitle('Figure 2. Labor Force Participation Rate, 1976-2015', fontsize=12, fontweight='bold')
plt.show()


I am working on the file "Summer_15/PISA_SusanChen_Aug_15.ipynb".
The first task is:
Excluding Outliers
I initially plotted the data and ran the regression without excluding any outliers. The resulting r-squared values for reading, math, and science were 0.29, 0.32, and 0.27, respectively. Looking at the scatter plot, there seem to be two obvious outliers, Qatar and Vietnam. I decided to exclude the data for these two countries because the remaining countries do seem to form a trend. I found upon excluding them that the correlation between GDP per capita and scores was much higher.
Qatar is an outlier as it placed relatively low, 63rd out of the 65 countries, with a relatively high GDP per capita at about $131000. Qatar has a high GDP per capita for a country with just 1.8 million people, and only 13% of which are Qatari nationals. Qatar is a high income economy as it contains one of the world's largest natural gas and oil reserves.
Vietnam is an outlier because it placed relatively high, 17th out of the 65 countries, with a relatively low GDP per capita at about $4900. Reasons for Vietnam's high score may be due to the investment of the government in education and the uniformity of classroom professionalism and discipline found across countries. At the same time, rote learning is much more emphasized than creative thinking, and it is important to note that many disadvantaged students are forced to drop out, reasons which may account for the high score.
Can you write Python code for it?

df3.index = df3.Country #set country column as the index
df3 = df3.drop(['Qatar', 'Vietnam']) # drop outlier


I am working on the file "Summer_15/PISA_SusanChen_Aug_15.ipynb".
The first task is:
Plotting the Data
I use the log of the GDP per capita to plot against each component of the PISA on a scatter plot.
Can you write Python code for it?

Reading = df3.Reading
Science = df3.Science
Math = df3.Math
GDP = np.log(df3.GDPpc)

#PISA reading vs GDP per capita
plt.scatter(x = GDP, y = Reading, color = 'r')
plt.title('PISA 2012 Reading scores vs. GDP per capita')
plt.xlabel('GDP per capita (log)')
plt.ylabel('PISA Reading Score')
plt.show()

#PISA math vs GDP per capita
plt.scatter(x = GDP, y = Math, color = 'b')
plt.title('PISA 2012 Math scores vs. GDP per capita')
plt.xlabel('GDP per capita (log)')
plt.ylabel('PISA Math Score')
plt.show()

#PISA science vs GDP per capita
plt.scatter(x = GDP, y = Science, color = 'g')
plt.title('PISA 2012 Science scores vs. GDP per capita')
plt.xlabel('GDP per capita (log)')
plt.ylabel('PISA Science Score')
plt.show()


I am working on the file "Summer_15/PISA_SusanChen_Aug_15.ipynb".
The first task is:
Regression Analysis
The OLS regression results indicate that the there is a 0.57 correlation betweeen reading scores and GDP per capita, 0.63 between math scores and GDP per capita, and 0.57 between science scores and GDP per capita.
Can you write Python code for it?

lm = smf.ols(formula='Reading ~ GDP', data=df3).fit()
lm2.params
lm.summary()

lm2 = smf.ols(formula='Math ~ GDP', data=df3).fit()
lm2.params
lm2.summary()

lm3 = smf.ols(formula='Science ~ GDP', data=df3).fit()
lm3.params
lm3.summary()


I am working on the file "Summer_school_2016.ipynb".
The first task is:
$\chi^2$ is biased in this case because it assumes the wrong noise model (gaussian) while our data have Poisson noise. If you were to use a larger value for a_true, then the situation will get better until the $\chi^2$ minimization would work as good as the Poisson likelihood maximization. The reason is that for large $n$ the Poisson distribution approaches the Gaussian distribution with $\sigma = \sqrt{n}$.
The Maximum Likelihood estimator is consistent
Consistent means that we can measure the unknown quantity up to an arbitrary precision level if we can increase the amount of data sufficiently.
To see that, let's try to repeat the procedure we did in the previous section with a variable amount of data:
Can you write Python code for it?

# Let's generate the model with a variable quantity of data

n_bins_to_try = [10,100,1000]

for n_bins in n_bins_to_try:

    # Generate number of bins in x
    # (NOTE: we are generating n_bins bins from 0 to n_bins)

    bin_boundaries = np.linspace(1, n_bins, n_bins)

    bins = Bins(bin_boundaries)
    data = data_generative_process.generate(bins)

    like = Likelihood(bins, data, model, 'poisson')

    this_a_mle = like.generate_and_fit(data_generative_process, 1000)

    # We can make an histogram of the MLE estimates
    histogram = plt.hist(this_a_mle, 20, label='N = %i' % n_bins, histtype='step')
    plt.xlabel("a")

# plot the vertical like of the true value
plt.axvline(a_true, color='red', lw=2, linestyle='--', zorder=100)

plt.legend()


I am working on the file "Summer_school_2016.ipynb".
The first task is:
Bias
Here we show that the MLE estimator is unbiased, i.e., that the expected value for the MLE estimate for a parameter is the true value. The expected value is the average over many (-> inf) observations of the same process.
Can you write Python code for it?

# Now repeat the whole analysis (from data generation on) a certain number of times

# I prepared a convenience function to do that.
# This function regenerates some data from the same generative process
# used above, then it fits them and returns the list of MLE estimates
# for a (one for each iteration)
# Let's do it 1000 times

many_a_mle = like.generate_and_fit(data_generative_process, 1000)

# Now let's plot the MLEs for a

plt.plot(many_a_mle,'.')
plt.ylabel(r"a$_{MLE}$")
plt.xlabel("iteration")

# Plot the true value
plt.axhline(a_true, color='red',lw=2,linestyle='--')

# We can make an histogram of the MLE estimates
histogram = plt.hist(many_a_mle, 20)
plt.xlabel("a")
# plot the vertical like of the true value
plt.axvline(a_true, color='red', lw=2, linestyle='--', zorder=100)


I am working on the file "Summer_school_2016.ipynb".
The first task is:
Errors on the Maximum Likelihood estimate
We have seen in the previous section that the average MLE estimate is close to the true value, and becomes closer and closer as the amount of data increases.
However the MLE estimate on a single dataset will be different from the true value, and might be distant from it.
Also, in reality we cannot usually re-generate the data that we have. In Astronomy in particular many times we have observed an event which will never repeat itself. So we cannot really produce histograms like the ones we produced before, because we have only one measurement.
We need to introduce an error measurement. Before doing that, let's see what a likelihood profile is:
Can you write Python code for it?

# Let's prepare a grid in possible values for a,
# between the 80% and 120% of the true value
# (this is arbitrary)

a_s = np.linspace(a_true * 0.8, a_true * 1.2,300)

# Let's generate data and fit them
# Let's use a small quantity of data first

n_bins1 = 100

bin_boundaries1 = np.linspace(1,n_bins1,n_bins1)
bins1 = Bins(bin_boundaries1)
data1 = data_generative_process.generate(bins1)

like1 = Likelihood(bins1, data1, model, 'poisson')
a_mle1 = like1.maximize()

# This goes through all the a_s values and for each a compute
# L(D|a)

profile1 = like1.profile(a_s)

# Now let's do the same for a larger quantity of data

n_bins2 = 10000

bin_boundaries2 = np.linspace(1,n_bins2, n_bins2)
bins2 = Bins(bin_boundaries2)
data2 = data_generative_process.generate(bins2)

like2 = Likelihood(bins2, data2, model, 'poisson')
a_mle2 = like2.maximize()

profile2 = like2.profile(a_s)

plt.plot(a_s, profile1 - profile1.max(), label='Few data')
plt.plot(a_s, profile2 - profile2.max(), label='Many data')
plt.xlabel("a")
plt.ylabel("log. likelihood shifted to 0")
plt.ylim([-5,1])
plt.axvline(a_true, linestyle='--',lw=2, color='red')


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
And now we can call it like this:
Can you write Python code for it?

tagged_tokens_baseline = extract_baseline(de_bello_gallico_book1)

tagged_tokens_baseline[-50:]


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
And the resulting output looks now ok:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
At this point the passage is available in various formats: text, but also TEI XML, etc.
Thus, we need to specify that we are interested in getting the text only:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
But we can also query the same API and get back the text of a specific text section, for example the entire book 1.
To do so, we need to append the indication of the reference scope (i.e. book 1) to the URN.
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
But, as you can see, the two lists are not aligned.
This is due to how the CLTK function tokenises the text. The comma after "tres" becomes a token on its own, whereas when we tokenise by white space the comma is attached to "tres" (i.e. "tres,").
A solution to this is to pass to the tag_ner function the text already tokenised by text.
Can you write Python code for it?

tagged_text_cltk = reshape_cltk_output(tag_ner('latin', input_text=de_bello_gallico_book1.split(" ")))

list(zip(tagged_text_baseline[:20], tagged_text_cltk[:20]))


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Excercise
Extract the named entities from the English translation of the De Bello Gallico book 1.
The CTS URN for this translation is urn:cts:latinLit:phi0448.phi001.perseus-eng2:1.
Modify the code above to use the English model of the Stanford tagger instead of the italian one.
Hint:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Let's a havea  look at the first 50 tokens that we just tagged:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Let's check that the text is there by printing the content of the variable de_bello_gallico_book1 where we stored it:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Let's grab some text
To start with, we need some text from which we'll try to extract named entities using various methods and libraries.
There are several ways of doing this e.g.:
1. copy and paste the text from Perseus or the Latin Library into a text document, and read it into a variable
2. load a text from one of the Latin corpora available via cltk (cfr. this blog post)
3. or load it from Perseus by leveraging its Canonical Text Services API
Let's gor for #3 :)
What's CTS?
CTS URNs stand for Canonical Text Service Uniform Resource Names.
You can think of a CTS URN like a social security number for texts (or parts of texts).

Here are some examples of CTS URNs with different levels of granularity:
- urn:cts:latinLit:phi0448 (Caesar)
- urn:cts:latinLit:phi0448.phi001 (Caesar's De Bello Gallico)
- urn:cts:latinLit:phi0448.phi001.perseus-lat2 DBG Latin edtion
- urn:cts:latinLit:phi0448.phi001.perseus-lat2:1 DBG Latin edition, book 1
- urn:cts:latinLit:phi0448.phi001.perseus-lat2:1.1.1 DBG Latin edition, book 1, chapter 1, section 1
How do I find out the CTS URN of a given author or text? The Perseus Catalog is your friend! (crf. e.g. http://catalog.perseus.org/catalog/urn:cts:latinLit:phi0448)
Querying a CTS API
The URN of the Latin edition of Caesar's De Bello Gallico is urn:cts:latinLit:phi0448.phi001.perseus-lat2.
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Let's have a look at the ouput, only the first 10 tokens (by using the list slicing notation):
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Let's have a look at the output
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Let's see how many words (tokens, more properly) there are in Caesar's De Bello Gallico I:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
NER with CLTK
The CLTK library has some basic support for the extraction of named entities from Latin and Greek texts (see CLTK's documentation).
The current implementation (as of version 0.1.47) uses a lookup-based method.
For each token in a text, the tagger checks whether that token is contained within a predefined list of possible named entities:
- list of Latin proper nouns: https://github.com/cltk/latin_proper_names_cltk
- list of Greek proper nouns: https://github.com/cltk/greek_proper_names_cltk
Let's run CLTK's tagger (it takes a moment):
Can you write Python code for it?

%%time
tagged_text_cltk = tag_ner('latin', input_text=de_bello_gallico_book1)


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
NER with NLTK
Can you write Python code for it?

stanford_model_italian = "/opt/nlp/stanford-tools/stanford-ner-2015-12-09/classifiers/ner-ita-nogpe-noiob_gaz_wikipedia_sloppy.ser.gz"

ner_tagger = StanfordNERTagger(stanford_model_italian)

tagged_text_nltk = ner_tagger.tag(de_bello_gallico_book1.split(" "))


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Saving the outputs to a pickle file for later
Can you write Python code for it?

import pickle

with open('data/DBG_tagged_baseline.pickle','wb') as f:
    pickle.dump(tagged_text_baseline, f)

with open('data/DBG_tagged_clkt.pickle','wb') as f:
    pickle.dump(tagged_text_cltk, f)

with open('data/DBG_tagged_nltk.pickle','wb') as f:
    pickle.dump(tagged_text_nltk, f)


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
So we retrieve the first book of the De Bello Gallico by passing its CTS URN (that we just stored in the variable my_passage) to the CTS API, via the resolver provided by MyCapytains:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
The output looks slightly different from the one of our baseline function (the size of the tuples in the list varies).
But we can write a function to fix this, we call it reshape_cltk_output:
Can you write Python code for it?

def reshape_cltk_output(tagged_tokens):
    reshaped_output = []
    for tagged_token in tagged_tokens:
        if(len(tagged_token)==1):
            reshaped_output.append((tagged_token[0], "O"))
        else:
            reshaped_output.append((tagged_token[0], tagged_token[1]))
    return reshaped_output


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
We apply this function to CLTK's output:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 1b.ipynb".
The first task is:
Wrap up
At this point we can "compare" the output of the three different methods we used, again by using the zip function.
Can you write Python code for it?

list(zip(tagged_text_baseline[:20], tagged_text_cltk[:20], tagged_text_nltk[:20]))

for baseline_out, cltk_out, nltk_out in zip(tagged_text_baseline[:20], tagged_text_cltk[:20], tagged_text_nltk[:20]):
    print("Baseline: %s\nCLTK: %s\nNLTK: %s\n"%(baseline_out, cltk_out, nltk_out))


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
A bonus: objects
Objects might be a bit complicated, but they're very important to understand the code written by other people, while most of the programs that you'll find around is written using classes and objects. Oh, and the good news is... you've already met them!
What are "objects" in a programming language like Python? Well, I like to think about them as... magical, animated tools!

Say that you want to fetch water from a well (and maybe clean some of the mess...). Well, the object-oriented approach to this tak consists in creating one or more magic brooms that go and fetch the water for you! In order to create them, you have to conceptualize the broom in terms of:

the special features it has (e.g. number of buckets carried, speed...)
the actions that it can execute (fetch water, clean the floor)

That's it! In programming parlance, the features are called properties of the object; the actions are called methods.
When you want to build your own magic brooms you first create a sort of prototype for each of them (which is called the class of magic brooms); then you can go on and create as many brooms as you want...
Here's how to do it! (very simplified)
Can you write Python code for it?

class MagicBroom():
    #this is called "constructor"; it's a special method
    def __init__(self, name, speed=20):
        self.name = name
        self.buckets = 2
        self.speed = speed

    def greet(self):
        print("Hello, my name is %s! What can I do for you?" % self.name)

    def fetchWater(self):
        if self.speed >= 20:
            print("Yes, sir! I'll be back in a sec!")
        else:
            print("Allright, but I am taking my time!")

mickey = MagicBroom("Mickey")
mickey.greet()

peter = MagicBroom("Peter", speed=5)

mickey.speed

mickey.fetchWater()

peter.fetchWater()


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Data types
Can you write Python code for it?

#interges and floats
3 + 0.5
#strings
"hello"
#Booleans
True


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Exercise
In the practical exercise, you are requested to extract the person names from the same article that we used for dates. You will annotate them using the Stanford NER with the pre-trained classifier for English that come with the software; extract the Person chunks; evaluate the results against a golden standard.
Here is a summary of the steps that you will have to execute in order to solve the exercise:

load the file: data/txt/article446_10k.txt and read its content
annotate the Named Entities using Stanford NER
define an appropriate chunker for Persons
chunk the extracted Named Entities
convert the chunked Tree into IOB format
evaluate the IOB annotation using the appropriate functions
use the file: data/iob/article_446_person_GOLD.iob as gold standard

report the final evaluation metrics (precision, recall, F-score)
Can you write Python code for it?

#just remember that the path to the English pre-trained classifier for Stanfor NER is
english_classifier = 'english.all.3class.distsim.crf.ser.gz'


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Functions
Can you write Python code for it?

def printMe(message):
    print(message)

printMe("Hello, world!")
printMe("goodbye...")


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Handling exceptions
Can you write Python code for it?

l = ["zero", "one", "two", "three"]
l[10]

try:
    l[10]
except IndexError:
    print("hey, your index is way too high!")


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Kind of a sloppy job we did! The negative numbers are not captured as negative; the footnote reference (e.g. [1], [4]) are also captured and we don't want them... We can do better. Let's improve our pattern so that we include the '-' signs (if present) and we get rid of the footnotes
Can you write Python code for it?

reg = re.compile(r'(?<!\[)−?\d+(?!\])') # the 'r' is there to make sure that we don't have to "escape the escape" sing (\)
reg.findall(wiki)


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Named Entity Recognition (using a tool like the Stanford NER that we saw in our last lecture) is also a way of tagging the text, this time using information not on the word class but on a different level of classification (place, person, organization or none of the above).
Let's do this too
Can you write Python code for it?

#first, we define the path to the English classifier for Stanford NER
english_classifier = 'english.all.3class.distsim.crf.ser.gz'
twords = [w[0] for w in tagged]

#then... guess what? Yes, we create a NER-tagger Magic Broom ;-)
from nltk.tag import StanfordNERTagger

ner_tagger = StanfordNERTagger(english_classifier)
ners = ner_tagger.tag(twords)

#not very pretty...
ners[:20]


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Now let's see it in action on the real stuff
Can you write Python code for it?

reg_tag = tagger.tag(twords)

reg_tag[:50]


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Now we just need to chunk it and export it to IOB. Then we are ready to evaluate the manual annotation...
First, we have to define a chunker
Can you write Python code for it?

date_chunker = RegexpParser(r'''
DATE:
    {<CD>*<Date><Date|CD>*}
DATE:
    {<CD>+}
''')

t = date_chunker.parse(reg_tag)

#we use that function to make sure that the tree is not too complex to be converted
flat = dainlp.flatten_tree(t)

iob_list = tree2conlltags(flat)

iob_list[:50]

#then we can write it on an output file
with open("data/iob/article_446_date_aut.iob", "w") as out:
    for i in iob_list:
        out.write("\t".join(i)+"\n")


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Part-Of-Speech (POS) and Named-Entity (NE) Tagging
Most of the time POS tagging is the precondition before you can perform any other advanced operation on a text
As we did with Matteo last time, by "tagging" we mean the coupling of each word with a tag that describe some property of the word itself. Part-of-speech tags define what word class (e.g. "verb", or "proper noun") a text token belongs to.
There are several tagset used for each language, and several software (pos taggers) who can tag your text automatically. One of the most used is TreeTagger, which has pretrained classifiers for many languages.
Let's run it from Python, using one of the few "wrappers" available
Can you write Python code for it?

#first we load the library
from treetagger import TreeTagger

#That's right! we start by creating a Tagger "magic broom" (a Tagger object)
tt = TreeTagger(language="english")

#then we tag our text
tagged = tt.tag(txt)

tagged[:20]


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Regex tagger
Now, to go back to our original task, how do we use all this to annotate the dates and export them to IOB?
Dates are often just numbers (e.g. "2017"); sometimes they come in more complex formats like: "14 September 2017" or "14-09-2017".
One very simple solutions to find them and annotate them with a chunking notation might be to tag the tokens of our text with a very simple custom tagset that we design for dates. We assign "O" to all tokens, save the numbers (that we tag "CD") and some selected time formats or expressions, like the months of the year or the sequence number-number. We use the tag "Date" for them.
In order to do this, we need:

regular expression syntax
a tagger that works with RegExp patterns

A module of NLTK provides with exactly that tagger that can work with RegExp syntax
Can you write Python code for it?

from nltk.tag import RegexpTagger

#here is our list of patterns
patterns = [
    (r'\d+$', 'CD'),
    (r'\d+[-–]\d+$', "Date"),
    (r'\d{1,2}[-\.\/]\d{1,2}[-\.\/]\d{2,4}', "Date"),
    (r'January|February|March|April|May|June|July|August|September|October|November|December', "Date"),
    (r'\d{4}$', "Date"),
    (r'BCE|BC|AD', "Date"),
    (r'.*', "O")
]

#Our RegexpTagger magic broom! We initialize it with our pattern list
tagger = RegexpTagger(patterns)

#let's test it with a trivial example
tagger.tag("I was born on September 14 , or 14-09".split(" "))


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Regular expressions
How would you find all the numbers in this sentence?

The set of integers consists of zero (0), the positive natural numbers (1, 2, 3, …), also called whole numbers or counting numbers,[1][2] and their additive inverses (the negative integers, i.e., −1, −2, −3, …). This is often denoted by a boldface Z ("Z") or blackboard bold Z {\displaystyle \mathbb {Z} } \mathbb {Z} (Unicode U+2124 ℤ) standing for the German word Zahlen ([ˈtsaːlən], "numbers").[3][4] ℤ is a subset of the sets of rational and real numbers and, like the natural numbers, is countably infinite.
Can you write Python code for it?

wiki = 'The set of integers consists of zero (0), the positive natural numbers (1, 2, 3, …), also called whole numbers or counting numbers,[1][2] and their additive inverses (the negative integers, i.e., −1, −2, −3, …). This is often denoted by a boldface Z ("Z") or blackboard bold Z {\displaystyle \mathbb {Z} } \mathbb {Z} (Unicode U+2124 ℤ) standing for the German word Zahlen ([ˈtsaːlən], "numbers").[3][4] ℤ is a subset of the sets of rational and real numbers and, like the natural numbers, is countably infinite.'


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
The Pattern object has a number of interesting methods to search and replace the pattern. Generally, you use them with the text that must be searched as an argument. For instance, findall returns all matches as a list
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
We'd need a way to tell our machine not to look for specific strings, bur rather for classes of strings, i.e. using some sort of meta-character to catch a whole group of signs (e.g. the numbers); then we'd need to tell to optionally include/exclude some other signs, or to catch the numbers only if they're not preceeded/followed by other signs...
That's precisely what Regular Expressions do! They allow you to express a query as a string of metacharacters (or groups of metacharacters).
How do we use them in Python? First, we need to import a module from the Standard Library (i.e. you already have them with Python: no need to install external libraries)
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2a-FM.ipynb".
The first task is:
Well... OK, "Roman Republic" is not a location, but at least the chunking is exactly what we wanted to have, right?
Export to IOB notation
OK, but now how do we convert this to the IOB notation?
Luckily, there's a ready-made function in a module from the NLTK library! Let's load and use it
(just in case, there is also a function that does the reverse: from IOB to tree)
Can you write Python code for it?

from nltk.chunk.util import tree2conlltags

iobs = tree2conlltags(tree)

iobs


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Accuracy
$$acc = \frac{tp+tn}{fp+fn}$$
Can you write Python code for it?

def calc_accuracy(d_errors):
    """
    Calculates the accuracy given the input error dictionary.
    """
    true_predictions = d_errors["tp"] + d_errors["tn"]
    false_predictions = d_errors["fp"] + d_errors["fn"]
    return true_predictions / (true_predictions + false_predictions)

calc_accuracy(errors)

print("Precision: {0:.2f}".format(precision))
print("Recall: {0:.2f}".format(recall))
print("F1-score: {0:.2f}".format(fscore))


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
As you will have noticed, one big limitation of counting entities this way, is that entities consisting of more than one token are treated as separate entities.
What we'd need, instead, is a data format that allows for stating that two or more consecutive tokens are part of the same entity...
Another limitation, is that we are actually counting surface forms of a given entity (cfr. "Caesar", "Caesaris", "Caesari", etc.). For this we'd need to disambiguate each entity by means of a unique identifier.
Counting, sorting, plotting
Two libraries that are very useful when dealing with data are pandas and seaborn.
Pandas is a data analysis library, while seaborn is used to visualise statistical data.
These libraries play nicely together and are often used in combination.
Can you write Python code for it?

import pandas as pd
import seaborn as sns

%matplotlib inline


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
F-1 Score
Finally, the F1 score is a global metric that combines both precision and recall giving them equal importance.
It's defined by the following formula:
$$f1 = \frac{2pr}{p+r}$$
Can you write Python code for it?

def calc_fscore(d_errors):
    """
    Calculates the f-score given the input error dictionary.
    """
    prec = calc_precision(d_errors)
    rec = calc_recall(d_errors)
    if(prec == 0 and rec == 0):
        return 0
    else:
        return 2*((prec * rec) / (prec + rec))

fscore = calc_fscore(errors)

print("Precision: {0:.2f}".format(precision))
print("Recall: {0:.2f}".format(recall))
print("F1-score: {0:.2f}".format(fscore))


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Gathering basic statistics about the extracted NEs
Let's say we want to:
- know how entities by type were extracted
- look at the most frequent NEs
Counting entities by type
Of the three methods we used in the previous session to extract NEs from Caesar's De Bello Gallico, let's take the output of NLTK. In fact, this is the only one with more granulary entity types, where as the other two have just a generic Entity type.
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Let's look at the result:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Let's now do the same for our groundtruth (i.e. the manually corrected data):
Can you write Python code for it?

with codecs.open("data/iob/article_446_date_GOLD.iob","r","utf-8") as f:
    data = f.read()

iob_data_gold = [line.split("\t") for line in data.split("\n") if line!=""]

list(zip(iob_data_gold[:50], iob_data_auto[:50]))

auto_labels = [line[2] for line in iob_data_auto]

len(auto_labels)

gold_labels = [line[2] for line in iob_data_gold]

len(auto_labels)

auto_labels[:20]


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Let's plot now, instead, the distribution of named entity types.
Can you write Python code for it?

# we initialize an empty dictionary
counts = {}

# we iterate through all NE tags
for tag in nltk_tags:
    # we check if our dictionary already contains an item
    # for that specific entity type
    if tag in counts:
        # if it does, we just increase the counter of 1
        counts[tag]+=1
    else:
        # otherwise we add it and set it to 1
        counts[tag] = 1

df_types = pd.DataFrame.from_dict(counts, orient="index")
df_types.columns = ["count"]
df_types


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Let's rename the column that contains the entity counts:
Can you write Python code for it?

df.columns = ["count"]

df.info()

df.head(10)

df.tail(10)


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Now that we have learned how to do this ourselves, it's important to know that the Python library collections already contains an objects that does exactly this: Counter.
Let's look at its documentation:
Can you write Python code for it?

Counter?

from collections import Counter

nltk_tag_counts = Counter(nltk_tags)

nltk_tag_counts


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Now, this is not very useful/readable. Let's try to remove the O labels.
We just modify the code used above and add an if statement:
Can you write Python code for it?

# we initialize an empty dictionary
counts = {}

# we iterate through all NE tags
for tag in nltk_tags:
    # do something only if `tag` is not "O"
    if tag != "O":
        # we check if our dictionary already contains an item
        # for that specific entity type
        if tag in counts:
            # if it does, we just increase the counter of 1
            counts[tag]+=1
        else:
            # otherwise we add it and set it to 1
            counts[tag] = 1
    else:
        pass

df_types = pd.DataFrame.from_dict(counts, orient="index")
df_types.columns = ["count"]
df_types

df_types.plot.pie(y="count")


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Pandas
A key data structure in pandas is the DataFrame, a tabular data structure that allows for arithmetic operations on its contents.
The functionalities provided by pandas' dataframes are very similar to those provided by a spreadsheet software.
First off, we initialise a dataframe representing the named entity counts that we have created previously. Remember?
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Remember the zip function to read two (or more) lists at a time? It's exactly what we need, so let's use it!
This is the kind of output it produces:
Can you write Python code for it?

list(zip(gold_labels, auto_labels))[:10]

assert len(gold_labels) == len(auto_labels)

# we iterate through the two lists of labels
for gold_label, auto_label in list(zip(gold_labels, auto_labels)):
    # label is a negative entity => error type is TN or FP
    if gold_label == "O":
        if gold_label == auto_label:
            errors["tn"]+=1
        else:
            errors["fp"]+=1
    # label is a positive entity => error type is TP or FN
    else:
        if gold_label == auto_label:
            errors["tp"]+=1
        else:
            errors["fn"]+=1

errors


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Seaborn
Seaborn is built on top of matplotlib a very powerful python library for plotting.
Seaborn provides a high-level layer on top of it, and makes some guesses on the nature of the data it receives in input.
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://seaborn.pydata.org/examples/', width=900, height=4000)


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
The first thing to do is to create a list of all named entity tags that were extracted by NLTK:
Can you write Python code for it?

nltk_tags = []
for token, tag in tagged_text_nltk:
    nltk_tags.append(tag)

nltk_tags[:10]


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Using sklearn
The good news is that you don't really need to implement all these maeasures, as there are libraries that can compute precision, recall and f-score for you.
Still, it's important to know what those scores mean and how they are obtained!
Can you write Python code for it?

from sklearn.metrics import precision_recall_fscore_support

precision, recall, fscore, support = precision_recall_fscore_support(gold_labels
                                                                     , auto_labels
                                , average="micro"
                                , labels=["B-DATE","I-DATE"])

print("Precision: {0:.2f}".format(precision))
print("Recall: {0:.2f}".format(recall))
print("F1-score: {0:.2f}".format(fscore))


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
We can generate a basic pie chart by using the plot method of a dataframe:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
We can now sort the entities based on their frequency by using the sort_values method of the dataframe:
Can you write Python code for it?

df.sort_values(by="count", ascending=False)

df


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
We use a utility function provided by the library to create a DataFrame starting from a dictionary.
Can you write Python code for it?

df = pd.DataFrame.from_dict(dict(nltk_entity_counts), orient="index")

df


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
We want to convert this into a list of tuples
Can you write Python code for it?

iob_data_auto = [line.split("\t") for line in data.split("\n") if line!=""]

iob_data_auto


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
Which is equivalent to:
Can you write Python code for it?


I am working on the file "Sunoikisis - Named Entity Extraction 2b.ipynb".
The first task is:
computing Precision, Recall and F-score
Let's create one function to compute each of these measures.
Precision
Precision is the fraction of retrieved entities that are correct.
This measure takes into account the correctly identified entities (TPs) as well as those that were mistakenly tagged as entities (FPs), but does not consider the entities that were missed (FNs).
It's defined by the following formula:
$$p = \frac{tp}{tp+fp}$$
Can you write Python code for it?

def calc_precision(d_errors):
    """
    Calculates the precision given the input error dictionary.
    """
    if(d_errors["tp"] + d_errors["fp"] == 0):
        return 0
    else:
        return d_errors["tp"] / (d_errors["tp"] + d_errors["fp"])

precision = calc_precision(errors)


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
1. Create a collection of plots
The main goal of supersight is to gather your plots and organize these plots within a mini web site.
Actually, you can embed any svg plots but this tool was designed to work with MatPlotlib.
We start by creating a plots gatherer object.
It will contain our plots for easy retrieval:
Can you write Python code for it?


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
And a third graph for the example :
Can you write Python code for it?

graph = plt.figure()
plt.scatter(np.random.randn(100), np.random.randn(100));

buf = io.BytesIO()
plt.savefig(buf, format='svg')
plots.add_plot("third graph", buf)


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
By default, the mini site comes with a home page. We will add two sections. The first section will contain only one page and the second will contain two pages.
As a convention a single-page section is composed of a page named "Page 1".
Can you write Python code for it?

ds.add_section("Section 1")
# We just created a section named Section 1. Pick a custom name that describe an underlying idea of your plots.
ds.sections["Section 1"].add_page("Page 1")

ds.add_section("Section 2")
# We just created a section named Section 2. We said we wanted two pages in this section :
ds.sections["Section 2"].add_page("Interesting Page")
ds.sections["Section 2"].add_page("Great Page") # name it as you want !


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
Each page is organised following a classic Bootstrap grid. Users can display elements on this grid. The default grid is [(6, 6), (6, 6), (6, 6)]. It means the page displays three rows with two elements on each row.
Of course, this layout can be overridden. For the Home page we will display the supersight logo.
According to the Bootstrap grid system the layout for a single row and a single element is [(12,)].
Can you write Python code for it?

ds.sections["Home"].pages["Home"].layout = [(12,)]

ds.sections["Home"].pages["Home"].add_element(name = "logo", plot_object = plots.get_plot("logo"))


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
In section 1, the default grid can stay (hence six plots). Users can populate the grid with their plots. This process populate the grid from left to right and from top to bottom :
Can you write Python code for it?

ds.sections["Section 1"].pages["Page 1"].add_element(name = "plot 1", plot_object = plots.get_plot("first graph"))
ds.sections["Section 1"].pages["Page 1"].add_element(name = "plot 2", plot_object = plots.get_plot("first graph"))

ds.sections["Section 1"].pages["Page 1"].add_element(name = "plot 3", plot_object = plots.get_plot("second graph"))
ds.sections["Section 1"].pages["Page 1"].add_element(name = "plot 4", plot_object = plots.get_plot("second graph"))

ds.sections["Section 1"].pages["Page 1"].add_element(name = "plot 5", plot_object = plots.get_plot("third graph"))
ds.sections["Section 1"].pages["Page 1"].add_element(name = "plot 6", plot_object = plots.get_plot("third graph"))
# add comments :
ds.sections["Section 1"].pages["Page 1"].elements["plot 3"].add_comment_below("This plot will convince you !")


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
Let's add some pandas dataframes as SuperSight can also display data tables within pages :
Can you write Python code for it?

data=pd.read_csv("https://s3.eu-central-1.amazonaws.com/camo-bucket/misc_to_link/data_sample.csv")

df1 = data.head() # a classic df

# a simple pivot table
df2 = pd.pivot_table(data, index = 'Gender',
                     columns = ['Smoking'],
                     aggfunc = np.average)


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
Let's generate a simple plot :
Can you write Python code for it?

graph = plt.figure()
x = np.linspace(0, 10, 30)
y = np.sin(x)
plt.plot(x, y, 'o', color='black');

buf = io.BytesIO()
plt.savefig(buf, format='svg')
plots.add_plot("first graph", buf)


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
Now we can try to vary the layout. For instance on the first row we will display a plot two third the width.
On the second row we can display a unique plot with a heading.
And then three small plots on the last row.
Can you write Python code for it?

ds.sections["Section 2"].pages["Interesting Page"].layout = [(8, 4), (12,), (4, 4, 4)]

ds.sections["Section 2"].pages["Interesting Page"].add_element(name = "plot 1", plot_object = plots.get_plot("third graph"), heading = None)
ds.sections["Section 2"].pages["Interesting Page"].add_element(name = "plot 2", plot_object = plots.get_plot("first graph"), heading = None)

ds.sections["Section 2"].pages["Interesting Page"].add_element(name = "plot 3", plot_object = plots.get_plot("second graph"), heading = "Interesting Heading")

ds.sections["Section 2"].pages["Interesting Page"].add_element(name = "plot 4", plot_object = plots.get_plot("third graph"), heading = None)
ds.sections["Section 2"].pages["Interesting Page"].add_element(name = "plot 5", plot_object = plots.get_plot("third graph"), heading = None)
ds.sections["Section 2"].pages["Interesting Page"].add_element(name = "plot 6", plot_object = plots.get_plot("third graph"), heading = None)

dummy ="""One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a
horrible vermin. He lay on his armour-like back, and if he lifted his head a little he could see his brown belly,
slightly domed and divided by arches into stiff sections. The bedding was hardly able to cover it and seemed ready
to slide off any moment. His many legs, pitifully thin compared with the size of the rest of him, waved about
helplessly as he looked. "What's happened to me?" he thought. It wasn't a dream. His room, a proper human room
although a little too small, lay peacefully between its four familiar walls. A collection of textile samples
lay spread out on the table - Samsa was a travelling salesman -
and above it there hung a picture that he had recently cut out of an illustrated magazine and housed"""

ds.sections["Section 2"].pages["Interesting Page"].elements["plot 1"].add_comment_above(dummy)
ds.sections["Section 2"].pages["Interesting Page"].elements["plot 2"].add_comment_below(dummy[:277])


I am working on the file "SuperSight_Intro.ipynb".
The first task is:
Once your are happy with it, supersight requires you to save your plot in  BytesIO objects.
SVG format is advised since it is lightweight and vector-based.
The three last lines are required to save your plot in a way supersight can handle.
We will make a new plot :
Can you write Python code for it?

graph = plt.figure()
plt.plot(x, y, '-ok')

buf = io.BytesIO()
plt.savefig(buf, format='svg')
plots.add_plot("second graph", buf)


I am working on the file "Super_readl.ipynb".
The first task is:
Classify a new tweet
Can you write Python code for it?

newTweetFeatureVector = vectorizer.transform(["I feel so bad now. Let's go to hell!"])

newTweetFeatureVector

hx_nb.predict(newTweetFeatureVector)

newTweetFeatureVector = vectorizer.transform(["scikit learn is so cool!"])
hx_nb.predict(newTweetFeatureVector)

newTweetFeatureVector = vectorizer.transform(["I am feeling not good with scikit learn"])
hx_nb.predict(newTweetFeatureVector)

hx_nb.predict_proba(newTweetFeatureVector)


I am working on the file "Super_readl.ipynb".
The first task is:
Feature extraction
Conversion of tweets to BOW feature matrix (using only default setting of CountVectorizer)
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
vectorizer

X = vectorizer.fit_transform([item['tweet'] for item in corpus])
X

#X.toarray()

vectorizer.get_feature_names()

y = [item['sent'] for item in corpus]


I am working on the file "Super_readl.ipynb".
The first task is:
Logistic regression with regularization (C is the regularization rate)
$$ \sim O(n)$$
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

hx_log = LogisticRegression(C=0.6)

hx_log.fit(X_train, y_train)

confusion_matrix(y_train, hx_log.predict(X_train))

print "Training set F1: %s" %f1_score(y_train, hx_log.predict(X_train))
print "Test set F1: %s" %f1_score(y_test, hx_log.predict(X_test))


I am working on the file "Super_readl.ipynb".
The first task is:
Randomly split the X and y into training and test set
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=1697)

X_train

X_test

y_train


I am working on the file "Super_readl.ipynb".
The first task is:
Training
Try to fit a naive bayes classifier $$h_{\theta}(X)$$
Naive Bayes convergence rate: $$\sim O(\log{n})$$
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB
hx_nb = MultinomialNB()

hx_nb.fit(X_train, y_train)

hx_nb.predict(X_train)


I am working on the file "Super_readl.ipynb".
The first task is:
Tuning
Tuning the value of C in the above LogisticRegression model
Bigram tokenization
Can you write Python code for it?

bigramvect = CountVectorizer(ngram_range = (1,2))

X_bi = bigramvect.fit_transform([item['tweet'] for item in corpus])

X_bi

X

X_train_bi, X_test_bi, y_train_bi, y_test_bi = train_test_split(X_bi, y, test_size = 0.3, random_state=1697)

bnb = MultinomialNB()
bi_nbhx = bnb.fit(X_train_bi, y_train_bi)

confusion_matrix(y_train_bi, bi_nbhx.predict(X_train_bi))

f1_score(y_train_bi, bi_nbhx.predict(X_train_bi))

f1_score(y_test_bi, bi_nbhx.predict(X_test_bi))

newTweetFeatureVector = bigramvect.transform(["I am feeling not good with scikit learn"])
bi_nbhx.predict(newTweetFeatureVector)[0]


I am working on the file "Super_readl.ipynb".
The first task is:
Typical Noisy data

escape character
url
@handle
Can you write Python code for it?

def getData(csvFname):
    sent = []
    tweet = []
    dataSource = open(csvFname, "r")
    for row in csv.DictReader(dataSource):
        sent.append(row['Sentiment'])
        tweet.append(row['SentimentText'])
    dataSource.close()
    return sent, tweet

sent, tweet = getData("SAsubset.csv")

from scipy.stats import itemfreq
itemfreq(sent)

tweet


I am working on the file "Super_readl.ipynb".
The first task is:
ballpark preprocessing: "unescape", lowercase, remove all puncts
Can you write Python code for it?

tweet[15]

from HTMLParser import HTMLParser
h = HTMLParser()
print h.unescape(tweet[15])

re.sub("[^\w\s]", " ", h.unescape(tweet[15])).lower()


I am working on the file "Super_readl.ipynb".
The first task is:
modify the getData a little and the the 200K tweets dataset.
Can you write Python code for it?

def getData(csvFname):
    h = HTMLParser()
    corpus = []
    dataSource = open(csvFname, "r")
    for row in csv.DictReader(dataSource):
        try:
            corpus.append({"tweet": re.sub("[^a-zA-Z\s]", " ", h.unescape(row['SentimentText'])).lower(), "sent": int(row['Sentiment'])})
        except:
            continue
    dataSource.close()
    return corpus
corpus = getData("SA200K.csv")

print len(corpus)
print corpus[2]


I am working on the file "Super_textbook.ipynb".
The first task is:
Classical algo: Linear Regression
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
hx_lin = LinearRegression()

hx_lin.fit(BX_train, By_train)

hx_lin.predict(BX_train)


I am working on the file "Super_textbook.ipynb".
The first task is:
Classical analysis: Logistic Regression Classifier (Mint)
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
hx_log = LogisticRegression()

hx_log.fit(X_train, y_train)

hx_log.predict(X_train)

# Training set evaluation
print confusion_matrix(y_train, hx_log.predict(X_train))
print f1_score(y_train, hx_log.predict(X_train))

# test set evaluation
print confusion_matrix(y_test, hx_log.predict(X_test))
print f1_score(y_test, hx_log.predict(X_test))


I am working on the file "Super_textbook.ipynb".
The first task is:
Data preparation
Split the data into training set and test set
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(ourdata.data, ourdata.target, test_size=0.3)

print X_train.shape
print y_train.shape
print X_test.shape
print y_test.shape

from scipy.stats import itemfreq
itemfreq(y_train)


I am working on the file "Super_textbook.ipynb".
The first task is:
Let's try an unrealistic algorithm: kNN
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
hx_knn = KNeighborsClassifier()

hx_knn.fit(X_train, y_train)

hx_knn.predict(X_train)


I am working on the file "Super_textbook.ipynb".
The first task is:
Moment of truth: test set performance evaluation
Can you write Python code for it?

print confusion_matrix(y_test, hx_knn.predict(X_test))
print f1_score(y_test, hx_knn.predict(X_test))


I am working on the file "Super_textbook.ipynb".
The first task is:
Plot a scatter plot of predicted and actual value
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.scatter(By_train, hx_lin.predict(BX_train))
plt.ylabel("predicted value")
plt.xlabel("actual value")
plt.show()

### performance evaluation: training set

from sklearn.metrics import mean_squared_error
mean_squared_error(By_train, hx_lin.predict(BX_train))

### performance evaluation: test set
mean_squared_error(By_test, hx_lin.predict(BX_test))


I am working on the file "Super_textbook.ipynb".
The first task is:
Training set performance evaluation
Confusion matrix
https://en.wikipedia.org/wiki/Confusion_matrix
Can you write Python code for it?

from sklearn.metrics import confusion_matrix, f1_score

print confusion_matrix(y_train, hx_knn.predict(X_train))
print f1_score(y_train, hx_knn.predict(X_train))


I am working on the file "Super_textbook.ipynb".
The first task is:
Your turn: Task 1
Suppose there is a learning algorithm called Naive Bayes and the API is located in
http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB
Create a hx_nb, fit the data, and evaluate the training set and test set performance.
Regression example
Boston house price dataset
https://archive.ics.uci.edu/ml/datasets/Housing
Can you write Python code for it?

from sklearn.datasets import load_boston
bostondata = load_boston()
print bostondata.target
print bostondata.data.shape

### Learn more about the dataset

print bostondata.DESCR

# how the first row of data looks like

bostondata.data[1,]

BX_train, BX_test, By_train, By_test = train_test_split(bostondata.data, bostondata.target, test_size=0.3)


I am working on the file "Supermarket_Story.ipynb".
The first task is:
Does the shopper's original allowance, minus what's in the Supermarket's cash account, match what's currently in the shopper's wallet?
Can you write Python code for it?


I am working on the file "Supermarket_Story.ipynb".
The first task is:
Now we're ready for a simulation.  The simulator has to know what to do in case exceptions get raised.
Can you write Python code for it?

def simulation():
    test_data()  # initialize the_stuff.json
    kirby = Shopper("Kirby", 1000)
    market = SuperMarket()

    print("------- Buy Event 1 -------")

    try:
        market.buy(kirby, "Snicker-Snacks", 2)
        market.buy(kirby, "Polly's Peanuts", 2)
        market.buy(kirby, "Dr. Soap", 100)  # triggers exception
    except NoMoney:
        pass
        print("Uh oh, out of money!")
    except OutOfStock:
        pass
        print("Uh oh, out of stock!")

    print(kirby)
    print("Inventory on hand:", market.inventory)

    print("------- Buy Event 2 -------")
    # kirby didn't get away with buying that much Dr. Soap, but
    # the first two transactions got through.  Undetered, kirby
    # tries again to buy the soap.  We expect all the soap to be
    # gone, with 8 left of the other two items.  We had 10 of
    # everything to start.

    try:
        market.buy(kirby, "Dr. Soap", 10)  # should be OK this time
    except NoMoney:
        pass
        print("Uh oh, out of money!")
    except OutOfStock:
        pass
        print("Uh oh, out of stock!")

    print(kirby)
    print("Inventory on hand:", market.inventory)

    market.close()
    print(market)


I am working on the file "Supermarket_Story.ipynb".
The first task is:
The simulation takes a Shopper through two "buying sprees". We're not sure which will buy will abort, if any.  We could allow only one buy at a time.
The Dr. Soap transaction, asking for 1000, does not go through, but then the customer comes back and shops for 10 of that same item, and this should work.
Lets see...
Can you write Python code for it?

simulation()  # finally, lets DO something...


I am working on the file "Superoperators.ipynb".
The first task is:
These are also consistent with "by hand" notation.
So everything individually works as it would "by hand" but the operator_to_vector takes a transpose first... presumably to align the numpy arrays?
Ok, so knowing that, it's a bit hard to compare to by-hand results, but we can check a few. First, the left-multiplication should be the same as spre. Defined as $\mathcal{L}(A) = A\otimes I$:
Can you write Python code for it?

spre(destroy(2))

tensor(destroy(2),identity(2))

tensor(identity(2),destroy(2))


I am working on the file "Superoperators.ipynb".
The first task is:
This is where I got confused. By hand I write this as 1,2,3,4 in order from top to bottom, based on the notation:
$$\rho = \sum_{i,j}\rho_{ij}|i\rangle\langle j| \rightarrow |\rho\rangle\rangle = \sum_{i,j}\rho_{ij}|i\rangle \otimes |j\rangle$$
I'll let $$|1\rangle = \begin{pmatrix}1\0\end{pmatrix}$$ and $$|2\rangle = \begin{pmatrix}0\1\end{pmatrix}$$ so $i=j=1,2$ in the sums.
The outer products are:
$$|1\rangle\langle 1| = \begin{pmatrix}1&0\0&0\end{pmatrix}$$
$$|1\rangle\langle 2| = \begin{pmatrix}0&1\0&0\end{pmatrix}$$
$$|2\rangle\langle 1| = \begin{pmatrix}0&0\1&0\end{pmatrix}$$
$$|2\rangle\langle 2| = \begin{pmatrix}0&0\0&1\end{pmatrix}$$
And the tensor products are:
$$|1\rangle \otimes |1\rangle = \begin{pmatrix}1\0\0\0\end{pmatrix}$$
$$|1\rangle \otimes |2\rangle = \begin{pmatrix}0\1\0\0\end{pmatrix}$$
$$|2\rangle \otimes |1\rangle = \begin{pmatrix}0\0\1\0\end{pmatrix}$$
$$|2\rangle \otimes |2\rangle = \begin{pmatrix}0\0\0\1\end{pmatrix}$$
I see the code for operator_to_vector takes the transpose, but I'm not sure why that is? Maybe to align to the underlying Fortran-style column-major data in the underlying implementation?
So, the bottom line is: Either trust and use the QuTiP functions or roll your own. Mixing and matching builds you a world that will be ripe with transpose errors :-)
Can you write Python code for it?

one = Qobj([[1],[0]])
one

two = Qobj([[0],[1]])
two

rho == 1*one*one.dag() + 2*one*two.dag() + 3*two*one.dag() + 4*two*two.dag()


I am working on the file "Supervised and Unsupervised ML.ipynb".
The first task is:
Instead of generating a set by hand, let's create a randomize set of points on a plot colored differently based on different distributions. We'll use three different gaussian distributions to create the set.
Can you write Python code for it?

red   = [(round(random.gauss(27, 10)), round(random.gauss(31, 10))) for x in range(100)]
green = [(round(random.gauss(40, 8)), round(random.gauss(40, 9))) for x in range(100)]
blue  = [(round(random.gauss(50, 10)), round(random.gauss(60, 12))) for x in range(100)]

import matplotlib.pyplot as plt # Let's plot!

plt.rc('axes', color_cycle=['r', 'g', 'b'])
plt.axis([0, 100, 0, 100])
plt.plot([x for x,y  in red], [y for x,y in red], 'ro')
plt.plot([x for x,y in green], [y for x,y in green], 'go')
plt.plot([x for x,y in blue], [y for x,y in blue], 'bo')
plt.show()


I am working on the file "Supervised and Unsupervised ML.ipynb".
The first task is:
It's clear for the most part where the boundaries between these different point groups is when we look at a colored chart. Now let's see if we can get sklearn to figure out the same, just looking at the data with no help.
Can you write Python code for it?

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=3) # We know there are three clusters

kmeans.fit(red2+green2+blue2)

print("Red:\t{}".format(" ".join(str(kmeans.predict(pt)[0]) for pt in red2)))
print("Green:\t{}".format(" ".join(str(kmeans.predict(pt)[0]) for pt in green2)))
print("Blue:\t{}".format(" ".join(str(kmeans.predict(pt)[0]) for pt in blue2)))