I am working on the file "Facebook粉絲頁分析三部曲-爬取篇(comments).ipynb".
The first task is:
這篇是承襲上一篇 fb粉絲團分析and輸出報告-抓取篇(posts)
上一篇提到說,會爬取每一則粉絲頁po文,而本文的目標就是在把該post裡面的留言
都給爬取出來,所以會需要用到每一個post的id,也就是說需要上一篇的檔案才有辦法爬取
基本上由4個function來完成：
request_until_succeed
來確保完成爬取
getFacebookCommentFeedData
來產生comment的各種資料(message,like_count,created_time,comments,from...)
processFacebookComment
是處理getFacebookPageFeedData得到的各種資料,把它們結構化
scrapeFacebookPageFeedComments
主程式
Can you write Python code for it?

# 判斷response有無正常 正常 200，若無隔五秒鐘之後再試
def request_until_succeed(url):
    success = False
    while success is False:
        try:
            req = requests.get(url)
            if req.status_code == 200:
                success = True
            if req.status_code == 400:
                return None
        except Exception as e:
            print(e)
            time.sleep(5)
            print("Error for URL %s: %s" % (url, datetime.datetime.now()))
            print("Retrying.")
    return req


I am working on the file "Facebook粉絲頁分析三部曲-爬取篇(posts).ipynb".
The first task is:
爬取的基本概念是送request透過Facebook Graph API來取得資料
而request就是一個url，這個url會根據你的設定(你要拿的欄位)而回傳你需要的資料
但是在爬取大型粉絲頁時,很可能會因為你送的request太多了,就發生錯誤
這邊的解決方法很簡單用一個while迴圈,發生錯誤就休息5秒,5秒鐘後,再重新送request
基本上由5個function來完成：
request_until_succeed
來確保完成爬取
getFacebookPageFeedData
來產生post的各種資料(message,link,created_time,type,name,id...)
getReactionsForStatus
來獲得該post的各reaction數目(like, angry, sad ...)
processFacebookPageFeedStatus
是處理getFacebookPageFeedData得到的各種資料,把它們結構化
scrapeFacebookPageFeedStatus
為主程式
Can you write Python code for it?

# 判斷response有無正常 正常 200，若無隔五秒鐘之後再試
def request_until_succeed(url):
    success = False
    while success is False:
        try:
            req = requests.get(url)
            if req.status_code == 200:
                success = True
        except Exception as e:
            print(e)
            time.sleep(5)
            print("Error for URL %s: %s" % (url, datetime.datetime.now()))
            print("Retrying.")

    return req


I am working on the file "Facebook粉絲頁分析三部曲-爬取篇(posts).ipynb".
The first task is:
生成status_link ，此連結可以回到該臉書上的post
status_published = status_published + datetime.timedelta(hours=8) 根據所在時區 TW +8
Can you write Python code for it?

def processFacebookPageFeedStatus(status, access_token):

    # 要去確認抓到的資料是否為空
    status_id = status['id']
    status_type = status['type']
    if 'message' not in status.keys():
        status_message = ''
    else:
        status_message = status['message']
    if 'name' not in status.keys():
        link_name = ''
    else:
        link_name = status['name']
    link = status_id.split('_')

    # 此連結可以回到該臉書上的post
    status_link = 'https://www.facebook.com/'+link[0]+'/posts/'+link[1]

    status_published = datetime.datetime.strptime(status['created_time'],'%Y-%m-%dT%H:%M:%S+0000')
    # 根據所在時區 TW +8
    status_published = status_published + datetime.timedelta(hours=8)
    status_published = status_published.strftime('%Y-%m-%d %H:%M:%S')

    # 要去確認抓到的資料是否為空
    if 'reactions' not in status:
        num_reactions = 0
    else:
        num_reactions = status['reactions']['summary']['total_count']
    if 'comments' not in status:
        num_comments = 0
    else:
        num_comments = status['comments']['summary']['total_count']
    if 'shares' not in status:
        num_shares = 0
    else:
        num_shares = status['shares']['count']

    def get_num_total_reactions(reaction_type, reactions):
        if reaction_type not in reactions:
            return 0
        else:
            return reactions[reaction_type]['summary']['total_count']

    # 取得該篇文章的 reactions like,love,wow,haha,sad,angry數目
    reactions = getReactionsForStatus(status_id, access_token)

    num_loves = get_num_total_reactions('love', reactions)
    num_wows = get_num_total_reactions('wow', reactions)
    num_hahas = get_num_total_reactions('haha', reactions)
    num_sads = get_num_total_reactions('sad', reactions)
    num_angrys = get_num_total_reactions('angry', reactions)
    num_likes = get_num_total_reactions('like', reactions)

    # 回傳tuple形式的資料
    return (status_id, status_message, link_name, status_type, status_link,
            status_published, num_reactions, num_comments, num_shares,
           ...


I am working on the file "Facebook粉絲頁分析三部曲-爬取篇(posts).ipynb".
The first task is:
第一步 - 要先取得應用程式的帳號,密碼 (app_id, app_secret)
第二步 - 輸入要分析的粉絲團的 id (page_id)
[教學]如何申請建立 Facebook APP ID 應用程式ID
Can you write Python code for it?

# 分析的粉絲頁的id
page_id = "appledaily.tw"

app_id = ""
app_secret = ""

access_token = app_id + "|" + app_secret


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
一共5234筆發文
Check missing
看看個欄位缺失的值
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
代表有5061篇發文和27518個詞
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
剩下5061篇發文
一樣處理時間,新增星期幾和單日第幾時po文的欄位
Can you write Python code for it?

df['datetime'] = df['status_published'].apply(lambda x: datetime.datetime.strptime(x,'%Y-%m-%d %H:%M:%S'))
df['weekday'] = df['datetime'].apply(lambda x: x.weekday_name)
df['hour'] = df['datetime'].apply(lambda x:  x.hour)


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
把status_message為空值的地方拿掉
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
把詞向量的feature和其他feature結合在一起
Can you write Python code for it?

X = np.concatenate((features, word_vector), axis=1)

X


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
接下剌我們要使用的model是logistic regression
logistic regression : 是機器學習分類算法的其中一,其算法作為一個二分類算法，主要解決的是線性可分的問題
方程式 :
$$
h(x)=\frac{1}{1+exp(-w^Tx)}
$$
假設我們可以知道一個病人的性別,年齡,身高,體重,血壓...為了要計算他得到癌症的機率,可以將這幾個特徵做加權求和
x代表每個特徵,w代表每個特徵相對應的權重,s為加權求和的值
$$
\color{purple}{s}=\sum_{i={0}}^d{w_i}{x_i}=w^Tx
$$
但是算出來的s範圍是負無限大到正無限大,我們會希望得到一個機率估計的值0到1,所以要把s經過sigmoid函數${\theta}$處理
$$
{\theta}({s})=\frac{e^{s}}{1+e^{s}}=\frac{1}{1+e^{-s}}
$$

綜合以上由於我們只要可以知道x和w,就可以進行預測了!
x是我們已經知道的,但是要如何得到w呢?
我們先需要也要有一個可以衡量這組w好不好的函數稱為cost function
這邊所使用的誤差函數為 cross entropy error (直接給方程式,就不附推導過程)
y為train data中的label,1就是得癌症,0就是沒得
$$
err({w},x,y)=ln(1+exp(-y{w}x))
$$
所以cost function為
$$
E({w})=\frac{1}{N}\sum_{n=1}^{N}ln(1+exp(-y_n{w}^Tx_n))
$$
有了cost function就可以用梯度下降法(Gradient Descent)去尋找適合的w,找到w後就可以進行預測了!
predict
套入scikit的LogisticRegression,並用cross validation去算平均準確率
cross validation :

假設cv=5 就是把整組資料分成五份,其中一份當作testing data,剩下四份當作training data,然後輪流做五次
讓每一份都會輪到當成testing data,最後把五次的結果平均,算出平均準確率
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

clf = LogisticRegression()
clf.fit(X, y)

score = cross_val_score(clf, X, y, cv=5)
print (score.mean())


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
看一看前五筆
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
處理 feature (status_message)
接下來要建立詞向量,什麼是詞向量
舉例 總共有三篇文章
第一篇 我想吃蘋果 提取關鍵字後 我,吃,蘋果
第二篇 我想吃香蕉 提取關鍵字後 我,吃,香蕉
第三篇 我想吃火鍋 提取關鍵字後 我,吃,火鍋
詞向量就是
我 | 吃 | 蘋果 | 香蕉 | 火鍋
--- | --- | --- | --- | ---
1 | 1 | 1 | 0 | 0
1 | 1 | 0 | 1 | 0
1 | 1 | 0 | 0 | 1
所以接下來就建立發文內容的詞向量
Can you write Python code for it?

# 載入繁體詞庫
import jieba.analyse

# 安裝jieba套件的時候,就有繁體詞庫
jieba.set_dictionary('/home/wy/anaconda3/envs/python3/lib/python3.6/site-packages/jieba/extra_dict/dict.txt.big')

def jieba_extract(message_list):
    word_count = {}
    for message in message_list:
        seg_list = jieba.analyse.extract_tags(message, topK=120)

        for seg in seg_list:
            if not seg in word_count:
                word_count[seg] = 1
            else:
                word_count[seg] += 1

    sorted_word_count = sorted(word_count.items(), key=operator.itemgetter(1))
    sorted_word_count.reverse()
    return sorted_word_count

sorted_word_count = jieba_extract(list(df['status_message']))
word = [one[0] for one in sorted_word_count]

def get_word_vector(message):
    v = np.zeros(len(word))
    v_word = jieba.analyse.extract_tags(message, topK=120)
    for w in v_word:
        v[words.index(w)] = 1
    return v

sorted_word_count = jieba_extract(list(df['status_message']))
words = [one[0] for one in sorted_word_count]

word_vector = [get_word_vector(one) for one in df['status_message']]
word_vector = np.array(word_vector)

word_vector.shape
word_vector


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
處理 label
把reaction大於中位數(1243)label成1,其餘label成0
Can you write Python code for it?

y = df['num_reactions']

y.describe()

y = df['num_reactions'].apply(lambda x: 1 if x > 1243 else 0)


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
處理後的feature 維度和數量
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-預測篇.ipynb".
The first task is:
處理後的label 數量
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
...but does remarkably well on the adjacent facies predictions.
Can you write Python code for it?

display_adj_cm(cv_conf, facies_labels, adjacent_facies,
               display_metrics=True, hide_zeros=True)

def compare_facies_plot(logs, compadre, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster1 = np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)
    cluster2 = np.repeat(np.expand_dims(logs[compadre].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=7, figsize=(9, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im1 = ax[5].imshow(cluster1, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)
    im2 = ax[6].imshow(cluster2, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[6])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im2, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-2):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
   ...


I am working on the file "Facies_classification.ipynb".
The first task is:
Applying the classification model to new data
Now that we have a trained facies classification model we can use it to identify facies in wells that do not have core data.  In this case, we will apply the classifier to two wells, but we could use it on any number of wells for which we have the same set of well logs for input.
This dataset is similar to the training data except it does not have facies labels.  It is loaded into a dataframe called test_data.
Can you write Python code for it?

well_data = pd.read_csv('validation_data_nofacies.csv')
well_data['Well Name'] = well_data['Well Name'].astype('category')
well_features = well_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)


I am working on the file "Facies_classification.ipynb".
The first task is:
As noted above, the boundaries between the facies classes are not all sharp, and some of them blend into one another.  The error within these 'adjacent facies' can also be calculated.  We define an array to represent the facies adjacent to each other.  For facies label i, adjacent_facies[i] is an array of the adjacent facies labels.
Can you write Python code for it?

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

print('Facies classification accuracy = %f' % accuracy(conf))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf, adjacent_facies))


I am working on the file "Facies_classification.ipynb".
The first task is:
Conditioning the data set
Now we extract just the feature variables we need to perform the classification.  The predictor variables are the five wireline values and two geologic constraining variables. We also get a vector of the facies labels that correspond to each feature vector.
Can you write Python code for it?

correct_facies_labels = training_data['Facies'].values

feature_vectors = training_data.drop(['Formation', 'Well Name', 'Depth','Facies','FaciesLabels'], axis=1)
feature_vectors.describe()


I am working on the file "Facies_classification.ipynb".
The first task is:
Considering adjacent facies, the F1 scores for all facies types are above 0.9, except when classifying SiSh or marine siltstone and shale.  The classifier often misclassifies this facies (recall of 0.66), most often as wackestone.
These results are comparable to those reported in Dubois et al. (2007).
Applying the classification model to the blind data
We held a well back from the training, and stored it in a dataframe called blind:
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
Finally we can write out a csv file with the well data along with the facies classification results.
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
Finally we predict facies labels for the unknown data, and store the results in a Facies column of the test_data dataframe.
Can you write Python code for it?

#predict facies of unclassified data
y_unknown = clf.predict(X_unknown)
well_data['Facies'] = y_unknown
well_data

well_data['Well Name'].unique()


I am working on the file "Facies_classification.ipynb".
The first task is:
In addition to individual wells, we can look at how the various facies are represented by the entire training set.  Let's plot a histogram of the number of training examples for each facies class.
Can you write Python code for it?

#count the number of unique entries for each facies, sort them by
#facies number (instead of by number of entries)
facies_counts = training_data['Facies'].value_counts().sort_index()
#use facies labels to index each count
facies_counts.index = facies_labels

facies_counts.plot(kind='bar',color=facies_colors,
                   title='Distribution of Training Data by Facies')
facies_counts


I am working on the file "Facies_classification.ipynb".
The first task is:
Let's see how we did with the confusion matrix:
Can you write Python code for it?

cv_conf = confusion_matrix(y_blind, y_pred)

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf, adjacent_facies))


I am working on the file "Facies_classification.ipynb".
The first task is:
Let's take a look at the data from individual wells in a more familiar log plot form.  We will create plots for the five well log variables, as well as a log for facies labels.  The plots are based on the those described in Alessandro Amato del Monte's excellent tutorial.
Can you write Python code for it?

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')

    ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([])
    ax[4].set_yticklabels([]); ax[5].set_yticklabels([])
    ax[5].set_xticklabels([])
    f.suptitle('Well: %s'%logs.iloc[0]['Well Name'], fontsize=14,y=0.94)


I am working on the file "Facies_classification.ipynb".
The first task is:
Model parameter selection
The classifier so far has been built with the default parameters.  However, we may be able to get improved classification results with optimal parameter choices.
We will consider two parameters.  The parameter C is a regularization factor, and tells the classifier how much we want to avoid misclassifying training examples.  A large value of C will try to correctly classify more examples from the training set, but if C is too large  it may 'overfit' the data and fail to generalize when classifying new data. If C is too small then the model will not be good at fitting outliers and will have a large error on the training set.
The SVM learning algorithm uses a kernel function to compute the distance between feature vectors.  Many kernel functions exist, but in this case we are using the radial basis function rbf kernel (the default).  The gamma parameter describes the size of the radial basis functions, which is how far away two vectors in the feature space need to be to be considered close.
We will train a series of classifiers with different values for C and gamma.  Two nested loops are used to train a classifier for every possible combination of values in the ranges specified.  The classification accuracy is recorded for each combination of parameter values.  The results are shown in a series of plots, so the parameter values that give the best classification accuracy on the test set can be selected.
This process is also known as 'cross validation'.  Often a separate 'cross validation' dataset will be created in addition to the training and test sets to do model selection.  For this tutorial we will just use the test set to choose model parameters.
Can you write Python code for it?

#model selection takes a few minutes, change this variable
#to true to run the parameter loop
do_model_selection = True

if do_model_selection:
    C_range = np.array([.01, 1, 5, 10, 20, 50, 100, 1000, 5000, 10000])
    gamma_range = np.array([0.0001, 0.001, 0.01, 0.1, 1, 10])

    fig, axes = plt.subplots(3, 2,
                        sharex='col', sharey='row',figsize=(10,10))
    plot_number = 0
    for outer_ind, gamma_value in enumerate(gamma_range):
        row = int(plot_number / 2)
        column = int(plot_number % 2)
        cv_errors = np.zeros(C_range.shape)
        train_errors = np.zeros(C_range.shape)
        for index, c_value in enumerate(C_range):

            clf = svm.SVC(C=c_value, gamma=gamma_value)
            clf.fit(X_train,y_train)

            train_conf = confusion_matrix(y_train, clf.predict(X_train))
            cv_conf = confusion_matrix(y_test, clf.predict(X_test))

            cv_errors[index] = accuracy(cv_conf)
            train_errors[index] = accuracy(train_conf)

        ax = axes[row, column]
        ax.set_title('Gamma = %g'%gamma_value)
        ax.semilogx(C_range, cv_errors, label='CV error')
        ax.semilogx(C_range, train_errors, label='Train error')
        plot_number += 1
        ax.set_ylim([0.2,1])

    ax.legend(bbox_to_anchor=(1.05, 0), loc='lower left', borderaxespad=0.)
    fig.text(0.5, 0.03, 'C value', ha='center',
             fontsize=14)

    fig.text(0.04, 0.5, 'Classification Accuracy', va='center',
             rotation='vertical', fontsize=14)


I am working on the file "Facies_classification.ipynb".
The first task is:
Now it's a simple matter of making a prediction and storing it back in the dataframe:
Can you write Python code for it?

y_pred = clf.predict(X_blind)
blind['Prediction'] = y_pred


I am working on the file "Facies_classification.ipynb".
The first task is:
Now that the model has been trained on our data, we can use it to predict the facies of the feature vectors in the test set.  Because we know the true facies labels of the vectors in the test set, we can use the results to evaluate the accuracy of the classifier.
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
Now we can train the classifier using the training set we created above.
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
Now we can transform this with the scaler we made before:
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
Placing the log plotting code in a function will make it easy to plot the logs from multiples wells, and can be reused later to view the results when we apply the facies classification model to other wells.  The function was written to take a list of colors and facies labels as parameters.
We then show log plots for wells SHRIMPLIN.
Can you write Python code for it?

make_facies_log_plot(
    training_data[training_data['Well Name'] == 'SHRIMPLIN'],
    facies_colors)


I am working on the file "Facies_classification.ipynb".
The first task is:
Precision and recall are metrics that give more insight into how the classifier performs for individual facies.  Precision is the probability that given a classification result for a sample, the sample actually belongs to that class.  Recall is the probability that a sample will be correctly classified for a given class.
Precision and recall can be computed easily using the confusion matrix.  The code to do so has been added to the display_confusion_matrix() function:
Can you write Python code for it?

display_cm(cv_conf, facies_labels,
           display_metrics=True, hide_zeros=True)


I am working on the file "Facies_classification.ipynb".
The first task is:
Scikit also includes a handy function to randomly split the training data into training and test sets.  The test set contains a small subset of feature vectors that are not used to train the network.  Because we know the true facies labels for these examples, we can compare the results of the classifier to the actual facies and determine the accuracy of the model.  Let's use 20% of the data for the test set.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
        scaled_features, correct_facies_labels, test_size=0.1, random_state=42)


I am working on the file "Facies_classification.ipynb".
The first task is:
Scikit includes a preprocessing module that can 'standardize' the data (giving each variable zero mean and unit variance, also called whitening).  Many machine learning algorithms assume features will be standard normally distributed data (ie: Gaussian with zero mean and unit variance).  The factors used to standardize the training set must be applied to any subsequent feature set that will be input to the classifier.  The StandardScalar class can be fit to the training set, and later used to standardize any training data.
Can you write Python code for it?

from sklearn import preprocessing

scaler = preprocessing.StandardScaler().fit(feature_vectors)
scaled_features = scaler.transform(feature_vectors)

feature_vectors


I am working on the file "Facies_classification.ipynb".
The first task is:
The best accuracy on the cross validation error curve was achieved for gamma = 1, and C = 10.  We can now create and train an optimized classifier based on these parameters:
Can you write Python code for it?

clf = svm.SVC(C=10, gamma=1)
clf.fit(X_train, y_train)

cv_conf = confusion_matrix(y_test, clf.predict(X_test))

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf, adjacent_facies))


I am working on the file "Facies_classification.ipynb".
The first task is:
The data needs to be scaled using the same constants we used for the training data.
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
The label vector is just the Facies column:
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
The rows of the confusion matrix correspond to the actual facies labels.  The columns correspond to the labels assigned by the classifier.  For example, consider the first row. For the feature vectors in the test set that actually have label SS, 23 were correctly indentified as SS, 21 were classified as CSiS and 2 were classified as FSiS.
The entries along the diagonal are the facies that have been correctly classified.  Below we define two functions that will give an overall value for how the algorithm is performing.  The accuracy is defined as the number of correct classifications divided by the total number of classifications.
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc


I am working on the file "Facies_classification.ipynb".
The first task is:
This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.
The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
1. Nonmarine sandstone
2. Nonmarine coarse siltstone
3. Nonmarine fine siltstone
4. Marine siltstone and shale
5. Mudstone (limestone)
6. Wackestone (limestone)
7. Dolomite
8. Packstone-grainstone (limestone)
9. Phylloid-algal bafflestone (limestone)
These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close.  Mislabeling within these neighboring facies can be expected to occur.  The following table lists the facies, their abbreviated labels and their approximate neighbors.
Facies |Label| Adjacent Facies
:---: | :---: |:--:
1 |SS| 2
2 |CSiS| 1,3
3 |FSiS| 2
4 |SiSh| 5
5 |MS| 4,6
6 |WS| 5,7
7 |D| 6,8
8 |PS| 6,7,9
9 |BS| 7,8
Let's clean up this dataset.  The 'Well Name' and 'Formation' columns can be turned into a categorical data type.
Can you write Python code for it?

training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data['Well Name'].unique()

training_data.describe()


I am working on the file "Facies_classification.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables.  Looking at the count values, there are 3232 feature vectors in the training set.
Remove a single well to use as a blind test later.
Can you write Python code for it?

blind = training_data[training_data['Well Name'] == 'SHANKLE']
training_data = training_data[training_data['Well Name'] != 'SHANKLE']


I am working on the file "Facies_classification.ipynb".
The first task is:
This shows the distribution of examples by facies for the examples in the training set.  Dolomite (facies 7) has the fewest with 81 examples.  Depending on the performance of the classifier we are going to train, we may consider getting more examples of these facies.
Crossplots are a familiar tool in the geosciences to visualize how two properties vary with rock type.  This dataset contains 5 log variables, and scatter matrix can help to quickly visualize the variation between the all the variables in the dataset.  We can employ the very useful Seaborn library to quickly create a nice looking scatter matrix. Each pane in the plot shows the relationship between two of the variables on the x and y axis, with each point is colored according to its facies.  The same colormap is used to represent the 9 facies.
Can you write Python code for it?

#save plot display settings to change back to when done plotting with seaborn
inline_rc = dict(mpl.rcParams)

import seaborn as sns
sns.set()
sns.pairplot(training_data.drop(['Well Name','Facies','Formation','Depth','NM_M','RELPOS'],axis=1),
             hue='FaciesLabels', palette=facies_color_map,
             hue_order=list(reversed(facies_labels)))

#switch back to default matplotlib plot style
mpl.rcParams.update(inline_rc)


I am working on the file "Facies_classification.ipynb".
The first task is:
Training the SVM classifier
Now we use the cleaned and conditioned training set to create a facies classifier.  As mentioned above, we will use a type of machine learning model known as a support vector machine.  The SVM is a map of the feature vectors as points in a multi dimensional space, mapped so that examples from different facies are divided by a clear gap that is as wide as possible.
The SVM implementation in scikit-learn takes a number of important parameters.  First we create a classifier using the default settings.
Can you write Python code for it?

from sklearn import svm

clf = svm.SVC()


I am working on the file "Facies_classification.ipynb".
The first task is:
We can form the feature matrix by dropping some of the columns and making a new dataframe:
Can you write Python code for it?

well_features = blind.drop(['Facies', 'Formation', 'Well Name', 'Depth'], axis=1)


I am working on the file "Facies_classification.ipynb".
The first task is:
We can use the well log plot to view the classification results along with the well logs.
Can you write Python code for it?

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'STUART'],
    facies_colors=facies_colors)

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'CRAWFORD'],
    facies_colors=facies_colors)


I am working on the file "Facies_classification.ipynb".
The first task is:
We managed 0.71 using the test data, but it was from the same wells as the training data. This more reasonable test does not perform as well...
Can you write Python code for it?


I am working on the file "Facies_classification.ipynb".
The first task is:
We need some metrics to evaluate how good our classifier is doing.  A confusion matrix is a table that can be used to describe the performance of a classification model.  Scikit-learn allows us to easily create a confusion matrix by supplying the actual and predicted facies labels.
The confusion matrix is simply a 2D array.  The entries of confusion matrix C[i][j] are equal to the number of observations predicted to have facies j, but are known to have facies i.
To simplify reading the confusion matrix, a function has been written to display the matrix along with facies labels and various error metrics.  See the file classification_utilities.py in this repo for the display_cm() function.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
from classification_utilities import display_cm, display_adj_cm

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, hide_zeros=True)


I am working on the file "Fairness/error-fairness.ipynb".
The first task is:
Studying the trade off
Focusing on the plane traced out by A, C, B ,D, we get a two dimensional plot which provides insight into the trade off between
1. false positives and false negatives
2. Favouritism, how much the majority group benefits
Can you write Python code for it?

fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)
ax.plot([0,0,1,1], [0,1,0,1], 'bo')
ax.set_xlabel('FN -- FP')
ax.set_ylabel('favouritism')
ax.text(0, 0, naming(1, 0, 0)[3], size=30)
ax.text(0, 1, naming(1, 0, 1)[3], size=30)
ax.text(1, 0, naming(0, 1, 0)[3], size=30)
ax.text(1, 1, naming(0, 1, 1)[3], size=30)


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
After the 2 byte header, the next 8 bytes of the packet can be used to identify its source or type.
Can you write Python code for it?

source_ids = [p[0][2:10].hex().upper() for p in packets]
collections.Counter(source_ids)


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
Only around 28% of the transmitted data is the transport stream video.
Can you write Python code for it?

len(video_ts)/sum([len(p[0]) for p in packets])

with open('/tmp/falcon9.ts', 'wb') as f:
    f.write(video_ts)

ts = np.frombuffer(video_ts, dtype = 'uint8').reshape((-1,188))

# sync byte 71 = 0x47
np.unique(ts[:,0])

# TEI = 0
np.unique(ts[:,1] >> 7)

pusi = (ts[:,1] >> 6) & 1
# priority = 0
np.unique((ts[:,1] >> 5) & 1)

pid = ts[:,1:3].ravel().view('uint16').byteswap() & 0x1fff
np.unique(pid)

for p in np.unique(pid):
    print(f'PID {p} ratio {np.average(pid == p) * 100:.1f}%')

# TSC = 0
np.unique(ts[:,3] >> 6)

adaptation = (ts[:,3] >> 4) & 0x3
np.unique(adaptation)

continuity = ts[:,3] & 0xf

for p in np.unique(pid):
    print('PID', p, 'PUSI values', np.unique(pusi[pid == p]),
          'adaptation field values', np.unique(adaptation[pid == p]))

pcr_pid = ts[pid == 511]
pcr = np.concatenate((np.zeros((pcr_pid.shape[0], 2), dtype = 'uint8'), pcr_pid[:,6:12]), axis = 1)
pcr = pcr.view('uint64').byteswap().ravel()
pcr_base = pcr >> 15
pcr_extension = pcr & 0x1ff
pcr_value = (pcr_base * 300 + pcr_extension) / 27e6

video_timestamps = timestamps[[s == video_source for s in source_ids]]
ts_timestamps = np.repeat(video_timestamps, 5)
pcr_pid_timestamps = ts_timestamps[pid == 511]

plt.plot(pcr_pid_timestamps, pcr_value, '.')
plt.title('Falcon-9 PCR timestamps')
plt.ylabel('PID 511 PCR (s)')
plt.xlabel('Packet timestamp (GPS time)');


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
Only ~76% of the frames payload contains packets. The rest is padding.
Can you write Python code for it?

sum([len(p[0]) for p in packets])/x[:,4:].size


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
Some packets have 64-bit timestamps starting 3 bytes after the packet source ID. These give nanoseconds since the GPS epoch.
Can you write Python code for it?

timestamps = np.datetime64('1980-01-06') + \
    np.array([np.frombuffer(p[0][13:][:8], dtype = 'uint64').byteswap()[0] for p in packets]) \
        * np.timedelta64(1, 'ns')
timestamps_valid = (timestamps >= np.datetime64('2021-01-01')) & (timestamps <= np.datetime64('2022-01-01'))

plt.plot(timestamps[timestamps_valid],
         np.array([p[1] for p in packets])[timestamps_valid], '.')
plt.title('Falcon-9 packet timestamps')
plt.xlabel('Timestamp (GPS time)')
plt.ylabel('Frame counter');


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
The first byte of all the frames is 0xe0. Here we see that one of the frames has an error in this byte.
Can you write Python code for it?


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
The next three bytes form a header composed of a 13 bit frame counter and an 11 bit field that indicates where the first packet inside the payload starts (akin to a first header pointer in CCSDS protocols).
Can you write Python code for it?

header = np.unpackbits(x[:,1:4], axis = 1)
counter = header[:,:13]
counter = np.concatenate((np.zeros((x.shape[0], 3), dtype = 'uint8'), counter), axis = 1)
counter = np.packbits(counter, axis = 1)
counter = counter.ravel().view('uint16').byteswap()
start_offset = header[:,-11:]
start_offset = np.concatenate((np.zeros((x.shape[0], 5), dtype = 'uint8'), start_offset), axis = 1)
start_offset = np.packbits(start_offset, axis = 1)
start_offset = start_offset.ravel().view('uint16').byteswap()

plt.plot(counter, '.')
plt.title('Falcon-9 frame counter')
plt.ylabel('13-bit frame counter')
plt.xlabel('Decoded frame');


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
This notebook shows an analysis of the Falcon-9 upper stage S-band telemetry frames. It is based on r00t.cz's analysis.
The frames are CCSDS Reed-Solomon frames with an interleaving depth of 5, a (255,239) code, and an (uncoded) frame size of 1195 bytes.
Can you write Python code for it?


I am working on the file "Falcon-9/Falcon-9 frames.ipynb".
The first task is:
Valid packets contain a 2 byte header where the 4 MSBs are set to 1 and the remaining 12 bits indicate the size of the packet payload in bytes (so the total packet size is this value plus 2). Using this header, the packets can be defragmented in the same way as CCSDS Space Packets transmitted using the M_PDU protocol.
Can you write Python code for it?

def packet_len(packet):
    packet = np.frombuffer(packet[:2], dtype = 'uint8')
    return (packet.view('uint16').byteswap()[0] & 0xfff) + 2

def valid_packet(packet):
    return packet[0] >> 4 == 0xf

def defrag(x, counter, start_offset):
    packet = bytearray()
    frame_count = None

    for frame, count, first in zip(x, counter, start_offset):
        frame = frame[4:]
        if frame_count is not None \
          and count != ((frame_count + 1) % 2**13):
            # broken stream
            packet = bytearray()
        frame_count = count

        if first == 0x7fe:
            # only idle
            continue
        elif first == 0x7ff:
            # no packet starts
            if packet:
                packet.extend(frame)
            continue

        if packet:
            packet.extend(frame[:first])
            packet = bytes(packet)
            yield packet, frame_count

        while True:
            packet = bytearray(frame[first:][:2])
            if len(packet) < 2:
                # not full header inside frame
                break
            first += 2
            if not valid_packet(packet):
                # padding found
                packet = bytearray()
                break
            length = packet_len(packet) - 2
            packet.extend(frame[first:][:length])
            first += length
            if first > len(frame):
                # packet does not end in this frame
                break
            packet = bytes(packet)
            yield packet, frame_count
            packet = bytearray()
            if first == len(frame):
                # packet just ends in this frame
                break

packets = list(defrag(x, counter, start_offset))


I am working on the file "Fall2017/.ipynb_checkpoints/DataLoading-checkpoint.ipynb".
The first task is:
Every dataloader in PyTorch needs this form. The class you write for a DataLoader contains two methods __init__ and __getitem__.
The __init__ method is where you define arguments telling the location of your dataset, transforms (if any). It is also a good idea to have a variable which can hold a list of all images/batches in your dataset. There are quite a number of things going on in the function we wrote. Let's break it down:

We create two lists train_data and train_labels to contain the data and their associated labels. This is helpful since we have 5 training batches of 10,000 images each and it will be time consuming to do the same operation over and over again for all 5 batches.

We read in all batches at once in our for loop. This is a better approach almost every time and should be applied to any dataset that you're trying to read.

We use a funcion called np.concatenate to "concatenate" the training data which is in the form of numpy arrays. The labels are in the form of lists and hence we simply use the python concat operator + to concatenate them.

The __getitem__ method should accept only one argument: the index of the image/batch you want to access. We get to enjoy our hardwork as we can simply load a data and label by one index and return it. When writing your custom dataloaders it is a good practice to keep your __getitem__ as simple as possible.
Having written a dataloader, we can now test it. For the time being, don't worry about the test code, we'll get to it in later talks. The code below can be used for testing the dataloader class above:
Can you write Python code for it?

import torchvision.transforms as transforms
import matplotlib.pyplot as plt

tfs  = transforms.Compose([transforms.ToTensor(),
                          transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])# convert any data into a torch tensor

root='/home/akulshr/cifar-10-batches-py/'

cifar_train = CIFARLoader(root, transform=tfs) # create a "CIFARLoader instance".
cifar_loader = data.DataLoader(cifar_train, batch_size=4, shuffle=True, num_workers=2)

data_iter = iter(cifar_loader)
data,label = data_iter.next()


I am working on the file "Fall2017/.ipynb_checkpoints/DataLoading-checkpoint.ipynb".
The first task is:
Let's test if we actually can load the data after the unpickling. For that we're going to load one the data batches and see if we can find the data and labels from it:
Can you write Python code for it?

Dic = unpickle(os.path.join('/home/akulshr','cifar-10-batches-py', 'data_batch_1'))
print(Dic[b'data'].shape)
print(len(Dic[b'labels']))


I am working on the file "Fall2017/.ipynb_checkpoints/DataLoading-checkpoint.ipynb".
The first task is:
This little helper function will call our "unpickling" function to unpickle the data and return the unpicked data back to us.In datasets where there is only images, you can use PIL.Image. Armed with this let's write a dataloader for PyTorch.
DataLoading in PyTorch
Dataloading in PyTorch is a two step process. First, you need define a custom class which is subclassed from a data.Dataset class in torch.utils.data. This class takes in arguments which tell PyTorch about the location of the dataset, any "transforms" that you need to make before the dataset is loading.
First lets import the relevant modules. The data module which contains the useful functions for dataloading is contained in torch.utils.data. We also need a utility called glob to get all our batches in a list. The dataset is split into 5 different parts and we need to have them all together so we can use the entire training set:
Can you write Python code for it?

import torch
import torch.utils.data as data
import glob
from PIL import Image
import numpy as np

class CIFARLoader(data.Dataset):
    """
    CIFAR-10 Loader: Loads the CIFAR-10 data according to an index value
    and returns the data and the labels.

    args:
    root: Root of the data directory.

    Optional args:
    transforms: The transforms you wish to apply to the data.
    target_transforms: The transforms you wish to apply to the labels.

    """
    def __init__(self, root,transform=None, target_transform=None):
        self.root = root
        self.transform = transform
        self.target_transform = target_transform
        patt = os.path.join(self.root, 'data_batch_*') # create the pattern we want to search for.
        self.batches = sorted(glob.glob(patt))
        self.train_data = []
        self.train_labels = []

        for batch in self.batches:
            entry = {}
            entry = load_data(batch)
            self.train_data.append(entry[b'data'])
            self.train_labels += entry[b'labels']
        #############################################
        # We need to "concatenate" all the different #
        # training samples into one big array. For   #
        # doing that we're going to use a numpy      #
        # function called "concatenate".             #
        ##############################################
        self.train_data = np.concatenate(self.train_data)
        self.train_data = self.train_data.reshape((50000, 3, 32,32))
        self.train_data = self.train_data.transpose((0,2,3,1)) # pay attention to this step!

    def __getitem__(self, index):
        image = self.train_data[index]
        label = self.train_labels[index]

        if self.transform is not None:
            image = self.transform(image)
        if self.target_transform is not None:
            label = self.target_transform(label)
        print(image.size())
        return image, label

    def __len__(self):
   ...


I am working on the file "Fall2017/.ipynb_checkpoints/DataLoading-checkpoint.ipynb".
The first task is:
We notice that the data is a numpy array of 1000x3072 which means that the data batch contains 10000 images of size 3072 (32x32x3). The labels are 1000x1 list in which the ith element corresponds to the correct label for the ith image. This is how data is in the other training batches. Whenever you've got a new dataset always try to load one or two images(or a data batch) like in this case to get a feel of how the data is.
Loading data like this can be tedious and time consuming. Let's write a helper function which will return the data and the labels:
Can you write Python code for it?

def load_data(batch):
    print ("Loading batch:{}".format(batch))
    return unpickle(batch)


I am working on the file "Fall2017/DataLoading.ipynb".
The first task is:
Every dataloader in PyTorch needs this form. The class you write for a DataLoader contains two methods __init__ and __getitem__.
The __init__ method is where you define arguments telling the location of your dataset, transforms (if any). It is also a good idea to have a variable which can hold a list of all images/batches in your dataset. There are quite a number of things going on in the function we wrote. Let's break it down:

We create two lists train_data and train_labels to contain the data and their associated labels. This is helpful since we have 5 training batches of 10,000 images each and it will be time consuming to do the same operation over and over again for all 5 batches.

We read in all batches at once in our for loop. This is a better approach almost every time and should be applied to any dataset that you're trying to read.

We use a funcion called np.concatenate to "concatenate" the training data which is in the form of numpy arrays. The labels are in the form of lists and hence we simply use the python concat operator + to concatenate them.

The __getitem__ method should accept only one argument: the index of the image/batch you want to access. We get to enjoy our hardwork as we can simply load a data and label by one index and return it. When writing your custom dataloaders it is a good practice to keep your __getitem__ as simple as possible.
Having written a dataloader, we can now test it. For the time being, don't worry about the test code, we'll get to it in later talks. The code below can be used for testing the dataloader class above:
Can you write Python code for it?


I am working on the file "Fall2017/DataLoading.ipynb".
The first task is:
Let's test if we actually can load the data after the unpickling. For that we're going to load one the data batches and see if we can find the data and labels from it:
Can you write Python code for it?


I am working on the file "Fall2017/DataLoading.ipynb".
The first task is:
This little helper function will call our "unpickling" function to unpickle the data and return the unpicked data back to us.In datasets where there is only images, you can use PIL.Image. Armed with this let's write a dataloader for PyTorch.
DataLoading in PyTorch
Dataloading in PyTorch is a two step process. First, you need define a custom class which is subclassed from a data.Dataset class in torch.utils.data. This class takes in arguments which tell PyTorch about the location of the dataset, any "transforms" that you need to make before the dataset is loading.
First lets import the relevant modules. The data module which contains the useful functions for dataloading is contained in torch.utils.data. We also need a utility called glob to get all our batches in a list. The dataset is split into 5 different parts and we need to have them all together so we can use the entire training set:
Can you write Python code for it?

import torch
import torch.utils.data as data
import glob
from PIL import Image
import numpy as np

-


I am working on the file "Fall2017/DataLoading.ipynb".
The first task is:
We notice that the data is a numpy array of 1000x3072 which means that the data batch contains 10000 images of size 3072 (32x32x3). The labels are 1000x1 list in which the ith element corresponds to the correct label for the ith image. This is how data is in the other training batches. Whenever you've got a new dataset always try to load one or two images(or a data batch) like in this case to get a feel of how the data is.
Loading data like this can be tedious and time consuming. Let's write a helper function which will return the data and the labels:
Can you write Python code for it?


I am working on the file "Fall2017/Module1.ipynb".
The first task is:
Given the dataset, let's test our dataset by seeing some of the images and their corresponding labels. PyTorch provides us with a neat little function called make_grid which plots "x" number of images together in a grid.
Can you write Python code for it?

import torchvision.transforms as transforms
import torch.utils.data as data
import numpy as np
import matplotlib.pyplot as plt
import torchvision

def imshow(torch_tensor):
    torch_tensor = torch_tensor/2 + 0.5
    npimg = torch_tensor.numpy()
    plt.imshow(npimg.transpose(1,2,0))
    plt.show()

tfs  = transforms.Compose([transforms.ToTensor(),
                          transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])

root='/home/akulshr/cifar-10-batches-py/'

cifar_train = CIFARLoader(root, train=True, transform=tfs) # create a "CIFARLoader instance".
cifar_loader = data.DataLoader(cifar_train, batch_size=4, shuffle=True, num_workers=2)

# all possible classes in the CIFAR-10 dataset
classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

data_iter = iter(cifar_loader)
data,label = data_iter.next()

#visualize data.
imshow(torchvision.utils.make_grid(data))

# print the labels
' '.join(classes[label[j]] for j in range(4))


I am working on the file "Fall2017/Module1.ipynb".
The first task is:
So we've got these images along with their labels as "ground truth". Now let's ask the neural network we just trained as to what it thinks the images are
Can you write Python code for it?

data_iter = iter(cifar_test_loader)
imgs,label = data_iter.next()
op = net(Variable(imgs))
_, pred = torch.max(op.data, 1)
print("Guessed class: ")
print(' '.join(classes[pred[j]] for j in range(4)))


I am working on the file "Fall2017/Module1_gpu.ipynb".
The first task is:
If we have a gpu then we need to "move" the network to the GPU. Under the hood, it copies the weights and the biases that are in CPU memory over to the GPU memory. In PyTorch, all of this can be accomplished by checking for GPU and then appending cuda() after the net.
Training the Network
Having defined our network and tested that our dataloader works to our satisfaction, we're going to train the network. For your convenience, the training script is included and it is highly recommended that you try to gain a sense of what's happening. We'll talk more about training in the coming meetings.
Using  GPUs for training
In order to use GPUs for training, we also need to move our data on the GPU. Since data is a tensor, we can move them in the similar way we moved our MyNet i.e. appending cuda() after checking for a GPU.
Can you write Python code for it?

import torch.optim as optim
import torch.utils.data as data
from torch.autograd import Variable

tfs  = transforms.Compose([transforms.ToTensor(),
                          transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])

root='/home/akulshr/cifar-10-batches-py/'

cifar_train = CIFARLoader(root, transform=tfs) # create a "CIFARLoader instance".
cifar_train_loader = data.DataLoader(cifar_train, batch_size=4, shuffle=True, num_workers=2)

lossfn = nn.NLLLoss()
optimz = optim.SGD(net.parameters(), lr=1e-3, momentum=0.9)

def train(net):
    net.train()
    for ep in range(2):
        running_loss = 0.0
        for ix, (img,label) in enumerate(cifar_train_loader, 0):
            if gpu_avail:
                img = img.cuda()
                label = label.cuda()
            img_var = Variable(img)
            label_var = Variable(label)
            optimz.zero_grad()
#             print(img_var.size())
            op = net(img_var)
            loss = lossfn(op, label_var)
            loss.backward()
            optimz.step()

            running_loss += loss.data[0]
            if ix%2000 == 1999:
                print("[%d/%5d] Loss: %f"%(ep+1, ix+1, running_loss/2000))
                running_loss = 0.0
    print("Finished Training\n")

train(net)


I am working on the file "Fall2017/Module1_gpu.ipynb".
The first task is:
Pretty sweet! our neural network seems to have learnt something. Let's see how it does on the overall dataset:
Can you write Python code for it?

correct = 0.0
total = 0.0
for cache in cifar_test_loader:
    img, label = cache
    op = net(Variable(img))
    _, pred = torch.max(op.data, 1)
    total += label.size(0)
    correct += (pred==label).sum()

print("accuracy: %f"%(100*(correct/total)))


I am working on the file "Fall2017/Module1_gpu.ipynb".
The first task is:
So far we've trained the network and we're seeing some output loss. However, that's only one part of the story, since we need the model to perform well on unseen inputs. In order to do that we'll evaluate the dataset on the test_batch.
Can you write Python code for it?

def imshow(torch_tensor):
    torch_tensor = torch_tensor/2 + 0.5
    npimg = torch_tensor.numpy()
    plt.imshow(npimg.transpose(1,2,0))
    plt.show()

tfs  = transforms.Compose([transforms.ToTensor(),
                          transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])

root='/home/akulshr/cifar-10-batches-py/'

cifar_test = CIFARLoader(root, train=False, transform=tfs)
cifar_test_loader = data.DataLoader(cifar_test,batch_size=4, shuffle=False, num_workers=2)

# all possible classes in the CIFAR-10 dataset
classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

data_iter = iter(cifar_test_loader)
imgs,label = data_iter.next()

# Show the test images.
imshow(torchvision.utils.make_grid(imgs))

# Print the "Ground Truth labels"
print("Ground Truth: ")
print(' '.join(classes[label[j]] for j in range(4)))


I am working on the file "Fall2017/Module1_gpu.ipynb".
The first task is:
So we've got these images along with their labels as "ground truth". Now let's ask the neural network we just trained as to what it thinks the images are
Can you write Python code for it?

##################################################
# If you're using GPUs for training, then care   #
# must be taken to use "cpu()" before getting any #
# data for printing, displaying etc.              #
# e.g
# _, pred = torch.max(op.cpu().data)              #
#                                                 #
#################################################
data_iter = iter(cifar_test_loader)
imgs,label = data_iter.next()
op = net(Variable(imgs))
_, pred = torch.max(op.data, 1)
print("Guessed class: ")
print(' '.join(classes[pred[j]] for j in range(4)))


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
2.2.2 Floats
Can you write Python code for it?

# Have the full set of "calculator functions" but need the numpy package
import numpy as np
print (6.0 * 3, np.sin(2*np.pi))

# Floats can have a null value called nan, not a number
a = np.nan
3*a


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
2.2.3 Strings
Can you write Python code for it?

# Can concatenate, substring, find, count, ...

a = "The lazy"
b = "brown fox"
print ("Concatenation: ", a + b)
print ("First three letters: " + a[0:3])
print ("Index of 'z': " + str(a.find('z')))


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
2.3 Tuples
A tuple is an ordered sequence of objects. Tuples cannot be changed; they are immuteable.
Can you write Python code for it?

a_tuple = (1, 'ab', (1,2))
a_tuple

a_tuple[2]


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
2.5 Dictionaries
A dictionary is a kind of associates a key with a value. A value can be any object, even another dictionary.
Can you write Python code for it?

dessert_dict = {}  # Empty dictionary
dessert_dict['Dave'] = "Cake"
dessert_dict["Joe"] = ["Cake", "Pie"]
print (dessert_dict)

dessert_dict["Dave"]

# This produces an error
dessert_dict["Bernease"] = {}
dessert_dict

dessert_dict["Bernease"] = {"Favorite": ["sorbet", "cobbler"], "Dislike": "Brownies"}


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
2.7 A Shakespearean Detour: "What's in a Name?"
Deep vs. Shallow Copies
A deep copy can be manipulated separately. A shallow copy is a pointer to the same data as the original.
Can you write Python code for it?

# A first name shell game
first_int = 1
second_int = first_int
second_int += 1
second_int

# What is first_int?
first_int

# A second name shell game
a_list = ['a', 'aa', 'aaa']
b_list = a_list
b_list.append('bb')
b_list

# What is a_list?
a_list

# Create a deep copy
import copy
# A second name shell game
a_list = ['a', 'aa', 'aaa']
b_list = copy.deepcopy(a_list)
b_list.append('bb')
print("b_list = %s" % str(b_list))
print("a_list = %s" % str(a_list))


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
Adjusting the Plot Style
Matplotlib has a number of plot styles you can use. For example, if you like R you might use the ggplot style:
Other plot types
Pandas supports a range of other plotting types; you can find these by using the <TAB> autocomplete on the plot method:
For example, we can create a histogram of trip durations:
If you'd like to adjust the x and y limits of the plot, you can use the set_xlim() and set_ylim() method of the resulting object:
Breakout: Exploring the Data
Make a plot of the total number of rides as a function of month of the year (You'll need to extract the month, use a groupby, and find the appropriate aggregation to count the number in each group).
Split this plot by gender. Do you see any seasonal ridership patterns by gender?
Split this plot by user type. Do you see any seasonal ridership patterns by usertype?
Repeat the above three steps, counting the number of rides by time of day rather thatn by month.
Are there any other interesting insights you can discover in the data using these tools?
Using Files

Writing and running python modules
Using python modules in your Jupyter Notebook
Can you write Python code for it?

# A script for creating a dataframe with counts of the occurrence of a columns' values
df_count = df.groupby('from_station_id').count()
df_count1 = df_count[['trip_id']]
df_count2 = df_count1.rename(columns={'trip_id': 'count'})

df_count2.head()

def make_table_count(df_arg, groupby_column):
    df_count = df_arg.groupby(groupby_column).count()
    column_name = df.columns[0]
    df_count1 = df_count[[column_name]]
    df_count2 = df_count1.rename(columns={column_name: 'count'})
    return df_count2

dff = make_table_count(df, 'from_station_id')
dff.head()


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
By default, the values rather than the index are sorted. Use sort=False to turn this behavior off:
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
Key insight: Deep vs. Shallow Copies
* A deep copy can be manipulated separately from the original.
* A shallow copy cannot.
* Assigning a python immutable creates a deep copy. Non-immutables are shallow copies.
Name Resolution
The most common errors that you'll see in your python codes are:
* NameError
* AttributeError
A common error when using the bash shell is command not found.
Name resolution: Associating a name with code or data.
Resolving a name in the bash shell is done by searching the directories in the PATH environment variable. The first executable with the name is run.
Can you write Python code for it?

# Example 1 of name resolution in python
var = 10
def func(val):
    var = val + 1
    return val

# What is returned?
print("func(2) = %d" % func(2))
# What is var?
print("var = %d" % var)

# Example 2 of name resolution in python
var = 10
def func(val):
    return val + var

# What is returned?
print("func(2) = %d" % func(2))
# What is var?
print("var = %d" % var)


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
Mathematical operations on columns happen element-wise:
Can you write Python code for it?

trip_duration_hours = df['tripduration']/3600
trip_duration_hours[:3]

df['trip_duration_hours'] = df['tripduration']/3600

del df['trip_duration_hours']

df.head()

df.loc[[0,1],:]

df_long_trips = df[df['tripduration'] >10000]

sel = df['tripduration'] >10000
df_long_trips = df[sel]

len(df)

# Make a copy of a slice
df_subset = df[['starttime', 'stoptime']].copy()
df_subset['trip_hours'] = df['tripduration']/3600


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
Now we can simply call the plot() method of any series or dataframe to get a reasonable view of the data:
Can you write Python code for it?

import matplotlib.pyplot as plt
df['tripduration'].hist()


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
Or to break down rides by age:
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
The columns attribute gives us the column names
The index attribute gives us the index names
The dtypes attribute gives the data types of each column:
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
The shape attribute shows us the number of elements:
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
The simplest version of a groupby looks like this, and you can use almost any aggregation function you wish (mean, median, sum, minimum, maximum, standard deviation, count, etc.)
<data object>.groupby(<grouping values>).<aggregate>()
for example, we can group by gender and find the average of all numerical columns:
It's also possible to indes the grouped object like it is a dataframe:
You can even group by multiple values: for example we can look at the trip duration by time of day and by gender:
The unstack() operation can help make sense of this type of multiply-grouped data. What this technically does is split a multiple-valued index into an index plus columns:
Visualizing data with pandas
Of course, looking at tables of data is not very intuitive.
Fortunately Pandas has many useful plotting functions built-in, all of which make use of the matplotlib library to generate plots.
Whenever you do plotting in the IPython notebook, you will want to first run this magic command which configures the notebook to work well with plots:
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
We can explore other things as well: day of week, hour of day, etc.
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
What operations do we perform on tables?
Can you write Python code for it?

df = pd.DataFrame({'A': [1,2,3], 'B': [2, 4, 6], 'ccc': [1.0, 33, 4]})
df

sub_df = df[['A', 'ccc']]
sub_df

df['A'] + 2*df['B']

# Operations on a Pandas DataFrame


I am working on the file "Fall2018/02_Procedural_Python/Lecture-Python-And-Data.ipynb".
The first task is:
uncomment this to download the data:
Can you write Python code for it?

#!curl -o pronto.csv https://data.seattle.gov/api/views/tw7j-dfaw/rows.csv?accessType=DOWNLOAD


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:

Exercise
Objective: Replace the bash magic bits for downloading the Pronto data and uncompressing it with Python code.  Since the download is big, check if the zip file exists first before downloading it again.  Then load it into a pandas dataframe.
Notes:
* The os package has tools for checking if a file exists: os.path.exists
import os
filename = 'pronto.csv'
if os.path.exists(filename):
    print("wahoo!")
* Use the requests package to get the file given a url (got this from the requests docs)
import requests
url = 'https://s3.amazonaws.com/pronto-data/open_data_year_two.zip'
req = requests.get(url)
assert req.status_code == 200 # if the download failed, this line will generate an error
with open(filename, 'wb') as f:
    f.write(req.content)
* Use the zipfile package to decompress the file while reading it into pandas
import pandas as pd
import zipfile
csv_filename = '2016_trip_data.csv'
zf = zipfile.ZipFile(filename)
data = pd.read_csv(zf.open(csv_filename))
Now, use your code from above for the following URLs and filenames
| URL | filename | csv_filename |
|-----|----------|--------------|
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set1.zip?raw=true | open_data_year_two_set1.zip | 2016_trip_data_set1.csv |
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set2.zip?raw=true | open_data_year_two_set2.zip | 2016_trip_data_set2.csv |
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set3.zip?raw=true | open_data_year_two_set3.zip | 2016_trip_data_set3.csv |
What pieces of the data structures and flow control that we talked about earlier can you use?

Functions
For loops let you repeat some code for every item in a list.  Functions are similar in that they run the same lines of code for new values of some variable.  They are different in that functions are not limited to looping over items.
Functions are a critical part of writing easy to read, reusable code.
Create a function like:
def function_name (parameters):
    """
   ...
Can you write Python code for it?

def print_string(str):
    """This prints out a string passed as the parameter."""
    print(str)
    return


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
But in all, it isn't very Pythonesque to use indexes like that (unless you have another reason in the loop) and you would opt instead for the instructor in instructors form.
More often, you are doing something with the numbers that requires them to be integers, e.g. math.
Can you write Python code for it?

sum = 0
for i in range(10):
    sum += i
print(sum)


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
Conditionals can take and and or and not.  E.g.
Can you write Python code for it?

my_favorite = 'pie'

if my_favorite is 'cake' or my_favorite is 'pie':
    print(my_favorite + " : I have a recipe for that!")
else:
    print("Ew!  Who eats that?")


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
For
For loops are the standard loop, though while is also common.  For has the general form:
for items in list:
    do stuff
For loops and collections like tuples, lists and dictionaries are natural friends.
Can you write Python code for it?

for instructor in instructors:
    print(instructor)


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
If you don't provide an argument or too many, you get an error.
Parameters (or arguments) in Python are all passed by reference.  This means that if you modify the parameters in the function, they are modified outside of the function.
See the following example:
```
def change_list(my_list):
   """This changes a passed list into this function"""
   my_list.append('four');
   print('list inside the function: ', my_list)
   return
my_list = [1, 2, 3];
print('list before the function: ', my_list)
change_list(my_list);
print('list after the function: ', my_list)
```
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
Play around with the above function.
Functions can contain any code that you put anywhere else including:
* if...elif...else
* for...else
* while
* other function calls
Can you write Python code for it?

def print_name_age(first, last, age):
    print_name(first, last)
    print('Your age is %d' % (age))
    if age > 35:
        print('You are really old.')
    return

print_name_age(age=40, last='Beck', first='Dave')


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
There is a unique language feature call for...else
Can you write Python code for it?

sum = 0
for i in range(10):
    sum += i
else:
    print('final i = %d, and sum = %d' % (i, sum))


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
This would probably be better written as
Can you write Python code for it?

for index in range(len(instructors)):
    instructor = instructors[index]
    if instructor.endswith('Clown'):
        print(instructor + " doesn't sound like a real instructor name!")
    else:
        print(instructor + " is so smart... all those gooey brains!")


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
To call the function, use:
print_string("Dave is awesome!")
Note: The function has to be defined before you can call it!
Can you write Python code for it?


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
Usually we want conditional logic on both sides of a binary condition, e.g. some action when True and some when False
Can you write Python code for it?

if 'Dorkus the Clown' in instructors:
    print('There are fake names for class instructors in your list!')
else:
    print("Nothing to see here")


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
Variables have scope: global and local
In a function, new variables that you create are not saved when the function returns - these are local variables.  Variables defined outside of the function can be accessed but not changed - these are global variables, Note there is a way to do this with the global keyword.  Generally, the use of global variables is not encouraged, instead use parameters.
```
my_global_1 = 'bad idea'
my_global_2 = 'another bad one'
my_global_3 = 'better idea'
def my_function():
    print(my_global)
    my_global_2 = 'broke your global, man!'
    global my_global_3
    my_global_3 = 'still a better idea'
    return
my_function()
print(my_global_2)
print(my_global_3)
```
In general, you want to use parameters to provide data to a function and return a result with the return. E.g.
def sum(x, y):
    my_sum = x + y
    return my_sum
If you are going to return multiple objects, what data structure that we talked about can be used?  Give and example below.
Parameters have four different types:
| type | behavior |
|------|----------|
| required | positional, must be present or error, e.g. my_func(first_name, last_name) |
| keyword | position independent, e.g. my_func(first_name, last_name) can be called my_func(first_name='Dave', last_name='Beck') or my_func(last_name='Beck', first_name='Dave') |
| default | keyword params that default to a value if not provided |
Can you write Python code for it?

def print_name(first, last='the Clown'):
    print('Your name is %s %s' % (first, last))
    return


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
You can combine loops and conditionals:
Can you write Python code for it?

for instructor in instructors:
    if instructor.endswith('Clown'):
        print(instructor + " doesn't sound like a real instructor name!")
    else:
        print(instructor + " is so smart... all those gooey brains!")


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
You can iterate over letters in a string
Can you write Python code for it?

my_string = "DIRECT"
for c in my_string:
    print(c)


I am working on the file "Fall2018/02_Procedural_Python/procedural_programming_in_python.ipynb".
The first task is:
You can skip stuff in a loop with continue
Can you write Python code for it?

sum = 0
for i in range(10):
    if (i == 5):
        continue
    else:
        sum += i
print(sum)


I am working on the file "Fall2018/04_ProjectOverview_AnalysisWorkflow/analysis_workflow.ipynb".
The first task is:
Colin will provide more details about function, such as variable scope, and multiple return values.
Can you write Python code for it?

# Function to download from a URL
def download(url, filename):
    print("Downloading", filename)
    #request.urlretrieve(url, filename)

download(TRIP_DATA, TRIP_FILE)

# Enhancing function to detect file already present
import os.path
def download(url, filename):
    if os.path.isfile(filename):
        print("Already present %s." % filename)
    else:
        print("Downloading %s" % filename)
        #request.urlretrieve(url, filename)

download(TRIP_DATA, "none.csv")

import download
download.download_file(TRIP_DATA, "none.csv")


I am working on the file "Fall2018/04_ProjectOverview_AnalysisWorkflow/analysis_workflow.ipynb".
The first task is:
Two challenges
1. The file is big. We don't want to download it if it's already present.
1. We're going to repeatedly download files. We don't want to just copy and paste the same code.
Encapsulating Repeated Code In Functions
A function is code that can be invoked by many callers. A function may have arguments that are specified by the caller, and returns values created by the function.
Can you write Python code for it?

# Example function
def xyz(input):     # The function's name is "func". It has one argument "input".
    return int(input) + 1 # The function returns one value, input + 1

print (xyz("3"))
#a = xyz(3)
#print (xyz(a))

def addTwo(input1, input2):
    return input1 + input2
#
addTwo(1, 2)


I am working on the file "Fall2018/05_Functions/python_functions.ipynb".
The first task is:
Exercise - Write a Function

Write a function that finds two factors of a provided number if it is not prime.
What did you name the function?
What are the formal arguments? What did you name them?
What value(s) are returned?

Crafting a Function
Crafting refers to what you want a function to do.
- What are the arguments?
- Should it refer to names outside the function definition?
- Should it return a value?
Problem: Write a function that finds all of the prime numbers less than a given value.
Can you write Python code for it?

# Function header: name, arguments
# Function logic
# return something


I am working on the file "Fall2018/05_Functions/python_functions.ipynb".
The first task is:
If you modify an object (like a list or a dataframe) inside of a function, the modifications will affect its value outside of the function
Can you write Python code for it?

def add_name_to_list(name_list, new_name):
    name_list.append(new_name)

teachers = ["Bernease", "Dave", "Joe"]
print(teachers)
add_name_to_list(teachers, "Colin")
print(teachers)


I am working on the file "Fall2018/05_Functions/python_functions.ipynb".
The first task is:
Test cases
What are some numbers that we might use as tests to see if our function is working right?
Think of different classes of numbers that might have different behavior, like even vs. odd numbers, and make
sure you are including examples from each class.
Also try to think of some "edge cases" that might cause strange behavior.
Think of some good test cases for your function and see if you get the results you expect:
Can you write Python code for it?

# Check your function on several test cases to make sure you are getting the correct output:


I am working on the file "Fall2018/05_Functions/python_functions.ipynb".
The first task is:
Variable names and scope
In Python, variables have a scope (a context in which they are valid).
Variables created in a function cannot be referenced outside of a function
Can you write Python code for it?

def print_message(message):
    message_to_print = "Here is your message: " + message
    print(message_to_print)

my_message = "Hello, class!"
print_message(my_message)

#print(message_to_print) # this will cause an error. This variable only exists within the function.


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
INSTRUCTOR NOTE:

Emphasize the difference between the single and double equal operator.
Can you write Python code for it?


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
INSTRUCTOR NOTE:

Run as-is. Run. Error. Returns SyntaxError: Missing parentheses in call to print.
Add parentheses. Run. Still an error. Returns SyntaxError: EOL while scanning string literal.
Add closing quotation mark. Run. Should be successful.
Can you write Python code for it?

x = 1; y = 2
b = x == y # Boolean variable that is true when x & y have the same value
b = 1 = 2


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
If we can't easily see the bug here, let's add print statements to see the variables change over time.
INSTRUCTOR NOTE:

Add print statements in tiered way, starting with simple print statements.

Point out that may need slight refactor on result.
python
def entropy(p):
    print(p)
    items = p * np.log(p)
    print(items)
    result = -np.sum(items)
    print(result)
    return result

Show complication of reading multiple print statements without labels.

Add labels so code looks like below.
python
def entropy(p):
    print("p=%s" % p)
    items = p * np.log(p)
    print("items=%s" % items)
    result = -np.sum(items)
    print("result=%s" % result)
    return result
Can you write Python code for it?


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
Now it works fine for the first set of inputs. Let's try other inputs.
We should have documented the inputs to the function!
Can you write Python code for it?

# Create a vector of probabilities.
p = np.arange(start=5., stop=-1., step=-0.5)
p /= np.sum(p)
p

entropy(p)


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
Semantic errors
Say we're trying to confirm that a trigonometric identity holds. Let's use the basic relationship between sine and cosine, given by the Pythagorean identity"
$$
\sin^2 \theta + \cos^2 \theta = 1
$$
We can write a function to check this:
Can you write Python code for it?

import math

'''Checks that Pythagorean identity holds for one input, theta'''
def check_pythagorean_identity(theta):
    return math.sin(theta)**2 + math.cos(theta)*2 == 1

check_pythagorean_identity(12)


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
Syntax errors
Can you write Python code for it?

print "This should only work in Python 2.x, not 3.x used in this class.


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Debugging.ipynb".
The first task is:
Using Python's debugger, pdb
Python comes with a built-in debugger called pdb.  It allows you to step line-by-line through a computation and examine what's happening at each step.  Note that this should probably be your last resort in tracing down a bug.  I've probably used it a dozen times or so in five years of coding.  But it can be a useful tool to have in your toolbelt.
You can use the debugger by inserting the line
python
import pdb; pdb.set_trace()
within your script. To leave the debugger, type "exit()". To see the commands you can use, type "help".
Let's try this out:
Can you write Python code for it?

def entropy(p):
    import pdb; pdb.set_trace()
    items = p * np.log(p)
    return -np.sum(items)


I am working on the file "Fall2018/06_Projects_Exceptions_Testing/Exceptions.ipynb".
The first task is:
What do you do when you get an exception?
First, you can feel relieved that you caught a problematic element of your software! Yes, relieved. Silent fails are much worse. (Again, another plug for testing.)
Generating Exceptions
Why generate exceptions? (Don't I have enough unintentional errors?)
Can you write Python code for it?

import pandas as pd
def func(df):
    """"
    :param pd.DataFrame df: should have a column named "hours"
    """
    if not "hours" in df.columns:
        raise ValueError("DataFrame should have a column named 'hours'.")

df = pd.DataFrame({'hours': range(10) })
func(df)

df = pd.DataFrame({'years': range(10) })
func(df)


I am working on the file "Fall2018/07-Jupyter-Notebook-In-Depth/LorenzSystem.ipynb".
The first task is:
After interacting with the system, we can take the result and perform further computations. In this case, we compute the average positions in \(x\), \(y\) and \(z\).
Can you write Python code for it?

xyz_avg = x_t.mean(axis=1)

xyz_avg.shape


I am working on the file "Fall2018/07-Jupyter-Notebook-In-Depth/LorenzSystem.ipynb".
The first task is:
Computing the trajectories and plotting the result
We define a function that can integrate the differential equations numerically and then plot the solutions. This function has arguments that control the parameters of the differential equation (\(\sigma\), \(\beta\), \(\rho\)), the numerical integration (N, max_time) and the visualization (angle).
Can you write Python code for it?

def solve_lorenz(N=10, angle=0.0, max_time=4.0, sigma=10.0, beta=8./3, rho=28.0):

    fig = plt.figure()
    ax = fig.add_axes([0, 0, 1, 1], projection='3d')
    ax.axis('off')

    # prepare the axes limits
    ax.set_xlim((-25, 25))
    ax.set_ylim((-35, 35))
    ax.set_zlim((5, 55))

    def lorenz_deriv(x_y_z, t0, sigma=sigma, beta=beta, rho=rho):
        """Compute the time-derivative of a Lorenz system."""
        x, y, z = x_y_z
        return [sigma * (y - x), x * (rho - z) - y, x * y - beta * z]

    # Choose random starting points, uniformly distributed from -15 to 15
    np.random.seed(1)
    x0 = -15 + 30 * np.random.random((N, 3))

    # Solve for the trajectories
    t = np.linspace(0, max_time, int(250*max_time))
    x_t = np.asarray([integrate.odeint(lorenz_deriv, x0i, t)
                      for x0i in x0])

    # choose a different color for each trajectory
    colors = plt.cm.viridis(np.linspace(0, 1, N))

    for i in range(N):
        x, y, z = x_t[i,:,:].T
        lines = ax.plot(x, y, z, '-', c=colors[i])
        plt.setp(lines, linewidth=2)

    ax.view_init(30, angle)
    plt.show()

    return t, x_t


I am working on the file "Fall2018/07-Jupyter-Notebook-In-Depth/LorenzSystem.ipynb".
The first task is:
Creating histograms of the average positions (across different trajectories) show that on average the trajectories swirl about the attractors.
Can you write Python code for it?

plt.hist(xyz_avg[:,0])
plt.title('Average $x(t)$');

plt.hist(xyz_avg[:,1])
plt.title('Average $y(t)$');


I am working on the file "Fall2018/07-Jupyter-Notebook-In-Depth/LorenzSystem.ipynb".
The first task is:
Let's call the function once to view the solutions. For this set of parameters, we see the trajectories swirling around two points, called attractors.
Can you write Python code for it?


I am working on the file "Fall2018/07-Jupyter-Notebook-In-Depth/LorenzSystem.ipynb".
The first task is:
Using IPython's interactive function, we can explore how the trajectories behave as we change the various parameters.
Can you write Python code for it?


I am working on the file "Fall2018/07-Jupyter-Notebook-In-Depth/WidgetsDemo.ipynb".
The first task is:
Interact with Plotting
This can become very powerful when interacting with a plotting command:
Can you write Python code for it?

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 1000)

def plot_sine(amplitude, frequency, phase):
    y = amplitude * np.sin(frequency * x - phase)
    plt.plot(x, y)
    plt.ylim(-6, 6)

interact(plot_sine,
         amplitude=(0.0, 5.0),
         frequency=(0.1, 10),
         phase=(-5.0, 5.0));


I am working on the file "Fall2018/09_UnitTests/unit-tests.ipynb".
The first task is:
Code for homework or your work should use test files. In this lesson, we'll show how to write test codes in a Jupyter notebook. This is done for pedidogical reasons. It is NOT not something you should do in practice, except as an intermediate exploratory approach.
As expected, the first test passes, but the second test fails.
Exercise

Rewrite the above one-shot test for entropy using the unittest infrastructure.
Can you write Python code for it?

# Implementating a pattern test. Use functions in the test.
import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_equal_probability(self):
        def test(count):
            """
            Invokes the entropy function for a number of values equal to count
            that have the same probability.
            :param int count:
            """
            raise RuntimeError ("Not implemented.")
        #
        test(2)
        test(20)
        test(200)

#test_setup(TestEntropy)

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):
  """Write the full set of tests."""


I am working on the file "Fall2018/09_UnitTests/unit-tests.ipynb".
The first task is:
Suppose that all of the probability of a distribution is at one point. An example of this is a coin with two heads. Whenever you flip it, you always get heads. That is, the probability of a head is 1.
What is the entropy of such a distribution? From the calculation above, we see that the entropy should be $log(1)$, which is 0. This means that we have a test case where we know the result!
Can you write Python code for it?

# One-shot test. Need to know the correct answer.
entries = [
    [0, [1]],
]

for entry in entries:
    ans = entry[0]
    prob = entry[1]
    if not np.isclose(entropy(prob), ans):
        print("Test failed!")
print ("Test completed!")


I am working on the file "Fall2018/09_UnitTests/unit-tests.ipynb".
The first task is:
Test Files
Although I presented the elements of unittest in a notebook. your tests should be in a file. If the name of module with the code under test is foo.py, then the name of the test file should be test_foo.py.
The structure of the test file will be very similar to cells above. You will import unittest. You must also import the module with the code under test. Take a look at test_prime.py in this directory to see an example.
Discussion
Question: What tests would you write for a plotting function?
Test Driven Development
Start by writing the tests. Then write the code.
We illustrate this by considering a function geomean that takes a list of numbers as input and produces the geometric mean on output.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntryopy(unittest.TestCase):

    def test_oneshot(self):
        self.assertEqual(geomean([1,1]), 1)

    def test_oneshot2(self):
        self.assertEqual(geomean([3, 3, 3]), 3)

#test_setup(TestGeomean)

#def geomean(argument?):
#    return ?


I am working on the file "Fall2018/09_UnitTests/unit-tests.ipynb".
The first task is:
Testing For Exceptions
Edge test cases often involves handling exceptions. One approach is to code this directly.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        try:
            entropy([0.1, 0.5])
            self.assertTrue(False)
        except ValueError:
            self.assertTrue(True)

#test_setup(TestEntropy)


I am working on the file "Fall2018/09_UnitTests/unit-tests.ipynb".
The first task is:
Unit Tests
Overview and Principles
Testing is the process by which you exercise your code to determine if it performs as expected. The code you are testing is referred to as the code under test.
There are two parts to writing tests.
1. invoking the code under test so that it is exercised in a particular way;
1. evaluating the results of executing code under test to determine if it behaved as expected.
The collection of tests performed are referred to as the test cases. The fraction of the code under test that is executed as a result of running the test cases is referred to as test coverage.
For dynamical languages such as Python, it's extremely important to have a high test coverage. In fact, you should try to get 100% coverage. This is because little checking is done when the source code is read by the Python interpreter. For example, the code under test might contain a line that has a function that is undefined. This would not be detected until that line of code is executed.
Test cases can be of several types. Below are listed some common classifications of test cases.
- Smoke test. This is an invocation of the code under test to see if there is an unexpected exception. It's useful as a starting point, but this doesn't tell you anything about the correctness of the results of a computation.
- One-shot test. In this case, you call the code under test with arguments for which you know the expected result.
- Edge test. The code under test is invoked with arguments that should cause an exception, and you evaluate if the expected exception occurrs.
- Pattern test - Based on your knowledge of the calculation (not implementation) of the code under test, you construct a suite of test cases for which the results are known or there are known patterns in these results that are used to evaluate the results returned.
Another principle of testing is to limit what is done in a single test case. Generally, a test case should focus on one use of one function. Sometimes, this is a challenge since the function being tested...
Can you write Python code for it?

import numpy as np
# Code Under Test
def entropy(ps):
    items = ps * np.log(ps)
    return np.abs(-np.sum(items))

# Smoke test
entropy([0.2, 0.8])


I am working on the file "Fall2018/09_UnitTests/unit-tests.ipynb".
The first task is:
unittest provides help with testing exceptions.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        with self.assertRaises(ValueError):
            entropy([0.1, 0.5])

suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)


I am working on the file "Fall2020/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
Forward mode vs. backward mode: $m \geq n$
Can you write Python code for it?

def fvec(x, A, b):
    y = A @ x + b
    return jnp.exp(y - jnp.max(y)) / jnp.sum(jnp.exp(y - jnp.max(y)))

grad_fvec = jax.jit(jax.grad(fvec))
jac_fvec = jax.jacobian(fvec)
fmode_fvec = jax.jit(jax.jacfwd(fvec))
bmode_fvec = jax.jit(jax.jacrev(fvec))

n = 1000
m = 1000
x = jax.random.normal(jax.random.PRNGKey(0), (n, ))
A = jax.random.normal(jax.random.PRNGKey(0), (m, n))
b = jax.random.normal(jax.random.PRNGKey(0), (m, ))

J = jac_fvec(x, A, b)
print(J.shape)
grad_fvec(x, A, b)

print("Check correctness", jnp.linalg.norm(fmode_fvec(x, A, b) - bmode_fvec(x, A, b)))
print("Check shape", fmode_fvec(x, A, b).shape, bmode_fvec(x, A, b).shape)
print("Time forward mode")
%timeit fmode_fvec(x, A, b).block_until_ready()
print("Time backward mode")
%timeit bmode_fvec(x, A, b).block_until_ready()

n = 10
m = 1000
x = jax.random.normal(jax.random.PRNGKey(0), (n, ))
A = jax.random.normal(jax.random.PRNGKey(0), (m, n))
b = jax.random.normal(jax.random.PRNGKey(0), (m, ))

print("Check correctness", jnp.linalg.norm(fmode_fvec(x, A, b) - bmode_fvec(x, A, b)))
print("Check shape", fmode_fvec(x, A, b).shape, bmode_fvec(x, A, b).shape)
print("Time forward mode")
%timeit fmode_fvec(x, A, b).block_until_ready()
print("Time backward mode")
%timeit bmode_fvec(x, A, b).block_until_ready()


I am working on the file "Fall2020/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
Hessian-by-vector product
Can you write Python code for it?

def hvp(f, x, z, *args):
    def g(x):
        return f(x, *args)
    return jax.jvp(jax.grad(g), (x,), (z,))[1]

n = 3000
x = jax.random.normal(jax.random.PRNGKey(0), (n, ))
A = jax.random.normal(jax.random.PRNGKey(0), (n, n))
b = jax.random.normal(jax.random.PRNGKey(0), (n, ))
z = jax.random.normal(jax.random.PRNGKey(0), (n, ))

print("Check correctness", jnp.linalg.norm(2 * A.T @ (A @ z) - hvp(f, x, z, A, b)))
print("Time for hvp by hands")
%timeit (2 * A.T @ (A @ z)).block_until_ready()
print("Time for hvp via jvp, NO jit")
%timeit hvp(f, x, z, A, b).block_until_ready()
print("Time for hvp via jvp, WITH jit")
%timeit jax.jit(hvp, static_argnums=0)(f, x, z, A, b).block_until_ready()


I am working on the file "Fall2020/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
Random numbers in JAX

JAX focuses on the reproducibility of the runs
Analogue of random seed is the necessary argument of all functions that generate something random
More details and references on the design of random submodule are here
Can you write Python code for it?

n = 1000
x = jax.random.normal(jax.random.PRNGKey(0), (n, ))
A = jax.random.normal(jax.random.PRNGKey(0), (n, n))
b = jax.random.normal(jax.random.PRNGKey(0), (n, ))

print("Check correctness", jnp.linalg.norm(gradf(x, A, b) - 2 * A.T @ (A @ x - b)))
print("Compare speed")
print("Analytical gradient")
%timeit 2 * A.T @ (A @ x - b)
print("Grad function")
%timeit gradf(x, A, b).block_until_ready()
jit_gradf = jax.jit(gradf)
print("Jitted grad function")
%timeit jit_gradf(x, A, b).block_until_ready()

hess_func = jax.jit(jax.hessian(f))
print("Check correctness", jnp.linalg.norm(2 * A.T @ A - hess_func(x, A, b)))
print("Time for hessian")
%timeit hess_func(x, A, b).block_until_ready()
print("Emulate hessian and check correctness",
      jnp.linalg.norm(jax.jit(hess_func)(x, A, b) - jax.jacfwd(jax.jacrev(f))(x, A, b)))
print("Time of emulating hessian")
hess_umul_func = jax.jit(jax.jacfwd(jax.jacrev(f)))
%timeit hess_umul_func(x, A, b).block_until_ready()


I am working on the file "Fall2021/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
By default, JAX exploits single-precision numbers float32
You can enable double precision (float64) by hands.
Can you write Python code for it?

from jax.config import config
config.update("jax_enable_x64", True)

@jax.jit
def f(x, A, b):
    res = A @ x - b
    return res @ res

gradf = jax.grad(f, argnums=0, has_aux=False)


I am working on the file "Fall2021/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
Forward mode vs. backward mode: $m \geq n$
Can you write Python code for it?


I am working on the file "Fall2021/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
Forward mode vs. backward mode: $m \ll n$
Can you write Python code for it?

fmode_f = jax.jit(jax.jacfwd(f))
bmode_f = jax.jit(jax.jacrev(f))
print("Check correctness", jnp.linalg.norm(fmode_f(x, A, b) - bmode_f(x, A, b)))
print("Forward mode")
%timeit fmode_f(x, A, b).block_until_ready()
print("Backward mode")
%timeit bmode_f(x, A, b).block_until_ready()


I am working on the file "Fall2021/03-MatrixCalculus/jax_autodiff_tutorial.ipynb".
The first task is:
Hessian-by-vector product
Can you write Python code for it?


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
By the way getting correlation coefficients among all members of the Data Frame is as simple as:
Can you write Python code for it?

aonao.corr()

AO_mm = AO.resample("A")
AO.plot(style='k-', alpha = 0.3)
AO_mm.plot(style='b--', linewidth =2)

aonao['mon'] = aonao.index.month
aonao

monmean = aonao['1950':'2013'].groupby('mon').aggregate(mean)
monmean.plot(kind='bar')