I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
4 - Moments feature generation
The simplest and most fundamental way to numerically describe the shape of a distribution of values is using moments. The first moment, mean $\mu$, characterizes the central tendency of the distribution. The second moment, variance $\sigma^2$, characterizes the spread of the values about the central tendency. The third moment, skewness $\gamma_1$, characterizes the symmetry (or lack thereof) about the central tendency.
We will calculate the first three moments (with one small modification) for each rolling window size at every depth. The small modification is that instead of variance $\sigma^2$, we are calculating standard deviation $\sigma$ because the results of variance $\sigma^2$ produce values with units of the mean squared $\mu^2$. As a result, feature importance of variance is artificially high due to the dimension of the variance values. Standard deviation $\sigma$ has the same dimension as mean $\mu$.
With respect to facies prediction, now, in addition to the raw WL log inputs, we will describe at multiple scales the shapes of the distributions of WL log responses associated with each facies.
Can you write Python code for it?

# function to calculate moments using a rolling window

def rolling_moments(arr, w, moment ='mean'):
    """- pad input array by (w-1)/2 samples at the top and bottom
       - apply rolling window function
       - calculate moment: mean (default), var, or skew"""
    mom = []
    arr = np.pad(arr, ((w-1)/2, (w-1)/2), 'edge')
    if moment == 'std':
        return np.array(np.std(rolling_window(arr, w), 1))
    elif moment == 'skew':
        return np.array(sp.stats.skew(rolling_window(arr, w), 1))
    else:
        return np.array(np.mean(rolling_window(arr, w), 1))

moments = ['mean', 'std', 'skew']

# for training data
# calculate all moments for all logs, for all wells

train_mome_df = pd.DataFrame()             # final dataframe

grouped = train_data['Well Name'].unique()

for well in grouped:                  # for each well
    new_df = pd.DataFrame()           # make a new temporary dataframe

    for log in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND' ,'PE']:
        for mo in moments:            # for each moment
                                      # calculate the rolling moments with each window size
                                      # and also the mean of moments (all window sizes)
            results = np.array([rolling_moments(train_feat_df[log][train_feat_df['Well Name'] == well],
                                               size, moment = mo) for size in sizes])
            mean_result = np.mean(results, axis=0)
                                      # write to temporary dataframe
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[0])] = results[0]
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[1])] = results[1]
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[2])] = results[2]
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[3])] = results[3]
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[4])] = results[4]
            new_df[str(log)+ '_' + str(mo)+'_wsize=' +str(sizes[5])] = results[5]
           ...


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
6 - Concatenate results with input into a single numpy array, then make it into final dataframe
Can you write Python code for it?

train_arr_fin = (np.concatenate((train_data.values, train_deriv_df.values, train_mome_df.values, train_glcm_sym_df.values, train_glcm_asym_df.values), axis=1))
print np.shape(train_arr_fin)
cols_trn = list(train_data) + list(train_deriv_df) + list(train_mome_df) + list(train_glcm_sym_df) + list(train_glcm_asym_df)
train_final_df = pd.DataFrame(train_arr_fin, columns=cols_trn)

train_final_df.describe()
#train_final_df.dtypes

test_arr_fin = (np.concatenate((test_data.values, test_deriv_df.values, test_mome_df.values, test_glcm_sym_df.values, test_glcm_asym_df.values), axis=1))
print np.shape(test_arr_fin)
cols_tst = list(test_data) + list(test_deriv_df) + list(test_mome_df) + list(test_glcm_sym_df) + list(test_glcm_asym_df)
test_final_df = pd.DataFrame(test_arr_fin, columns=cols_tst)
test_final_df.describe()
#test_final_df.dtypes

train_headings = list(train_data)[3:] + list(train_deriv_df) + list(train_mome_df) + list(train_glcm_sym_df) + list(train_glcm_asym_df)
for head in train_headings:
    train_final_df[head] = train_final_df[head].astype('float64')

train_final_df['Facies'] = train_final_df['Facies'].astype('int64')
train_final_df['Formation'] = train_final_df['Formation'].astype('category')
train_final_df['Well Name'] = train_final_df['Well Name'].astype('category')
train_final_df['NM_M'] = train_final_df['NM_M'].astype('int64')

train_final_df.describe()

test_headings = list(test_data)[3:] + list(test_deriv_df) + list(test_mome_df) + list(test_glcm_sym_df) + list(test_glcm_asym_df)
for head in test_headings:
    test_final_df[head] = test_final_df[head].astype('float64')

test_final_df['Formation'] = test_final_df['Formation'].astype('category')
test_final_df['Well Name'] = test_final_df['Well Name'].astype('category')
test_final_df['NM_M'] = test_final_df['NM_M'].astype('int64')

test_final_df.describe()

# just a quick test
train_final_df['PE_GLCM_correlation_asym_wsize=33'] == train_final_df['PE_GLCM_correlation_wsize=33']


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
7 - PCA dimensionality analysis
Run PCA, and look at the significance of the components.
The explained variance shows how much information (variance) can be attributed to each of the principal components, and its cumulative sum can be used to determine the number of components to select:
Can you write Python code for it?

pca = decomposition.PCA()
scld = train_final_df.drop(['Well Name', 'Formation', 'Facies'],axis=1)
stdscaler2 = preprocessing.StandardScaler().fit(scld)
scld = stdscaler2.transform(scld)
pca.fit(scld)

np.set_printoptions(suppress=True) # so output is not in scientific notation
print np.cumsum(pca.explained_variance_ratio_)[:170]

fig = plt.figure(figsize=(14,8))
plt.plot(np.arange(1, len(np.cumsum(pca.explained_variance_ratio_))+1, 1)[:170],
         np.cumsum(pca.explained_variance_ratio_)[:170])

plt.show


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
And now let's calculate asymmetric GLCM properties using only the upward neighbour:
Can you write Python code for it?

# for training data
train_glcm_asym_df = pd.DataFrame()        # final dataframe
grouped1 = train_data['Well Name'].unique()

for well1 in grouped1:                   # for each well
    new_dfg1 = pd.DataFrame()           # make a new temporary dataframe

    for log1 in ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE']:   # for each log
        for me in methods:            # for each property
                                      # calculate rolling GLCM properties with each window size
                                      # and also the mean of glcms (all window sizes)
            lg1 = train_eq_glcm_df[log][train_eq_glcm_df['Well Name'] == well1]
            results1 = np.array([gprops_calc(lg1.astype(int), wd, lv = 64, sym = False, prop = me) for wd in sizes])
            mean_result1 = np.mean(results1, axis=0)

                                      # write to temporary dataframe
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[0])] = results1[0]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[1])] = results1[1]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[2])] = results1[2]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[3])] = results1[3]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[4])] = results1[4]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[5])] = results1[5]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=' +str(sizes[6])] = results1[6]
            new_dfg1[str(log1)+ '_GLCM_' + str(me)+'_asym_wsize=ave'] = mean_result1
                                      # append all rows of temporary dataframe to final dataframe

    train_glcm_asym_df = pd.concat([train_glcm_asym_df, new_dfg1])
train_glcm_asym_df.describe()

# for test data
test_glcm_asym_df = pd.DataFrame()        # final dataframe
grouped1 = test_data['Well Name'].unique()

for well1 in grouped1:                   # for each...


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
Feature engineering

The code and ideas to engineer new features used in this notebook,  by Matteo Niccoli and Mark Dahl, with contributions by Daniel Kittridge, are licensed under a Creative Commons Attribution 4.0 International License.
1 - Clean up and rescale data
Can you write Python code for it?

# for training data
# import data and filling missing PE values with average

filename = 'facies_vectors.csv'
train_data = pd.read_csv(filename)

train_data['PE'].fillna((train_data['PE'].mean()), inplace=True)
print  np.shape(train_data)
train_data['PE'].fillna((train_data['PE'].mean()), inplace=True)
print  np.shape(train_data)

# for blind test data
# import data and filling missing PE values with average

filename = 'nofacies_data.csv'
test_data = pd.read_csv(filename)

test_data['PE'].fillna((test_data['PE'].mean()), inplace=True)
print  np.shape(test_data)
test_data['PE'].fillna((test_data['PE'].mean()), inplace=True)
print  np.shape(test_data)

pd.set_option('display.float_format', lambda x: '%.4f' % x)
train_data.describe()

pd.set_option('display.float_format', lambda x: '%.4f' % x)
test_data.describe()


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
It looks like from the plot above that it would take more than 100 PCs for the cumulative explained variance ratio reache 0.99. We will use another technique to reduce the number of features to go into the classification.
Can you write Python code for it?

train_final_df.to_csv('train_engineered_features.csv', sep=',',  index=False)

test_final_df.to_csv('test_engineered_features.csv', sep=',',  index=False)


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
Similar to the step preceeding moments calculation, we will rescale the raw WL logs for GLCM property calculation so each resulting property is unaffected by the magnitude of the raw WL log values. skimage.feature.greycomatrix requires uint8 values, so we need an alternative to sklearn.preprocessing.StandardScaler. Unlike with calculating moments, preserving the shape of the histogram is not important to the integrity of a GLCM property. We will use histogram equalization, which flattens a histogram (puts an equal number of values in each bin). To maximize the effectiveness of a GLCM, it is commonly wise to reduce the bit depth from 8 to avoid processing expense and noise caused by empty matrix entries. After some trial and error, we found the 64 bins works nicely. Note that 64 bins results in a 64x64 matrix at every depth for every rolling window size.
Can you write Python code for it?

# functions to equalize histogram of features to go into GLCM calculation
def eqlz(arr, bins):
    return (bins-1) * exposure.equalize_hist(arr)
def eqlz_along_axis(arr, bins):
    return np.apply_along_axis(eqlz, 0, arr, bins)

# for training data
# equalize features
train_vect_glcm = train_data.drop(['Formation', 'Well Name', 'Depth','Facies'], axis=1)
train_eqvect_glcm = eqlz_along_axis(train_vect_glcm, 64)
# values into dataframe
train_eqvect_glcm_df = pd.DataFrame(train_eqvect_glcm, columns=list(train_vect_glcm))
train_eqvect_glcm_df = np.round(train_eqvect_glcm_df).astype(int)
train_eq_glcm_df = pd.concat((train_data[['Depth', 'Well Name', 'Formation', 'Facies']], train_eqvect_glcm_df),1)
train_eq_glcm_df.head()

# for blind test data
# equalize features
test_vect_glcm = test_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
test_eqvect_glcm = eqlz_along_axis(test_vect_glcm, 64)
# values into dataframe
test_eqvect_glcm_df = pd.DataFrame(test_eqvect_glcm, columns=list(test_vect_glcm))
test_eqvect_glcm_df = np.round(test_eqvect_glcm_df).astype(int)
test_eq_glcm_df = pd.concat((test_data[['Depth', 'Well Name', 'Formation']], test_eqvect_glcm_df),1)
test_eq_glcm_df.head()


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
To keep feature importance on a level playing field, we will rescale each WL log before calculating moments. We will use sklearn.preprocessing.StandardScaler.
Can you write Python code for it?

# standardize features for processing
train_vectors = train_data.drop(['Formation', 'Well Name', 'Depth','Facies'], axis=1)
stdscaler1 = preprocessing.StandardScaler().fit(train_vectors)
train_features = stdscaler1.transform(train_vectors)

# build dataframe with scaled features
train_feat_df = pd.DataFrame(train_features, columns=list(train_vectors))
train_feat_df = pd.concat((train_data[['Depth', 'Well Name', 'Formation', 'Facies']], train_feat_df),1)
print train_feat_df.shape
train_feat_df.describe()


I am working on the file "MandMs/04_faciesClassification_MandMs_featureEngineering_v2.ipynb".
The first task is:
We want to scale the blind test data too, but we want it scaled using the same fit as the training data
Can you write Python code for it?

# standardize features for processing
test_vectors = test_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
test_features = stdscaler1.transform(test_vectors)

# build dataframe with scaled features
test_feat_df = pd.DataFrame(test_features, columns=list(test_vectors))
test_feat_df = pd.concat((test_data[['Depth', 'Well Name', 'Formation']], test_feat_df),1)
print test_feat_df.shape
test_feat_df.describe()


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
Confusion matrix
Let's see how we do with predicting the actual facies, by looking at a confusion matrix. We do this by keeping the parameters from the previous section, but creating a brand new classifier. So when we fit the data, it won't have seen it before.
Can you write Python code for it?

from sklearn import svm
SVC_classifier_conf = svm.SVC(C=2, cache_size=800, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.01, kernel='rbf',
  max_iter=-1, probability=False, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

svc_pred = SVC_classifier_conf.fit(X,y)
svc_pred = SVC_classifier_conf.predict(X)

from sklearn.metrics import confusion_matrix
from classification_utilities import display_cm, display_adj_cm

conf = confusion_matrix(svc_pred, y)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
Final classifier
We now train our final classifier with leave-one-well-out validation. Again, we keep the parameters from the previous section, but creating a brand new classifier. So when we fit the data, it won't have seen it before.
Can you write Python code for it?

from sklearn import svm
SVC_classifier_LOWO = svm.SVC(C=2, cache_size=800, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.01, kernel='rbf',
  max_iter=-1, probability=False, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

training_data = pd.read_csv('../training_data.csv')
X = training_data.drop(['Formation', 'Well Name', 'Facies'], axis=1).values
scaler = preprocessing.StandardScaler().fit(X)
X = scaler.transform(X)
y = training_data['Facies'].values
wells = training_data["Well Name"].values

logo = LeaveOneGroupOut()

f1_SVC = []

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    SVC_classifier_LOWO.fit(X[train], y[train])
    pred = SVC_classifier_LOWO.predict(X[test])
    sc = f1_score(y[test], pred, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_SVC.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_SVC)/(1.0*(len(f1_SVC))))


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
NB:  the final classifier above resulted in a validated F1 score of 0.536 with the blind facies in the STUART and CRAWFORD wells. This compares favourably with the previous SVM implementations.
However, had we used a parameter C equal to 5:
Can you write Python code for it?

from sklearn import svm
SVC_classifier_LOWO_C5 = svm.SVC(C=5, cache_size=800, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.01, kernel='rbf',
  max_iter=-1, probability=False, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

training_data = pd.read_csv('../training_data.csv')
X = training_data.drop(['Formation', 'Well Name', 'Facies'], axis=1).values
scaler = preprocessing.StandardScaler().fit(X)
X = scaler.transform(X)
y = training_data['Facies'].values
wells = training_data["Well Name"].values

logo = LeaveOneGroupOut()

f1_SVC = []

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    SVC_classifier_LOWO_C5.fit(X[train], y[train])
    pred = SVC_classifier_LOWO_C5.predict(X[test])
    sc = f1_score(y[test], pred, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_SVC.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_SVC)/(1.0*(len(f1_SVC))))


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
On average, we'll have about 2830 samples for training curves and 400 for testing curves.
Can you write Python code for it?

from sklearn import svm
SVC_classifier_learn = svm.SVC(C=5, cache_size=800, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.01, kernel='rbf',
  max_iter=-1, probability=False, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

training_data = pd.read_csv('../training_data.csv')
X = training_data.drop(['Formation', 'Well Name', 'Facies'], axis=1).values
scaler = preprocessing.StandardScaler().fit(X)
X = scaler.transform(X)
y = training_data['Facies'].values

title = "Learning Curves (SVC)"
# Learning curves with 50 iterations to get smoother mean test and train
# score curves; each time we hold 15% of the data randomly as a validation set.
# This is equivalent to leaving about 1 well out, on average (3232 minus ~2800 samples)
cv = ShuffleSplit(n_splits=50, test_size=0.15, random_state=1)

plot_learning_curve(SVC_classifier_learn, title, X, y, cv=cv, ylim=(0.45, 0.75), n_jobs=4)
plt.show()


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
SVM classifier
SImilar to the classifier in the article (but, as you will see, it uses a different kernel). We will re-import the data so as to pre-process it as in the tutorial.
Can you write Python code for it?

from sklearn import svm
SVC_classifier = svm.SVC(cache_size = 800, random_state=1)

training_data = pd.read_csv('../training_data.csv')
X = training_data.drop(['Formation', 'Well Name', 'Facies'], axis=1).values
scaler = preprocessing.StandardScaler().fit(X)
X = scaler.transform(X)
y = training_data['Facies'].values

parm_grid={'kernel': ['linear', 'rbf'],
            'C': [0.5, 1, 5, 10, 15],
            'gamma':[0.0001, 0.001, 0.01, 0.1, 1, 10]}

grid_search = GridSearchCV(SVC_classifier,
                           param_grid=parm_grid,
                           scoring = Fscorer,
                           cv=10) # Stratified K-fold with n_splits=10
                                  # For integer inputs, if the estimator is a
                                  # classifier and y is either binary or multiclass,
                                  # as in our case, StratifiedKFold is used

grid_search.fit(X, y)

print('Best score: {}'.format(grid_search.best_score_))
print('Best parameters: {}'.format(grid_search.best_params_))

grid_search.best_estimator_


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
These are the names of the 10 training wells in the Council Grove reservoir.  Data has been recruited into pseudo-well 'Recruit F9' to better represent facies 9, the Phylloid-algal bafflestone.
Before we plot the well data, let's define a color map so the facies are represented by consistent color in all the plots in this tutorial.  We also create the abbreviated facies labels, and add those to the facies_vectors dataframe.
Can you write Python code for it?

# 1=sandstone  2=c_siltstone   3=f_siltstone # 4=marine_silt_shale
#5=mudstone 6=wackestone 7=dolomite 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041', '#DC7633','#A569BD',
       '#000000', '#000080', '#2E86C1', '#AED6F1', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)
training_data.describe()


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.
The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
1. Nonmarine sandstone
2. Nonmarine coarse siltstone
3. Nonmarine fine siltstone
4. Marine siltstone and shale
5. Mudstone (limestone)
6. Wackestone (limestone)
7. Dolomite
8. Packstone-grainstone (limestone)
9. Phylloid-algal bafflestone (limestone)
These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close.  Mislabeling within these neighboring facies can be expected to occur.  The following table lists the facies, their abbreviated labels and their approximate neighbors.
Facies |Label| Adjacent Facies
:---: | :---: |:--:
1 |SS| 2
2 |CSiS| 1,3
3 |FSiS| 2
4 |SiSh| 5
5 |MS| 4,6
6 |WS| 5,7
7 |D| 6,8
8 |PS| 6,7,9
9 |BS| 7,8
Let's clean up this dataset.  The 'Well Name' and 'Formation' columns can be turned into a categorical data type.
Can you write Python code for it?

training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data['Well Name'].unique()


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables.  Looking at the count values, most values have 4149 valid values except for PE, which has 3232.  We will drop the feature vectors that don't have a valid PE entry.
Can you write Python code for it?

PE_mask = training_data['PE'].notnull().values
training_data = training_data[PE_mask]

training_data.describe()


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel.ipynb".
The first task is:
Two birds with a stone
Below we will perform grid search with stratified K-fold: http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html#sphx-glr-auto-examples-model-selection-grid-search-digits-py.
This will give us reasonable values for the more critical (for performance) classifier's parameters.
Make performance scorers
Used to evaluate training, testing, and validation performance.
Can you write Python code for it?

Fscorer = make_scorer(f1_score, average = 'micro')
Ascorer = make_scorer(accuracy_score)


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Confusion matrix and average test F1 score
Let's see how we do with predicting the actual facies, by looking at a confusion matrix. We do this by keeping the parameters from the previous section, but creating a brand new classifier. So when we fit the data, it won't have seen it before.
Can you write Python code for it?

from sklearn import svm
SVC_classifier_conf = svm.SVC(C = 100, cache_size=2400, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.01, kernel='rbf',
  max_iter=-1, probability=False, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

SVC_classifier_conf.fit(X,y)
svc_pred = SVC_classifier_conf.predict(X)

from sklearn.metrics import confusion_matrix
from classification_utilities import display_cm, display_adj_cm

conf = confusion_matrix(svc_pred, y)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)

f1_SVC = []

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    SVC_classifier_conf.fit(X[train], y[train])
    pred = SVC_classifier_conf.predict(X[test])
    sc = f1_score(y[test], pred, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_SVC.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_SVC)/(1.0*(len(f1_SVC))))


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Displaying predicted versus original facies in the training data
This is a nice display to finish up with, as it gives us a visual idea of the predicted faces where we have facies from the core observations.
The plot we will use a function from the original notebook. Let's look at the well with the lowest F1 from the previous code block, CROSS H CATTLE, and the one with the highest F1 (excluding Recruit F9), which is SHRIMPLIN.
Can you write Python code for it?

def compare_facies_plot(logs, compadre, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster1 = np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)
    cluster2 = np.repeat(np.expand_dims(logs[compadre].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=7, figsize=(9, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im1 = ax[5].imshow(cluster1, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)
    im2 = ax[6].imshow(cluster2, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[6])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im2, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-2):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')
   ...


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Let's get the average number of training samples (with the exclusion of Recruit F9). This is going to be the desired size for the folds.
Can you write Python code for it?

n_samples =  np.delete(n_samples,5,0)
ave_n = (n_samples.sum()/len(n_samples))


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Make performance scorers
Used to evaluate performance.
Can you write Python code for it?

Fscorer = make_scorer(f1_score, average = 'micro')
# Ascorer = make_scorer(accuracy_score)


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Now we extract just the feature variables we need to perform the classification.  The predictor variables are the five log values and two geologic constraining variables, and we are also using depth. We also get a vector of the facies labels that correspond to each feature vector.
Can you write Python code for it?

y = training_data['Facies'].values
print y[25:40]
print np.shape(y)

X = training_data.drop(['Formation', 'Well Name','Facies','FaciesLabels'], axis=1)
print np.shape(X)
X.describe(percentiles=[.05, .25, .50, .75, .95])

scaler = preprocessing.StandardScaler().fit(X)
X = scaler.transform(X)


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Predicting, displaying, and saving facies for blind wells
For the plot we will use a function from the original notebook.
Can you write Python code for it?

blind = pd.read_csv('validation_data_nofacies.csv')
X_blind = np.array(blind.drop(['Formation', 'Well Name'], axis=1))
X_blind = scaler.transform(X_blind)
y_pred = SVC_classifier_conf.fit(X, y).predict(X_blind)
blind['Facies'] = y_pred

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')
  ...


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
Putting it all together: crossvalidated grid search
First things first, how many samples do we have for each leave-one-well-out split?
Since this will be our validation method, it is good to select a number of k-fold splits that will use for training the same number of samples (on average) we'd have with validation.
Can you write Python code for it?

wells = training_data["Well Name"].values
logo = LeaveOneGroupOut()

n_samples =[]
for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    print well_name, 'out: ', np.shape(train)[0], 'training samples - ', np.shape(test)[0], 'test samples'
    n_samples.append(np.shape(train)[0])


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.
The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
1. Nonmarine sandstone
2. Nonmarine coarse siltstone
3. Nonmarine fine siltstone
4. Marine siltstone and shale
5. Mudstone (limestone)
6. Wackestone (limestone)
7. Dolomite
8. Packstone-grainstone (limestone)
9. Phylloid-algal bafflestone (limestone)
These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close.  Mislabeling within these neighboring facies can be expected to occur.  The following table lists the facies, their abbreviated labels and their approximate neighbors.
Facies |Label| Adjacent Facies
:---: | :---: |:--:
1 |SS| 2
2 |CSiS| 1,3
3 |FSiS| 2
4 |SiSh| 5
5 |MS| 4,6
6 |WS| 5,7
7 |D| 6,8
8 |PS| 6,7,9
9 |BS| 7,8
Let's clean up this dataset.  The 'Well Name' and 'Formation' columns can be turned into a categorical data type.
Can you write Python code for it?


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables.  Looking at the count values, most values have 4149 valid values except for PE, which has 3232.  We will drop the feature vectors that don't have a valid PE entry.
Can you write Python code for it?


I am working on the file "MandMs/Facies_classification-M&Ms_SVM_rbf_kernel_optimal.ipynb".
The first task is:
We've found in our previous notebook that the rbf kernel performs better than the linear kernel.
We will now run grid search with a reasonable range for C (based on our experience) and a large range of gamma values. The intention is to ensure we do not settle on a local minimum for gamma, and then to revisit the value of C with validation curves.
Can you write Python code for it?

parm_grid={'C': [0.1, 0.05, 0.1, 0.5, 1, 5, 10, 15, 20, 25, 50],
           'gamma':[0.00001, 0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000]}

grid_search = GridSearchCV(SVC_classifier,
                           param_grid=parm_grid,
                           scoring = Fscorer,
                           n_jobs = 7, cv = 7)

grid_search.fit(X, y)

print('Best score: {}'.format(grid_search.best_score_))
print('Best parameters: {}'.format(grid_search.best_params_))

grid_search.best_estimator_


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
Comments
Using the average F1 score from the leave-one-well out cross validation as a metric, the majority voting is superior to the individual classifiers, including the pre-tuned Random Forest from the leading submission. However, the Random Forest in the official leading submission was trained using additional new features engineered by George, and outperforms our majority voting classifier, with an F1 score of 0.580 against our 0.579. A clear indication, in our view, that the feature engineering is a key element to achieve the best possible prediction.
Predicting, displaying, and saving facies for blind wells
Can you write Python code for it?

blind = pd.read_csv('validation_data_nofacies.csv')
X_blind = np.array(blind.drop(['Formation', 'Well Name'], axis=1))
X_blind = scaler.transform(X_blind)
y_pred = eclf_cv.fit(X, y).predict(X_blind)
blind['Facies'] = y_pred

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')

   ...


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
Make performance scorers
Used to evaluate performance.
Can you write Python code for it?

Fscorer = make_scorer(f1_score, average = 'micro')


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
Plurality voting classifier (multi-class majority voting)
We will use a weighted approach, where the weights are somewhat arbitrary, but their proportion is based on the average f1 score of the individual classifiers.
Can you write Python code for it?

from sklearn.ensemble import VotingClassifier

eclf_cv = VotingClassifier(estimators=[
        ('SVC', SVC_classifier), ('MLP', mlp_classifier), ('ET', ET_classifier)],
                        voting='soft', weights=[0.3,0.33,0.37])


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
Pre-tuned SVM classifier classifier and leave one well out average F1 score
This is the Support Vector Machine classifier from our first submission.
Can you write Python code for it?

from sklearn import svm
SVC_classifier = svm.SVC(C = 100, cache_size=2400, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.01, kernel='rbf',
  max_iter=-1, probability=True, random_state=49, shrinking=True,
  tol=0.001, verbose=False)

f1_svc = []

wells = training_data["Well Name"].values
logo = LeaveOneGroupOut()

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    SVC_classifier.fit(X[train], y[train])
    pred_svc = SVC_classifier.predict(X[test])
    sc = f1_score(y[test], pred_svc, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_svc.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_svc)/(1.0*(len(f1_svc))))


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
Pre-tuned multi-layer perceptron classifier and average F1 score
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
mlp_classifier = MLPClassifier(activation='logistic', alpha=0.01, batch_size='auto',
       beta_1=0.9, beta_2=0.999, early_stopping=False, epsilon=1e-08,
       hidden_layer_sizes=(100,), learning_rate='adaptive',
       learning_rate_init=0.001, max_iter=1000, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=49, shuffle=True,
       solver='adam', tol=0.0001, validation_fraction=0.1, verbose=False,
       warm_start=False)

f1_mlp = []

wells = training_data["Well Name"].values
logo = LeaveOneGroupOut()

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    mlp_classifier.fit(X[train], y[train])
    pred_mlp = mlp_classifier.predict(X[test])
    sc = f1_score(y[test], pred_mlp, labels = np.arange(10), average = 'micro')
    print("{:>20s}  {:.3f}".format(well_name, sc))
    f1_mlp.append(sc)

print "-Average leave-one-well-out F1 Score: %6f" % (sum(f1_mlp)/(1.0*(len(f1_mlp))))


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.
The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
1. Nonmarine sandstone
2. Nonmarine coarse siltstone
3. Nonmarine fine siltstone
4. Marine siltstone and shale
5. Mudstone (limestone)
6. Wackestone (limestone)
7. Dolomite
8. Packstone-grainstone (limestone)
9. Phylloid-algal bafflestone (limestone)
These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close.  Mislabeling within these neighboring facies can be expected to occur.  The following table lists the facies, their abbreviated labels and their approximate neighbors.
Facies |Label| Adjacent Facies
:---: | :---: |:--:
1 |SS| 2
2 |CSiS| 1,3
3 |FSiS| 2
4 |SiSh| 5
5 |MS| 4,6
6 |WS| 5,7
7 |D| 6,8
8 |PS| 6,7,9
9 |BS| 7,8
Let's clean up this dataset.  The 'Well Name' and 'Formation' columns can be turned into a categorical data type.
Can you write Python code for it?


I am working on the file "MandMs/Facies_classification-M&Ms_plurality_voting_classifier.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables.  Looking at the count values, most values have 4149 valid values except for PE, which has 3232.  We will drop the feature vectors that don't have a valid PE entry.
Can you write Python code for it?


I am working on the file "Mandelbrot Set.ipynb".
The first task is:
By reshaping r2 into r2_T, we make it a vertical column.  Then we add.  The result is a 4x4 matrix, from (-1, -1) upper left, to (1, 1) lower right.
Can you write Python code for it?


I am working on the file "Mandelbrot Set.ipynb".
The first task is:
Lets look at a smaller slice of the complex plane to being with.  This rectangle of complex numbers starts from (-1, -1j) and ends at (1, 1j), dividing both the real and imaginary axes into three intervals marked by four tick marks.
Can you write Python code for it?

r1 = np.linspace(-1, 1, 4, dtype=np.float32)
r2 = np.linspace(-1, 1, 4, dtype=np.float32)
r2_T = (r2 * 1j).reshape(4,1)
r2_T


I am working on the file "Mandelbrot Set.ipynb".
The first task is:
Notice how we were able to add rectangular arrays r1 and r2_T in one line, with no loops.
Can you write Python code for it?

def mandelbrot_set(xmin,xmax,ymin,ymax,width,height,maxiter):
    r1 = np.linspace(xmin, xmax, width, dtype=np.float32)
    r2 = np.linspace(ymin, ymax, height, dtype=np.float32)
    c = r1 + (r2 * 1j).reshape(width,1)  # x row + each y column
    n3 = mandelbrot_numpy(c, maxiter)
    return (r1,r2,n3)


I am working on the file "Mandelbrot Set.ipynb".
The first task is:
The Python standard library has a timeit module.  Jupyter Notebooks add a layer of %magic by giving us the %timeit directive.  We're able to see how long the operation takes.
Can you write Python code for it?


I am working on the file "Mandelbrot Set.ipynb".
The first task is:
We are all set! Let's calculate it:
Can you write Python code for it?

# Width and height of the image
(width, height) = (1024, 768)

# The complex plane to map
(zmin, zmax) = (-2.2-1.5j, +1.2+1.5j)

# The Mandelbrot set
mandel = mandel(width, height, zmin, zmax)


I am working on the file "Mandelbrot Set.ipynb".
The first task is:
We now have the array we want to plot in result.
Can you write Python code for it?

%matplotlib notebook
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
im = ax.imshow(result, extent=[-2.0,0.5,-1.25,1.25])
fig.colorbar(im)
plt.show()


I am working on the file "Mandelbrot_CPU_Example.ipynb".
The first task is:
Next we make a function to create the fracal.
It will fill a two-dimensional integer array data with the number of iterations before
the sequence diverged. Points inside the Mandelbrot set will have the value max_iters.
Can you write Python code for it?

@jit(nopython=True)
def make_fractal(xmin, xmax, ymin, ymax, data, max_iters):
    height, width = data.shape
    dx = (xmax - xmin) / width
    dy = (ymax - ymin) / height
    for i in range(width):
        x = xmin + dx * (i + 0.5)
        for j in range(height):
            y = ymin + dy * (j + 0.5)
            data[j,i] = mandel(x, y, max_iters)
    return data


I am working on the file "Mandelbrot_CPU_Example.ipynb".
The first task is:
Using IPython widgets and the ”interact_manual” command, we can make this more interactive. Note that most of the delay is the time to send the generate the graphical image and send it from the server. As you zoom in, round-off error in the floating point precision of the math will cause numerical artifacts.
Can you write Python code for it?

interact_manual(calculate_plot, x=FloatText(-0.003001005), y=FloatText(0.64400092), logscale=(-8,0,0.1))


I am working on the file "Mandelbrot_Intro.ipynb".
The first task is:
Interesting! To explore this more, lets make a function that generates the sequence itself.
We'll supply two numbers: the value of the constant $c$ and the number of terms in the sequence.
Can you write Python code for it?

def make_sequence(c, n):
    x = 0.0
    sequence = [x]
    for i in range(n):
        x = f(x, c)
        sequence.append(x)
    return sequence

make_sequence(1, 10)

make_sequence(-1, 10)

make_sequence(-2, 10)

make_sequence(-2.01, 10)

make_sequence(0.25, 10)


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Age - Analysis | Graph

Can you write Python code for it?

printA = "Youngest Passenger in the passenger list was " + str(titAge['Age'].min()) + " years of age." \
    + "\n" + "Oldest Passenger in the passenger list was " + str(titAge['Age'].max()) + " years of age." \
    + "\n" + "Mean of Passengers ages in the passenger list is " + str(titAge['Age'].mean()) + " years of age."

print(printA)


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Distrbution of Age in passenger population
Can you write Python code for it?


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Distribution of Age in pClass population
Can you write Python code for it?

gCPS = sns.FacetGrid(titAge,hue='Pclass', aspect=4, hue_order=[1,2,3])
gCPS.map(sns.kdeplot,'Age', shade=True)

gCPS.set(xlim=(0,titAge['Age'].max()))
gCPS.add_legend()


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Distribution of Genders in pClass populations
Can you write Python code for it?

gGCSC= sns.factorplot('Pclass',order=[1,2,3], data=dTitTrain_DF, hue='Sex', kind='count')

gGCSC.despine(left=True)
gGCSC.set_ylabels("count of passengers")


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Distribution of Passengers
Gender -  Analysis | Graph

Distribution of Genders in Passenger Population
Can you write Python code for it?

printG  =  "Men account for " + str(ACmenDataCount) +  " and "  + "Women account for " + str(ACwomenDataCount) + " (Total Passengers: " + str(dTitTrain_DF.count()['Age']) + ")"
print(printG)

gSSC = sns.factorplot('Sex', data=dTitTrain_DF, kind='count')
gSSC.despine(left=True)
gSSC.set_ylabels("count of passengers")


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Distribution of passengers into adult and children age groups (Child = less than 21 years of age)
Reference:
Source: http://history.stackexchange.com/questions/17481/what-was-the-age-of-majority-in-1900-united-states

By the common law the age of majority is fixed at twenty-one years for both sexes, and, in the absence of any statute to >the contrary, every person under that age, whether male or female, is an infant. (21)

-- The American and English Encyclopedia of Law, Garland and McGeehee, 1900

By the common law, every person is, technically, an infant, until he is twenty-one years old; and, in legal presumption, is >not of sufficient discretion to contract an obligation at an earlier age.

-- Institutes of the Lawes of England by Coke (1628-1644). The laws on infants are at 171b.
Can you write Python code for it?

# splits passengers into 3 categories (male of female if considered adult, and child if below 21 of age)

def minorOrAdult(passenger):
    age, sex = passenger

    if age < 21:
        return 'child'
    else:
        return sex

# adds new column to dataframe that distinguishes a passenger as a child or an adult

dTitTrain_DF['PersonStatus'] = dTitTrain_DF[['Age', 'Sex']].apply(minorOrAdult, axis=1)

dTitTrain_DF['PersonStatus'].value_counts()


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Locational: Disembark Analysis | Graph

Can you write Python code for it?

sns.factorplot('Embarked', order=['C','Q','S'], data=dTitTrain_DF, hue='Pclass', kind='count', hue_order=[1,2,3])

# titCherbourg
# titQueenstown
# titSouthampton

print("Total:")
print("Cherbourg: " + str(titCherbourg.count()['Embarked']))
print("Queenstown: " + str(titQueenstown.count()['Embarked']))
print("Southampton: " + str(titSouthampton.count()['Embarked']))

print("")

print("Cherbourg: ")
print("Pclass 1 - " + str(titCherbourg[titCherbourg.Pclass == 1].count()['Embarked']))
print("Pclass 2 - " + str(titCherbourg[titCherbourg.Pclass == 2].count()['Embarked']))
print("Pclass 3 - " + str(titCherbourg[titCherbourg.Pclass == 3].count()['Embarked']))

print("")

print("Queenstown: ")
print("Pclass 1 - " + str(titQueenstown[titQueenstown.Pclass == 1].count()['Embarked']))
print("Pclass 2 - " + str(titQueenstown[titQueenstown.Pclass == 2].count()['Embarked']))
print("Pclass 3 - " + str(titQueenstown[titQueenstown.Pclass == 3].count()['Embarked']))

print("")

print("Southampton: ")
print("Pclass 1 - " + str(titSouthampton[titSouthampton.Pclass == 1].count()['Embarked']))
print("Pclass 2 - " + str(titSouthampton[titSouthampton.Pclass == 2].count()['Embarked']))
print("Pclass 3 - " + str(titSouthampton[titSouthampton.Pclass == 3].count()['Embarked']))


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Process CSV - Generation of Estimation Survival Table
Can you write Python code for it?

# Determining better odds which will be compared to test group  (First comparison - Pclass and age group)

import csv

# # Manual - Age Group and gender adult with highest above 49%
# print(define_pClassChildProb(dTitTrain_DF, 1))
# print(define_pClassAdultProb(dTitTrain_DF, 1, 'female'))

# print(define_pClassChildProb(dTitTrain_DF, 2))
# print(define_pClassAdultProb(dTitTrain_DF, 2, 'female'))

# print(define_pClassAdultProb(dTitTrain_DF, 3, 'female'))

# #sibsp and parch

test_file = open('test.csv', 'rb')
test_file_object = csv.reader(test_file)
header = test_file_object.next()

prediction_file = open("genderPclassbasedmodel.csv", "wb")
prediction_file_object = csv.writer(prediction_file)

prediction_file_object.writerow(["PassengerId", "Survived"])
for row in test_file_object:       # For each row in test.csv
    weight = 0.0

    if row[1] == 1:
        weight = weight + 9
    elif row[1] == 2:
        weight = weight + 8
    else:
        weight = 5

    if row[3] == 'female':
        weight = weight + 8
    else:
        weight = weight + 2

    if row[4] < 21:
        # child
        weight = weight + 6
    else:
        # adult
        weight = weight + 5

    aFam = row[5] + row[6]

    if aFam > 0:
        weight = weight + 5

    else:
        weight = weight + 3

    weightScore = weight/40.0

    print(str(weightScore))

    if(weight >= .5):
        prediction_file_object.writerow([row[0],'1'])
    else:
        prediction_file_object.writerow([row[0],'0'])

#prediction_file_object.writerow([row[0],'1'])
#prediction_file_object.writerow([row[0],'0'])

test_file.close()
prediction_file.close()


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Surivival Graph Comparison
Survival Count (Overall)

Can you write Python code for it?

# Survivors Overall

gSOA = sns.factorplot('Survivor', data=dTitTrain_DF, kind='count')

gSOA.despine(left=True)
gSOA.set_ylabels("count of passengers")

print("Survivor: " + str(dTitTrain_DF[dTitTrain_DF.Survivor == "Yes"].count()['Survivor']))
print("Non-Survivor: " + str(dTitTrain_DF[dTitTrain_DF.Survivor == "No"].count()['Survivor']))


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Survival By Pclass and Age Group (Adult (Male / Female) / Child)

Can you write Python code for it?

#Determine probability of survival of children in a given Pclass

def define_pClassChildProb(dataFrameIN, numClass):

    ChildDF = dataFrameIN[dataFrameIN.Pclass == numClass][dataFrameIN.PersonStatus == 'child']
    ChildSurvived = dataFrameIN[dataFrameIN.Pclass == numClass][dataFrameIN.PersonStatus == 'child'][dataFrameIN.Survivor == 'Yes']

    totalCChild = ChildDF.count()['PassengerId'].astype(float)
    CChildSurvived = ChildSurvived.count()['PassengerId'].astype(float)

    return CChildSurvived/totalCChild

def define_pClassAdultProb(dataFrameIN, numClass, sex):

    AdultDF = dataFrameIN[dataFrameIN.Pclass == numClass][dataFrameIN.PersonStatus == sex]
    AdultSurvived = dataFrameIN[dataFrameIN.Pclass == numClass][dataFrameIN.PersonStatus == sex][dataFrameIN.Survivor == 'Yes']

    totalCAdult = AdultDF.count()['PassengerId'].astype(float)
    CAdultSurvived = AdultSurvived.count()['PassengerId'].astype(float)

    return CAdultSurvived/totalCAdult

print("PClass 1 Survival Child: ")
print(define_pClassChildProb(dTitTrain_DF, 1))

print("PClass 1 Survival Female: ")
print(define_pClassAdultProb(dTitTrain_DF, 1, 'female'))

print("PClass 1 Survival Male: ")
print(define_pClassAdultProb(dTitTrain_DF, 1, 'male'))

print("-----------")

print("PClass 2 Survival Child: ")
print(define_pClassChildProb(dTitTrain_DF, 2))

print("PClass 2 Survival Female: ")
print(define_pClassAdultProb(dTitTrain_DF, 2, 'female'))

print("PClass 2 Survival Male: ")
print(define_pClassAdultProb(dTitTrain_DF, 2, 'male'))

print("-----------")

print("PClass 3 Survival Child: ")
print(define_pClassChildProb(dTitTrain_DF, 3))

print("PClass 3 Survival Female: ")
print(define_pClassAdultProb(dTitTrain_DF, 3, 'female'))

print("PClass 3 Survival Male: ")
print(define_pClassAdultProb(dTitTrain_DF, 3, 'male'))

sns.factorplot("Pclass", "Survived", hue='PersonStatus',order=[1,2,3], data=dTitTrain_DF, kind='point')


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Survival Gender by Age Distribution

Can you write Python code for it?

#sns.lmplot('Age', 'Survived', hue='Sex' ,data=dTitanic_DF)
pAGSB = sns.boxplot(data=dTitTrain_DF.dropna(subset = ['Age']), x= 'Sex', y= 'Age', hue='Survivor')
pAGSB.set(title='Age by Gender and Survival - Box Plot')
pAGSB.legend(bbox_to_anchor=(1.05, .7), loc=2, title = 'Survived',borderaxespad=0.)


I am working on the file "Manual-Titanic/.ipynb_checkpoints/Kaggle - Titanic - Manual-checkpoint.ipynb".
The first task is:
Survival by Pclass

Can you write Python code for it?

# Determines the probability of survival for a given Pclass

def define_pClassProb(dataFrameIN, numClass):
    classEntries = dataFrameIN[dataFrameIN.Pclass == numClass]
    sClassEntries = classEntries[classEntries.Survived == 1]

    cClassEntries = (classEntries.count(numeric_only=True)['Pclass']).astype(float)
    cSClassEntries = (sClassEntries.count(numeric_only=True)['Pclass']).astype(float)

    return (cSClassEntries/cClassEntries)

print("Class 1 Survivality: ")
print(define_pClassProb(dTitTrain_DF, 1))

print("Class 2 Survivality: ")
print(define_pClassProb(dTitTrain_DF, 2))

print("Class 3 Survivality: ")
print(define_pClassProb(dTitTrain_DF, 3))

gCS = sns.factorplot("Pclass", "Survived",order=[1,2,3],data=dTitTrain_DF, kind="bar", size=5)

gCS.despine(left=True)
gCS.set_ylabels("survival probability")

print("Class 1 Survivality: ")
print(define_pClassProb(dTitTrain_DF, 1))

print("Class 2 Survivality: ")
print(define_pClassProb(dTitTrain_DF, 2))

print("Class 3 Survivality: ")
print(define_pClassProb(dTitTrain_DF, 3))

sns.factorplot("Pclass", "Survived",order=[1,2,3], data=dTitTrain_DF, kind='point')


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Alone or Family

Can you write Python code for it?

sns.factorplot('SoloOrFamily', data=dTitTrain_DF, kind='count')

print("Alone: " + str(dTitTrain_DF[dTitTrain_DF.SoloOrFamily == "Alone"].count()['SoloOrFamily']))
print("Family: " + str(dTitTrain_DF[dTitTrain_DF.SoloOrFamily == "Family"].count()['SoloOrFamily']))


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Distrbution of Age in passenger population
Can you write Python code for it?


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Distribution of Genders in pClass populations
Can you write Python code for it?


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Distribution of child and adult (male and female) age groups by age
Can you write Python code for it?

gACPS = sns.FacetGrid(dTitTrain_DF, hue='PersonStatus', aspect=4, hue_order=['child', 'male', 'female'])
gACPS.map(sns.kdeplot,'Age', shade=True)

gACPS.set(xlim=(0,titAge['Age'].max()))
gACPS.add_legend()


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Distribution of child and adult (male and female) by pClass
Can you write Python code for it?

gGAC= sns.factorplot('Pclass', order=[1,2,3], data=dTitTrain_DF, hue='PersonStatus', kind='count',hue_order=['child','male','female'])

gGAC.despine(left=True)
gGAC.set_ylabels("count of passengers")


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Distribution of passengers into adult and children age groups (Child = less than 21 years of age)
Reference:
Source: http://history.stackexchange.com/questions/17481/what-was-the-age-of-majority-in-1900-united-states

By the common law the age of majority is fixed at twenty-one years for both sexes, and, in the absence of any statute to >the contrary, every person under that age, whether male or female, is an infant. (21)

-- The American and English Encyclopedia of Law, Garland and McGeehee, 1900

By the common law, every person is, technically, an infant, until he is twenty-one years old; and, in legal presumption, is >not of sufficient discretion to contract an obligation at an earlier age.

-- Institutes of the Lawes of England by Coke (1628-1644). The laws on infants are at 171b.
Can you write Python code for it?


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Locational: Cabin Analysis | Graph

Can you write Python code for it?

def prepareDeckGraph(titDecksDF):
    deckLevels = []

    for level in titDecksDF:
        deckLevels.append(level[0])

    T_DF = pd.DataFrame(deckLevels)

    T_DF.columns = ['Cabin']

    T_DF = T_DF[T_DF.Cabin != 'T']

    return T_DF

gTD_DF = prepareDeckGraph(titDecks)
sns.factorplot('Cabin', order=['A','B','C','D','E','F','G'], data=gTD_DF, kind='count')

print("A: " + str(deckCount(titDecks, "A")))
print("B: " + str(deckCount(titDecks, "B")))
print("C: " + str(deckCount(titDecks, "C")))
print("D: " + str(deckCount(titDecks, "D")))
print("E: " + str(deckCount(titDecks, "E")))
print("F: " + str(deckCount(titDecks, "F")))
print("G: " + str(deckCount(titDecks, "G")))


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Locational: Disembark Analysis | Graph

Can you write Python code for it?


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Process CSV - Generation of Estimation Survival Table
Can you write Python code for it?

# Determining better odds which will be compared to test group  (First comparison - Pclass and age group)

import csv

# # Manual - Age Group and gender adult with highest above 49%
# print(define_pClassChildProb(dTitTrain_DF, 1))
# print(define_pClassAdultProb(dTitTrain_DF, 1, 'female'))

# print(define_pClassChildProb(dTitTrain_DF, 2))
# print(define_pClassAdultProb(dTitTrain_DF, 2, 'female'))

# print(define_pClassAdultProb(dTitTrain_DF, 3, 'female'))

# #sibsp and parch

test_file = open('test.csv', 'rb')
test_file_object = csv.reader(test_file)
header = test_file_object.next()

prediction_file = open("genderPclassbasedmodel.csv", "wb")
prediction_file_object = csv.writer(prediction_file)

prediction_file_object.writerow(["PassengerId", "Survived"])
for row in test_file_object:       # For each row in test.csv
    weight = 0.0

    if row[1] == 1:
        weight = weight + 9
    elif row[1] == 2:
        weight = weight + 8
    else:
        weight = 5

    if row[3] == 'female':
        weight = weight + 8
    else:
        weight = weight + 2

    if row[4] < 21:
        # child
        weight = weight + 6
    else:
        # adult
        weight = weight + 5

    aFam = row[5] + row[6]

    if aFam > 0:
        weight = weight + 5

    else:
        weight = weight + 3

    weightScore = weight/40.0

    print(str(weightScore))

    if(weightScore >= .5):
        prediction_file_object.writerow([row[0],'1'])
    else:
        prediction_file_object.writerow([row[0],'0'])

#prediction_file_object.writerow([row[0],'1'])
#prediction_file_object.writerow([row[0],'0'])

test_file.close()
prediction_file.close()


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Survival By Pclass and Age Group (Adult (Male / Female) / Child)

Can you write Python code for it?


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Survival by Age Distribution

Can you write Python code for it?

#sns.lmplot('Age', 'Survived', data=dTitTrain_DF)
pSBA = sns.boxplot(data=dTitTrain_DF, x='Survived', y='Age')
pSBA.set(title='Age Distribution by Survival',
            xlabel = 'Survival',
            ylabel = 'Age Distrobution',
            xticklabels = ['Died', 'Survived'])


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Survival by Alone or with Family

Can you write Python code for it?

# Using Solo or family column created earlier in passenger distributions section created a separate dataframes for traveling
#alone and with family passengers

familyPass = dTitTrain_DF[dTitTrain_DF['SoloOrFamily'] == "Family"]
alonePass = dTitTrain_DF[dTitTrain_DF['SoloOrFamily'] == "Alone"]

# Creates a list of surviving family and alone passengers

AFamilyPass = familyPass[familyPass.Survivor == "Yes"]
AAlonePass = alonePass[alonePass.Survivor == "Yes"]

# Determines the probability of survival for passengers that traveled alone and with family

pAF = float(AFamilyPass['SoloOrFamily'].count()) / float(familyPass['SoloOrFamily'].count())
pAA = float(AAlonePass['SoloOrFamily'].count()) / float(alonePass['SoloOrFamily'].count())

print("Probability of Survival being with Family: ")
print(pAF)

print("")

print("Probability of Survival being alone: ")
print(pAA)

gSSP = sns.factorplot("SoloOrFamily", "Survived", data=dTitTrain_DF, kind="bar", size=5)

gSSP.despine(left=True)
gSSP.set_ylabels("survival probability")


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Survival by Gender

Can you write Python code for it?

# Series probability - access probability of survived in men and women
menProb = ACmenData.groupby('Sex').Survived.mean()
womenProb = ACwomenData.groupby('Sex').Survived.mean()

menPercent = menProb[0]*100
womenPercent = womenProb[0]*100

print("Men Survivalbility: ")
print(menProb[0])

print("Women Survivalbility: ")
print(womenProb[0])

gSSP = sns.factorplot("Sex", "Survived", data=dTitTrain_DF, kind="bar", size=5)

gSSP.despine(left=True)
gSSP.set_ylabels("survival probability")


I am working on the file "Manual-Titanic/Kaggle - Titanic - Manual.ipynb".
The first task is:
Survival pClass by Age Distribution

Can you write Python code for it?

#sns.lmplot('Age', 'Survived',hue='Pclass', data=dTitanic_DF, hue_order=[1,2,3])
pACSB = sns.boxplot(data = dTitTrain_DF.dropna(subset = ['Age']).sort_values('Pclass'), x='Pclass', y='Age', hue='Survivor')
pACSB.set(title='Age by Class and Survival - Box Plot', xlabel='Pclass')
pACSB.legend(bbox_to_anchor=(1.05, .7), loc=2, title = 'Survived',borderaxespad=0.)


I am working on the file "Manual_cmds.ipynb".
The first task is:
Reading a housekeeping value has this form:
fpe1.house_keeping["analogue"]["parameter_name"]
Here's a couple of sample reads of housekeeping values:
Can you write Python code for it?


I am working on the file "Manual_cmds.ipynb".
The first task is:
Useful Commands:
ping()
fpe1.cmd_start_frames() # Starts frame generation.
fpe1.cmd_stop_frames() # Stops frame generation.
fpe1.cmd_camrst # Don't know how to work this. As-is, it fails.
fpe1.cmd_cam_status() # Returns the camera status register values.
fpe1.cmd_version() # Returns ObsSim version info.
fpe1.house_keeping # Returns a set of HK data in alphabetical order, in engineering units, without frames running. This includes all the FPGA digital housekeeping values.
fpe1.house_keeping["analogue"] #Returns only the analog values of the housekeeping set.
{fpe1.cmd_cam_hsk() # Returns raw, un-parsed housekeeping data, two samples per word (decimal), mostly useless here.}
check_house_keeping_voltages(fpe1, tolerance=0.05) # Returns True if standard set of supply voltages are in tolerance.
If you plan on setting operating parameters (DACs), run this cell:
Can you write Python code for it?

from tessfpe.data.operating_parameters import operating_parameters
operating_parameters["heater_1_current"]


I am working on the file "Manual_cmds.ipynb".
The first task is:
Workspace:
Can you write Python code for it?

operating_parameters["ccd1_output_drain_a_offset"]
#operating_parameters["ccd1_reset_drain"]

fpe1.ops.ccd1_reset_drain = 15
fpe1.ops.ccd1_output_drain_a_offset = 10
fpe1.ops.send()
fpe1.house_keeping["analogue"]["ccd1_output_drain_a"]

#operating_parameters["ccd1_reset_high"]
operating_parameters['ccd1_reset_low_offset']

fpe1.ops.ccd1_reset_high = -10.3
fpe1.ops.ccd1_reset_low_offset = -9.9
fpe1.ops.send()
fpe1.house_keeping["analogue"]["ccd1_reset_low"]

fpe1.cmd_start_frames() # Starts frame generation.

fpe1.cmd_stop_frames() # Stops frame generation.

from tessfpe.data.housekeeping_channels import housekeeping_channels

from tessfpe.data.housekeeping_channels import housekeeping_channel_memory_map

print fpe1.house_keeping

print fpe1.house_keeping["analogue"]

from numpy import var
samples=100
from tessfpe.data.housekeeping_channels import housekeeping_channels
# We make sample_data a dictionary and each value will be a set of HK data, with key = sample_name.
sample_data = {}

# For later:
signal_names = []
signal_values = []
signal_data = {}
variance_values = {}

#my_dict["new key"] = "New value"

for i in range(samples):
    # Get a new set of HK values
    house_keeping_values = fpe1.house_keeping["analogue"]
    data_values = house_keeping_values.values()
    # Add the new HK values to the sample_data dictionary:
    sample_number = "sample_" + str(i)
    sample_data[sample_number] = data_values

# Get the signal names for use later
signal_names = house_keeping_values.keys()

"""Assign the set of all HK values of the same signal (e.g. substrate_1)
to the dictionary 'signal_data'"""

for k in range(len(signal_names)):
    # Build the list 'signal_values' for this signal:
    for i in range(samples):
        sample_number = "sample_" + str(i)
        signal_values.append(sample_data[sample_number][k])
    # Add signal_values to the signal_data dictionary:
    signal_data[signal_names[k]] = signal_values
    signal_values = []

""" Now get the variance of each of the 'signal_values' in the...


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
$\implies$ We obtained the same result, OK our function works!
More comparison
Can you write Python code for it?

def test_SHA224():
    x = random_string()
    return hash_SHA224(x) == true_hash_SHA224(x)

%timeit test_SHA224()

def test_SHA512():
    x = random_string()
    return hash_SHA512(x) == true_hash_SHA512(x)

%timeit test_SHA512()

def test_SHA384():
    x = random_string()
    return hash_SHA384(x) == true_hash_SHA384(x)

%timeit test_SHA384()


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
$\implies$ We obtained the same result, OK our function works!
SHA-384
As said on the Wikipedia page:

SHA-384 is identical to SHA-512, except that:
- the initial hash values h0 through h7 are different (taken from the 9th through 16th primes), and
- the output is constructed by omitting h6 and h7.

The SHA384 class
Can you write Python code for it?

class SHA384(Hash):
    """SHA384 hashing, see https://en.wikipedia.org/wiki/SHA-2#Pseudocode."""

    def __init__(self):
        self.name        = "SHA384"
        self.byteorder   = 'big'
        self.block_size  = 96
        self.digest_size = 48
        # Note 2: For each round, there is one round constant k[i] and one entry in the message schedule array w[i], 0 ≤ i ≤ 79
        # Note 3: The compression function uses 8 working variables, a through h
        # Note 4: Big-endian convention is used when expressing the constants in this pseudocode

        # Initialize hash values:
        # (The second 64 bits of the fractional parts of the square roots of the first 9th through 16th primes 23..53)
        h0 = 0xcbbb9d5dc1059ed8
        h1 = 0x629a292a367cd507
        h2 = 0x9159015a3070dd17
        h3 = 0x152fecd8f70e5939
        h4 = 0x67332667ffc00b31
        h5 = 0x8eb44a8768581511
        h6 = 0xdb0c2e0d64f98fa7
        h7 = 0x47b5481dbefa4fa4

        # Initialize array of round constants:
        # (first 64 bits of the fractional parts of the cube roots of the first 80 primes 2..409):
        self.k = [
            0x428a2f98d728ae22, 0x7137449123ef65cd, 0xb5c0fbcfec4d3b2f, 0xe9b5dba58189dbbc, 0x3956c25bf348b538,
            0x59f111f1b605d019, 0x923f82a4af194f9b, 0xab1c5ed5da6d8118, 0xd807aa98a3030242, 0x12835b0145706fbe,
            0x243185be4ee4b28c, 0x550c7dc3d5ffb4e2, 0x72be5d74f27b896f, 0x80deb1fe3b1696b1, 0x9bdc06a725c71235,
            0xc19bf174cf692694, 0xe49b69c19ef14ad2, 0xefbe4786384f25e3, 0x0fc19dc68b8cd5b5, 0x240ca1cc77ac9c65,
            0x2de92c6f592b0275, 0x4a7484aa6ea6e483, 0x5cb0a9dcbd41fbd4, 0x76f988da831153b5, 0x983e5152ee66dfab,
            0xa831c66d2db43210, 0xb00327c898fb213f, 0xbf597fc7beef0ee4, 0xc6e00bf33da88fc2, 0xd5a79147930aa725,
            0x06ca6351e003826f, 0x142929670a0e6e70, 0x27b70a8546d22ffc, 0x2e1b21385c26c926, 0x4d2c6dfc5ac42aed,
            0x53380d139d95b3df, 0x650a73548baf63de, 0x766a0abb3c77b2a8, 0x81c2c92e47edaee6, 0x92722c851482353b,
...


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
$\implies$ We obtained the same result, OK our function works!
SHA-512
As said on the Wikipedia page:

SHA-512 is identical in structure to SHA-256, but:

the message is broken into 1024-bit chunks,
the initial hash values and round constants are extended to 64 bits,
there are 80 rounds instead of 64,
the message schedule array w has 80 64-bit words instead of 64 32-bit words,
to extend the message schedule array w, the loop is from 16 to 79 instead of from 16 to 63,
the round constants are based on the first 80 primes 2..409,
the word size used for calculations is 64 bits long,
the appended length of the message (before pre-processing), in bits, is a 128-bit big-endian integer, and
the shift and rotate amounts used are different.

Useful functions the SHA-512 algorithm
This is exactly like for SHA-256, except we work with 64-bits words instead of 32-bits.
Can you write Python code for it?

def leftrotate_64(x, c):
    """ Left rotate the number x by c bytes, for 64-bits numbers."""
    x &= 0xFFFFFFFFFFFFFFFF
    return ((x << c) | (x >> (64 - c))) & 0xFFFFFFFFFFFFFFFF

def rightrotate_64(x, c):
    """ Right rotate the number x by c bytes, for 64-bits numbers."""
    x &= 0xFFFFFFFFFFFFFFFF
    return ((x >> c) | (x << (64 - c))) & 0xFFFFFFFFFFFFFFFF


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
$\implies$ We obtained the same result, OK our function works!
Trying 1000 random examples
On a small sentence:
Can you write Python code for it?

hash_MD5("My name is Zorro !")

h = hashlib.md5()
h.update(b"My name is Zorro !")
h.hexdigest()


hash_SHA1("My name is Zorro !")

h = hashlib.sha1()
h.update(b"My name is Zorro !")
h.hexdigest()


hash_SHA2("My name is Zorro !")

h = hashlib.sha256()
h.update(b"My name is Zorro !")
h.hexdigest()


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
A less stupid check on MD5
Let try the example from MD5 Wikipedia page :
Can you write Python code for it?

hash_MD5("The quick brown fox jumps over the lazy dog")
assert hash_MD5("The quick brown fox jumps over the lazy dog") == '9e107d9d372bb6826bd81d3542a419d6'


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
A less stupid check on SHA-1
Let try the example from SHA-1 Wikipedia page :
Can you write Python code for it?

hash_SHA1("The quick brown fox jumps over the lazy dog")
assert hash_SHA1("The quick brown fox jumps over the lazy dog") == '2fd4e1c67a2d28fced849ee1bb76e7391b93eb12'


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
As SHA-1 plays with big-endian and little-endian integers, and at the end it requires a leftshift to combine the 5 hash pieces into one.
Can you write Python code for it?

def leftshift(x, c):
    """ Left shift the number x by c bytes."""
    return x << c


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
As SHA-2 plays with big-endian and little-endian integers, and at the end it requires a leftshift to combine the 5 hash pieces into one.
Can you write Python code for it?

def leftshift(x, c):
    """ Left shift the number x by c bytes."""
    return x << c

def rightshift(x, c):
    """ Right shift the number x by c bytes."""
    return x >> c


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Check on SHA-2
Let try the example from SHA-2 Wikipedia page :
Can you write Python code for it?

hash_SHA2("The quick brown fox jumps over the lazy dog")
assert hash_SHA2("The quick brown fox jumps over the lazy dog") == 'd7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592'


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Checking the the hashlib module in Python standard library
Can you write Python code for it?


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Checks on SHA-224
Let try the example from SHA-2 Wikipedia page :
Can you write Python code for it?

def true_hash_SHA224(data):
    h = hashlib.sha224()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()

hash_SHA224("The quick brown fox jumps over the lazy dog")
assert hash_SHA224("The quick brown fox jumps over the lazy dog") == '730e109bd7a8a32b1cb9d9a09aa2325d2430587ddbc0c38bad911525'
assert hash_SHA224("The quick brown fox jumps over the lazy dog") == true_hash_SHA224("The quick brown fox jumps over the lazy dog")


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Checks on SHA-384
Let try the example from SHA-2 Wikipedia page :
Can you write Python code for it?

def true_hash_SHA384(data):
    h = hashlib.sha384()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()

hash_SHA384("The quick brown fox jumps over the lazy dog")
assert hash_SHA384("The quick brown fox jumps over the lazy dog") == true_hash_SHA384("The quick brown fox jumps over the lazy dog")


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Comparison : MD5 vs SHA-1 vs SHA-2
It can be interesting to compare each hash functions, with respect to its time complexity.
Can you write Python code for it?

def test_MD5():
    x = random_string()
    return hash_MD5(x) == true_hash_MD5(x)

%timeit test_MD5()

def test_SHA1():
    x = random_string()
    return hash_SHA1(x) == true_hash_SHA1(x)

%timeit test_SHA1()

def test_SHA2():
    x = random_string()
    return hash_SHA2(x) == true_hash_SHA2(x)

%timeit test_SHA2()


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Even a small change in the message will (with overwhelming probability) result in a mostly different hash, due to the avalanche effect. For example, adding a period at the end of the sentence:
Can you write Python code for it?

hash_SHA224("The quick brown fox jumps over the lazy dog.")
assert hash_SHA224("The quick brown fox jumps over the lazy dog.") == '619cba8e8e05826e9b8c519c0a5c68f4fb653e8a3d8aa04bb2c8cd4c'
assert hash_SHA224("The quick brown fox jumps over the lazy dog.") == true_hash_SHA224("The quick brown fox jumps over the lazy dog.")


hash_SHA512("The quick brown fox jumps over the lazy dog.")
assert hash_SHA512("The quick brown fox jumps over the lazy dog.") == true_hash_SHA512("The quick brown fox jumps over the lazy dog.")


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
First check on SHA-2
Let us try it:
Can you write Python code for it?

h4 = SHA2()
h4
print(h4)

h4.update(data)
h4.digest()

digest = h4.digest()
bin(digest)
len(bin(digest))

h4.hexdigest()


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
First stupid example: a stupid hashing function
This "stupid" hashing function will use digest_size of 128 bytes (= 16 bits), and compute it by ... just looking at the first 128 bytes of the input data.
This is just to check the API and how to read from a bytes buffer.
Can you write Python code for it?

class HeaderHash(Hash):
    """ This "stupid" hashing function will use `digest_size` of 128 bytes, and compute it by ... just looking at the first 128 bytes of the input data.
    """

    def __init__(self):
        # Common part
        self.digest_size = 16
        self.block_size  = 16
        self.name = "Header"
        # Specific part
        self._data = b""

    def update(self, arg):
        """ Update the hash object with the object arg, which must be interpretable as a buffer of bytes."""
        if len(self._data) == 0:
            self._data = arg[:self.block_size]

    def digest(self):
        """ Return the digest of the data passed to the update() method so far. This is a bytes object of size digest_size which may contain bytes in the whole range from 0 to 255."""
        return self._data


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
It starts to look good.
Can you write Python code for it?

def true_hash_SHA2(data):
    h = hashlib.sha256()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


def true_hash_SHA1(data):
    h = hashlib.sha1()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Let us try it:
Can you write Python code for it?


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Lets check that they have the block size and digest size announced:
Can you write Python code for it?

for name, s in zip(
        ('MD5', 'SHA1', 'SHA256', 'SHA224', 'SHA512', 'SHA384'),
        (hashlib.md5(), hashlib.sha1(), hashlib.sha256(), hashlib.sha224(), hashlib.sha512(), hashlib.sha384())
    ):
    print("For {:<8} : the block size is {:<3} and the digest size is {:<2}.".format(name, s.block_size, s.digest_size))


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
On some random data:
Can you write Python code for it?

random_string(10)

from tqdm import tqdm_notebook as tqdm

%%time
for _ in tqdm(range(1000)):
    x = random_string()
    assert hash_SHA1(x) == true_hash_SHA1(x), "Error: x = {} gave two different SHA1 hashes: my implementation = {} != hashlib implementation = {}...".format(x, hash_SHA1(x), true_hash_SHA1(x))


random_string(10)

from tqdm import tqdm_notebook as tqdm

%%time
for _ in tqdm(range(1000)):
    x = random_string()
    assert hash_SHA2(x) == true_hash_SHA2(x), "Error: x = {} gave two different SHA2 hashes: my implementation = {} != hashlib implementation = {}...".format(x, hash_SHA2(x), true_hash_SHA2(x))


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Second real example: the SHA-1 hashing function
Let now study and implement another hashing function, slightly harder to write but more secure: SHA1, "Secure Hash Algorithm, version 1".
See the SHA1 hashing function on Wikipedia, if needed.
Warning: it is considered broken since at least 2011, it is not advised to use it for real security purposes. SHA-2 or SHA-3 is better advised.
For instance, see this nice blog post.
Useful functions the SHA-1 algorithm
Pretty similar to the ones used for the MD5 algorithm.
Can you write Python code for it?

def SHA1_f1(b, c, d):
    """ First ternary bitwise operation."""
    return ((b & c) | ((~b) & d)) & 0xFFFFFFFF

def SHA1_f2(b, c, d):
    """ Second ternary bitwise operation."""
    return (b ^ c ^ d) & 0xFFFFFFFF

def SHA1_f3(b, c, d):
    """ Third ternary bitwise operation."""
    return ((b & c) | (b & d) | (c & d) ) & 0xFFFFFFFF

def SHA1_f4(b, c, d):
    """ Forth ternary bitwise operation, = SHA1_f1."""
    return (b ^ c ^ d) & 0xFFFFFFFF


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
The MD5 class
It is a direct implementation of the pseudo-code, as given for instance on the Wikipedia page, or the original research article by Rivest.
Can you write Python code for it?

class MD5(Hash):
    """MD5 hashing, see https://en.wikipedia.org/wiki/MD5#Algorithm."""

    def __init__(self):
        self.name        = "MD5"
        self.byteorder   = 'little'
        self.block_size  = 64
        self.digest_size = 16
        # Internal data
        s = [0] * 64
        K = [0] * 64
        # Initialize s, s specifies the per-round shift amounts
        s[ 0:16] = [7, 12, 17, 22,  7, 12, 17, 22,  7, 12, 17, 22,  7, 12, 17, 22]
        s[16:32] = [5,  9, 14, 20,  5,  9, 14, 20,  5,  9, 14, 20,  5,  9, 14, 20]
        s[32:48] = [4, 11, 16, 23,  4, 11, 16, 23,  4, 11, 16, 23,  4, 11, 16, 23]
        s[48:64] = [6, 10, 15, 21,  6, 10, 15, 21,  6, 10, 15, 21,  6, 10, 15, 21]
        # Store it
        self._s = s
        # Use binary integer part of the sines of integers (Radians) as constants:
        for i in range(64):
            K[i] = floor(2**32 * abs(sin(i + 1))) & 0xFFFFFFFF
        # Store it
        self._K = K
        # Initialize variables:
        a0 = 0x67452301   # A
        b0 = 0xefcdab89   # B
        c0 = 0x98badcfe   # C
        d0 = 0x10325476   # D
        self.hash_pieces = [a0, b0, c0, d0]

    def update(self, arg):
        s, K = self._s, self._K
        a0, b0, c0, d0 = self.hash_pieces
        # 1. Pre-processing
        data = bytearray(arg)
        orig_len_in_bits = (8 * len(data)) & 0xFFFFFFFFFFFFFFFF
        # 1.a. Add a single '1' bit at the end of the input bits
        data.append(0x80)
        # 1.b. Padding with zeros as long as the input bits length ≡ 448 (mod 512)
        while len(data) % 64 != 56:
            data.append(0)
        # 1.c. append original length in bits mod (2 pow 64) to message
        data += orig_len_in_bits.to_bytes(8, byteorder='little')
        assert len(data) % 64 == 0, "Error in padding"
        # 2. Computations
        # Process the message in successive 512-bit = 64-bytes chunks:
        for offset in range(0, len(data), 64):
            # 2.a. 512-bits = 64-bytes chunks
            chunks = data[offset :...


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
The SHA2 class
I will use a simple class, very similar to the class used for the SHA-1 algorithm (see above).
It is a direct implementation of the pseudo-code, as given for instance on the Wikipedia page.
I will only implement the simpler one, SHA-256, of digest size of 256 bits. Other variants are SHA-224, SHA-384, SHA-512 (and others include SHA-512/224, SHA-512/256).
Can you write Python code for it?

class SHA2(Hash):
    """SHA256 hashing, see https://en.wikipedia.org/wiki/SHA-2#Pseudocode."""

    def __init__(self):
        self.name        = "SHA256"
        self.byteorder   = 'big'
        self.block_size  = 64
        self.digest_size = 32
        # Note 2: For each round, there is one round constant k[i] and one entry in the message schedule array w[i], 0 ≤ i ≤ 63
        # Note 3: The compression function uses 8 working variables, a through h
        # Note 4: Big-endian convention is used when expressing the constants in this pseudocode,
        #         and when parsing message block data from bytes to words, for example,
        #         the first word of the input message "abc" after padding is 0x61626380

        # Initialize hash values:
        # (first 32 bits of the fractional parts of the square roots of the first 8 primes 2..19):
        h0 = 0x6a09e667
        h1 = 0xbb67ae85
        h2 = 0x3c6ef372
        h3 = 0xa54ff53a
        h4 = 0x510e527f
        h5 = 0x9b05688c
        h6 = 0x1f83d9ab
        h7 = 0x5be0cd19

        # Initialize array of round constants:
        # (first 32 bits of the fractional parts of the cube roots of the first 64 primes 2..311):
        self.k = [
            0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
            0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
            0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
            0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
            0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
            0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
            0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
            0x748f82ee, 0x78a5636f, 0x84c87814,...


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
The SHA512 class
Can you write Python code for it?

class SHA512(Hash):
    """SHA384 hashing, see https://en.wikipedia.org/wiki/SHA-2#Pseudocode."""

    def __init__(self):
        self.name        = "SHA512"
        self.byteorder   = 'big'
        self.block_size  = 128
        self.digest_size = 64
        # Note 2: For each round, there is one round constant k[i] and one entry in the message schedule array w[i], 0 ≤ i ≤ 79
        # Note 3: The compression function uses 8 working variables, a through h
        # Note 4: Big-endian convention is used when expressing the constants in this pseudocode

        # Initialize hash values:
        # (The second 64 bits of the fractional parts of the square roots of the first 8 primes 2..19)
        h0 = 0x6a09e667f3bcc908
        h1 = 0xbb67ae8584caa73b
        h2 = 0x3c6ef372fe94f82b
        h3 = 0xa54ff53a5f1d36f1
        h4 = 0x510e527fade682d1
        h5 = 0x9b05688c2b3e6c1f
        h6 = 0x1f83d9abfb41bd6b
        h7 = 0x5be0cd19137e2179

        # Initialize array of round constants:
        # (first 64 bits of the fractional parts of the cube roots of the first 80 primes 2..409):
        self.k = [
            0x428a2f98d728ae22, 0x7137449123ef65cd, 0xb5c0fbcfec4d3b2f, 0xe9b5dba58189dbbc, 0x3956c25bf348b538,
            0x59f111f1b605d019, 0x923f82a4af194f9b, 0xab1c5ed5da6d8118, 0xd807aa98a3030242, 0x12835b0145706fbe,
            0x243185be4ee4b28c, 0x550c7dc3d5ffb4e2, 0x72be5d74f27b896f, 0x80deb1fe3b1696b1, 0x9bdc06a725c71235,
            0xc19bf174cf692694, 0xe49b69c19ef14ad2, 0xefbe4786384f25e3, 0x0fc19dc68b8cd5b5, 0x240ca1cc77ac9c65,
            0x2de92c6f592b0275, 0x4a7484aa6ea6e483, 0x5cb0a9dcbd41fbd4, 0x76f988da831153b5, 0x983e5152ee66dfab,
            0xa831c66d2db43210, 0xb00327c898fb213f, 0xbf597fc7beef0ee4, 0xc6e00bf33da88fc2, 0xd5a79147930aa725,
            0x06ca6351e003826f, 0x142929670a0e6e70, 0x27b70a8546d22ffc, 0x2e1b21385c26c926, 0x4d2c6dfc5ac42aed,
            0x53380d139d95b3df, 0x650a73548baf63de, 0x766a0abb3c77b2a8, 0x81c2c92e47edaee6, 0x92722c851482353b,
           ...


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
The hash of the zero-length string is:
Can you write Python code for it?


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Third real example: the SHA-2 hashing function
Let now study and implement a last hashing function, again slightly harder to write but more secure: SHA-2, "Secure Hash Algorithm, version 2".
See the SHA-2 hashing function on Wikipedia, if needed.
Remark: it is not (yet) considered broken, and it is the military standard for security and cryptographic hashing. SHA-3 is preferred for security purposes.
Useful functions the SHA-2 algorithm
This is exactly like for MD5. But SHA-2 requires right-rotate as well.
Can you write Python code for it?

def leftrotate(x, c):
    """ Left rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x << c) | (x >> (32 - c))) & 0xFFFFFFFF

def rightrotate(x, c):
    """ Right rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x >> c) | (x << (32 - c))) & 0xFFFFFFFF


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
This is exactly like for MD5.
Can you write Python code for it?

def leftrotate(x, c):
    """ Left rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x << c) | (x >> (32 - c))) & 0xFFFFFFFF


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
We can also write a function to directly compute the hex digest from some bytes data.
Can you write Python code for it?

def hash_SHA1(data):
    """ Shortcut function to directly receive the hex digest from SHA1(data)."""
    h = SHA1()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


def hash_SHA2(data):
    """ Shortcut function to directly receive the hex digest from SHA2(data)."""
    h = SHA2()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


def hash_MD5(data):
    """ Shortcut function to directly receive the hex digest from MD5(data)."""
    h = MD5()
    if isinstance(data, str):
        data = bytes(data, encoding='utf8')
    h.update(data)
    return h.hexdigest()


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
We can check the available algorithms, some of them being guaranteed to be on any platform, some are not.
Can you write Python code for it?


I am working on the file "Manual_implementation_of_some_hash_functions.ipynb".
The first task is:
Well... It seems to work, even if this first example is stupid.

First real example: the MD5 hashing function
Let start with a simple one: the MD5 hashing function, from Rivest in 1992.
Warning: it is considered broken since at least 2012, never use it for security purposes.
Useful functions for the MD5 algorithm
Instead of writing the complete MD5 algorithm in the class below, I preferred to define here some useful functions, using Bitwise operators.
Can you write Python code for it?

def MD5_f1(b, c, d):
    """ First ternary bitwise operation."""
    return ((b & c) | ((~b) & d)) & 0xFFFFFFFF

def MD5_f2(b, c, d):
    """ Second ternary bitwise operation."""
    return ((b & d) | (c & (~d))) & 0xFFFFFFFF

def MD5_f3(b, c, d):
    """ Third ternary bitwise operation."""
    return (b ^ c ^ d) & 0xFFFFFFFF

def MD5_f4(b, c, d):
    """ Forth ternary bitwise operation."""
    return (c ^ (b | (~d))) & 0xFFFFFFFF

def leftrotate(x, c):
    """ Left rotate the number x by c bytes."""
    x &= 0xFFFFFFFF
    return ((x << c) | (x >> (32 - c))) & 0xFFFFFFFF

def leftshift(x, c):
    """ Left shift the number x by c bytes."""
    return x << c

from math import floor, sin


I am working on the file "Map the acoustic environment.ipynb".
The first task is:
Create grid of points
load study area boundary from the database
Can you write Python code for it?

boundary = Boundary.objects.get(name = "study area").geometry.envelope