I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
Making batches
Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to grab all the words in a window around that word, with size $C$.
From Mikolov et al.:
"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $< 1; C >$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."

Exercise: Implement a function get_target that receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you choose a random number of words from the window.
Can you write Python code for it?

def get_target(words, idx, window_size=5):
    ''' Get a list of words in a window around an index. '''

    # Your code here
    R = random.randint(1, window_size + 1)
    start = idx - R if (idx - R) > 0 else 0
    stop = idx + R
    target_words = set(words[start:idx] + words[idx + 1:stop+1])

    return target_words


I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
Negative sampling
For every example we give the network, we train it using the output from the softmax layer. That means for each input, we're making very small changes to millions of weights even though we only have one true example. This makes training the network very inefficient. We can approximate the loss from the softmax layer by only updating a small subset of all the weights at once. We'll update the weights for the correct label, but only a small number of incorrect labels. This is called "negative sampling". Tensorflow has a convenient function to do this, tf.nn.sampled_softmax_loss.

Exercise: Below, create weights and biases for the softmax layer. Then, use tf.nn.sampled_softmax_loss to calculate the loss. Be sure to read the documentation to figure out how it works.
Can you write Python code for it?

# Number of negative labels to sample
n_sampled = 100
with train_graph.as_default():
    # create softmax weight matrix here
    softmax_w = tf.Variable(tf.truncated_normal((n_vocab, n_embedding), stddev=0.1))
    # create softmax biases here
    softmax_b = tf.Variable(tf.zeros(n_vocab))

    # Calculate the loss using negative sampling
    loss = tf.nn.sampled_softmax_loss(softmax_w, softmax_b,
                                      labels, embed,
                                      n_sampled, n_vocab)

    cost = tf.reduce_mean(loss)
    optimizer = tf.train.AdamOptimizer().minimize(cost)


I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
Subsampling
Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by
$$ P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}} $$
where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.
I'm going to leave this up to you as an exercise. This is more of a programming challenge, than about deep learning specifically. But, being able to prepare your data for your network is an important skill to have. Check out my solution to see how I did it.

Exercise: Implement subsampling for the words in int_words. That is, go through int_words and discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data to train_words.
Can you write Python code for it?

## Your code here
from collections import Counter
import random

threshold = 1e-5
word_counts = Counter(int_words)
total_count = len(int_words)
freqs = {word: count / total_count for word, count in word_counts.items()}
p_drop = {word: 1 - np.sqrt(threshold / freqs[word]) for word in word_counts}
train_words = [word for word in int_words if p_drop[word] < random.random()]


I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
Visualizing the word vectors
Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out this post from Christopher Olah to learn more about T-SNE and other ways to visualize high-dimensional data.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

viz_words = 500
tsne = TSNE()
embed_tsne = tsne.fit_transform(embed_mat[:viz_words, :])

fig, ax = plt.subplots(figsize=(14, 14))
for idx in range(viz_words):
    plt.scatter(*embed_tsne[idx, :], color='steelblue')
    plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)


I am working on the file "Emod Angles.ipynb".
The first task is:
In order to use "parallel assignment" we may stretch this column vector into a numpy array of shape (5, ), making it akin to an ordinary Python list.
Can you write Python code for it?

angles = coeffs.dot(m).ravel()
angles.shape

a,b,c,d,e = angles  # get the five values


I am working on the file "Emod Angles.ipynb".
The first task is:
Now we'll use the pandas concat function to create a pandas Dataframe.  Since we're expanding along the column axis, not the row axis, we say axis = 1.  Otherwise concat will try to put the columns on top of one another.
Can you write Python code for it?

df = pd.concat([col1, col2], axis=1)  # build a 2-column dataframe

df.rename(columns={0: 'Radians', 1: 'Degrees'}, inplace=True) # renaming columns

df


I am working on the file "Emod Angles.ipynb".
The first task is:
Per David's comments above: f, g, h serve as basis angles such that linear combinations thereof net additional angles a-e.
<pre>
a = 2f + 2g + h
b = 2f + 3g + h
c = 2f + g
d = f + g
e = g + h
</pre>

When it comes to linear combinations, we tend to use "matrix operations" and in this case simple integer coefficients do the work.  Create [f, g, h] as a vertical column vector, and multiply it on the left with a matrix.  For example, in the top row of this matrix we have
[2,2,1] meaning we'll get 2 * f + 2 * g + h as the top cell of our 5 by 1 result matrix, or angle a.
Can you write Python code for it?


I am working on the file "Emod Angles.ipynb".
The first task is:
The arctangent function is the inverse of the tangent function.  Going from the radians column back to some initial argument to arctangent, should give us a final cross-check of our initial phi expressions, derived by David Koski.
Here we go then:
Can you write Python code for it?

col0 = np.tan(col1)  # work back from radians

col0


I am working on the file "Emod Angles.ipynb".
The first task is:
We get our arctangent results in radians.  To confirm we getting the right numbers in terms of degrees, lets set up a parallel column.  Numpy operations apply to entire columns or vectors, no need to write a Python loop.
Can you write Python code for it?

col2 = np.degrees(col1)  # convert radians to degrees
col2


I am working on the file "Emod Angles.ipynb".
The first task is:
We say "square" root (in quotes) because in Synergetics we may want to use a triangular area to show the same idea.  Two edges A, B at 60 degrees multiply to give an area A times B.  Just add the 3rd vector (A-B) to "close the lid".  The same technique applies to multiplying A times B times C:  three vectors from an origin, at 60 degrees to one another, define a tetrahedron.  "Close the lid" to get the corresponding volume.
Here's another figure, Fig. 990.01 from Synergetics to help make sense of this idea.

It's important to introduce these alternative models of 2nd and 3rd powering because our idea of "tetravolumes" derives from there.
As of 2018, very few mathematicians have dared to comment on this 1900s literary innovation within New England Transcendentalism.
Hugh Kenner, a James Joyce scholar, is one of the few to have contributed any secondary literature regarding Fuller's forays into mathematics and logic.  Given how alien this thinking remains, we sometimes refer to it as "Martian Math".
Can you write Python code for it?

rt5 = sqrt(5)    # 2nd root of 5 or "square" root
Ø = (1 + rt5)/2  # golden ratio

f = atan((2*rt5 - 3)/11)
g = atan((5*rt5 - 9)/22)
h = atan(Ø ** -5)


I am working on the file "EmpiricalResults/Demo_Network2NetworkInformationTransfer.ipynb".
The first task is:
1.0 Run information transfer mapping procedure
1.1 First load in resting-state FC matrices for subjects
Can you write Python code for it?

fcmat = {}
for subj in subjNums:
    fcmat[subj] = np.loadtxt('data/FC_Estimates/' + subj + '_multregconn_restfc.csv', delimiter=',')


I am working on the file "EmpiricalResults/Demo_Network2NetworkInformationTransfer.ipynb".
The first task is:
1.2 Perform network-to-network information transfer mapping procedure using python module
1.2.1 First construct a wrapper to pass through a multiprocessing scheme
Can you write Python code for it?

def informationTransferMappingWrapper((subj,fcmat)):
    """
    A wrapper so we can use multiprocessing to run subjects in parallel
    """
    out = n2n.networkToNetworkInformationTransferMapping(subj,fcmat,null=False)
    return out


I am working on the file "EmpiricalResults/Demo_Network2NetworkInformationTransfer.ipynb".
The first task is:
1.2.2 Run using multiprocessing (parallel processing) to speed-up computation
Can you write Python code for it?

inputs = []
for subj in subjNums:
    inputs.append((subj,fcmat[subj]))

pool = mp.Pool(processes=32)
results = pool.map_async(informationTransferMappingWrapper,inputs).get()
pool.close()
pool.join()

# Collect results
ruledims = ['logic','sensory','motor']
ite_matrix = {}
for ruledim in ruledims:
    ite_matrix[ruledim] = np.zeros((len(networkmappings),len(networkmappings),len(subjNums)))

scount = 0
for result in results:
    for ruledim in ruledims:
        ite_matrix[ruledim][:,:,scount] = result[ruledim]

    scount += 1


I am working on the file "EmpiricalResults/Demo_Network2NetworkInformationTransfer.ipynb".
The first task is:
2.1 Perform multiple comparisons (using false discovery rate)
Can you write Python code for it?

# Compute group stats
baseline = 0.0
triu_indices = np.triu_indices(len(networkmappings),k=1)
tril_indices = np.tril_indices(len(networkmappings),k=-1)

qmat = {}
for ruledim in ruledims:

    qmat[ruledim] = np.zeros((num_networks,num_networks))

    tmpq = []
    tmpq.extend(pvals[ruledim][triu_indices])
    tmpq.extend(pvals[ruledim][tril_indices])
    tmpq = mc.fdrcorrection0(tmpq)[1]
    qmat[ruledim][triu_indices] = tmpq[0:len(triu_indices[0])]
    qmat[ruledim][tril_indices] = tmpq[len(triu_indices[0]):]


I am working on the file "EmpiricalResults/Demo_Network2NetworkInformationTransfer.ipynb".
The first task is:
2.2 Plot results
Can you write Python code for it?

for ruledim in ruledims:
    plt.figure(figsize=(12,10))
    # First visualize unthresholded results
    plt.subplot(121)
    plt.title('NetworkToNetwork Information Transfer Mapping\n' + ruledim + ' domain', fontsize=14, y=1.04)
    mat = avg_rho[ruledim]
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.colorbar(fraction=.046)

    # Next visualize thresholded results (after multiple comparisons)
    plt.subplot(122)
    plt.title('NetworkToNetwork Information Transfer Mapping\n' + ruledim + ' domain' , fontsize=14, y=1.04)
    mat = avg_rho[ruledim]
    thresh = qmat[ruledim] < 0.05
    # Threshold using q < 0.05
    mat = np.multiply(mat,thresh)
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.colorbar(fraction=.046)
    plt.tight_layout()


I am working on the file "EmpiricalResults/Demo_NetworkInformationEstimate.ipynb".
The first task is:
0.0 Basic parameters
Can you write Python code for it?

# Set basic parameters
datadir = './data/'
runLength = 4648
subjNums = ['032', '033', '037', '038', '039', '045',
            '013', '014', '016', '017', '018', '021',
            '023', '024', '025', '026', '027', '031',
            '035', '046', '042', '028', '048', '053',
            '040', '049', '057', '062', '050', '030', '047', '034']

# Load in network array
networkdef = np.loadtxt(datadir + 'network_array.csv', delimiter=',')

# Load in network keys (each network associated with a number in network array)
networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud1':8, 'aud2':9, 'dan':11}
# Force aud2 key to be the same as aud1 (merging two auditory networks)
aud2_ind = np.where(networkdef==networkmappings['aud2'])[0]
networkdef[aud2_ind] = networkmappings['aud1']
# Redefine new network mappings with no aud1/aud2 distinction
networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud':8, 'dan':11}


I am working on the file "EmpiricalResults/Demo_NetworkInformationEstimate.ipynb".
The first task is:
Set up basic functions
Can you write Python code for it?

def loadBetas(subj, net='all'):
    """Loads in task betas"""
    datafile = datadir + 'MiniblockActivations/' + subj + '_miniblock_taskBetas_Glasser.csv'
    betas = np.loadtxt(datafile, delimiter=',')
    betas = betas[:,17:]
    if net == 'all':
        return betas
    else:
        net_ind = np.where(networkdef==net)[0]
        return betas[net_ind,:].T

def setupMatrix(subj,ruledim,net):
    """
    Sets up basic SVM Matrix for a classification of a particular rule dimension and network
    """

    svm_mat = loadBetas(subj,net=net)
    # Subtract miniblock indices by 1, since they were originally created for Matlab array indices
    labels = np.loadtxt('./data/CPROTaskIdentifiers/' + subj + '_' + ruledim + '_miniblocksPerRule_MatlabIndices.csv',delimiter=',') - 1

    labels = np.asarray(labels,dtype=int)

    return svm_mat, labels


I am working on the file "EmpiricalResults/Demo_NetworkInformationEstimate.ipynb".
The first task is:
Set up functions for RSA analysis (instead of SVM decoding)
Can you write Python code for it?

def rsaCV(svm_mat,labels,ruledim):
    """Runs a leave-4-out CV for a 4 way  classification"""

    # 32 folds, if we do a leave 4 out for 128 total miniblocks
    # Want to leave a single block from each rule from each CV
    # labels is a sample x rule matrix, so 32 samples x 4 rules

    # Number of CVs is columns
    ncvs = labels.shape[0]
    nrules = labels.shape[1]
    corr_rho_cvs = []
    err_rho_cvs = []
    for cv in range(ncvs):
        # Select a test set from the CV Fold matrix
        test_ind = labels[cv,:].copy()
        # Delete the CV included from the train set
        train_ind = np.delete(labels,cv,axis=0)

        # Identify the train and test sets
        svm_train = svm_mat[np.reshape(train_ind,-1),:]
        svm_test = svm_mat[test_ind,:]

        prototype = {}
        # Construct RSA prototypes
        for rule in range(nrules):
            prototype_ind = np.reshape(train_ind[:,rule],-1)
            prototype[rule] = np.mean(svm_mat[prototype_ind],axis=0)

        corr_rho = []
        err_rho = []
        for rule1 in range(nrules):
            for rule2 in range(nrules):
                r = stats.spearmanr(prototype[rule1],svm_test[rule2,:])[0]
                r = np.arctanh(r)
                if rule1==rule2:
                    corr_rho.append(r)
                else:
                    err_rho.append(r)

        corr_rho_cvs.append(np.mean(corr_rho))
        err_rho_cvs.append(np.mean(err_rho))

    return np.mean(corr_rho_cvs), np.mean(err_rho_cvs)

def subjRSACV((subj,ruledim,net)):
    svm_mat, labels = setupMatrix(subj,ruledim,net)
    # Demean each sample
    svmmean = np.mean(svm_mat,axis=1)
    svmmean.shape = (len(svmmean),1)
    svm_mat = svm_mat - svmmean

#     svm_mat = preprocessing.scale(svm_mat,axis=0)

    corr_rho, err_rho = rsaCV(svm_mat, labels, ruledim)
    diff_rho = corr_rho - err_rho

    return corr_rho, err_rho, diff_rho

netkeys = {0:'fpn', 1:'dan', 2:'con', 3:'dmn',...


I am working on the file "EmpiricalResults/Demo_NetworkInformationEstimate.ipynb".
The first task is:
Show results as dataframe
Can you write Python code for it?

results_dframe_fwe = {}
for ruledim in ruledims:
    print 'Dataframe for', ruledim, 'classification'
    results_dframe_fwe[ruledim] = pandas.DataFrame(data=results_dict_fwe[ruledim])
    display(results_dframe_fwe[ruledim])

#### Compute statistics for bar plot
bar_avgs = {}
bar_sems = {}
bar_avg_all = {}
for net in netkeys.keys(): bar_avg_all[net] = np.zeros((len(ruledims),len(subjNums)))

rulecount = 0
for ruledim in ruledims:
    bar_avgs[ruledim] = {}
    bar_sems[ruledim] = {}
    for net in netkeys.keys():
        bar_avgs[ruledim][net] = np.mean(diff_rho[ruledim][net])
        bar_sems[ruledim][net] = np.std(diff_rho[ruledim][net])/np.sqrt(len(subjNums))
        bar_avg_all[net][rulecount,:] = diff_rho[ruledim][net]
    rulecount += 1

bar_sem_all = {}
for net in netkeys.keys():
    meanacc = np.mean(bar_avg_all[net],axis=0)
    bar_avg_all[net] = np.mean(meanacc)
    bar_sem_all[net] = np.std(meanacc)/np.sqrt(len(subjNums))

##### Generate figures
width=0.25
width=.265
networks = netkeys.keys()
nbars = len(networks)
fig = plt.figure()
ax = fig.add_subplot(111)
rects = {}
widthcount = 0
colors = ['b','g','r']
colorcount = 0
for ruledim in ruledims:
    rects[ruledim] = ax.bar(np.arange(nbars)+widthcount, bar_avgs[ruledim].values(), width,align='center',
                     yerr=bar_sems[ruledim].values(), color=colors[colorcount], error_kw=dict(ecolor='black'))
    widthcount += width
    colorcount += 1

ax.set_title('Network RSA of CPRO rules (FWER)',
             y=1.04, fontsize=16)
ax.set_ylabel('Match V. Mismatch Difference (Rho)',fontsize=12)
ax.set_xlabel('Rule types by Networks', fontsize=12)
ax.set_xticks(np.arange(nbars)+width)
ax.set_xticklabels(netkeys.values(),rotation=-45)
vmax=0.07
ax.set_ylim([0,vmax])

plt.legend((rects['logic'], rects['sensory'], rects['motor']),
           ('Logic', 'Sensory', 'Motor'), loc=((1.08,.65)))

## Add asterisks
def autolabel(rects,df,ruledim):
    # attach some text labels
    netcount = 0
    for rect in rects:
        height =...


results_dframe_fdr = {}
for ruledim in ruledims:
    print 'Dataframe for', ruledim, 'classification'
    results_dframe_fdr[ruledim] = pandas.DataFrame(data=results_dict_fdr[ruledim])
    display(results_dframe_fdr[ruledim])

#### Compute statistics for bar plot
bar_avgs = {}
bar_sems = {}
bar_avg_all = {}
for net in netkeys.keys(): bar_avg_all[net] = np.zeros((len(ruledims),len(subjNums)))

rulecount = 0
for ruledim in ruledims:
    bar_avgs[ruledim] = {}
    bar_sems[ruledim] = {}
    for net in netkeys.keys():
        bar_avgs[ruledim][net] = np.mean(diff_rho[ruledim][net])
        bar_sems[ruledim][net] = np.std(diff_rho[ruledim][net])/np.sqrt(len(subjNums))
        bar_avg_all[net][rulecount,:] = diff_rho[ruledim][net]
    rulecount += 1

bar_sem_all = {}
for net in netkeys.keys():
    meanacc = np.mean(bar_avg_all[net],axis=0)
    bar_avg_all[net] = np.mean(meanacc)
    bar_sem_all[net] = np.std(meanacc)/np.sqrt(len(subjNums))

##### Generate figures
width=0.25
width=.265
networks = netkeys.keys()
nbars = len(networks)
fig = plt.figure()
ax = fig.add_subplot(111)
rects = {}
widthcount = 0
colors = ['b','g','r']
colorcount = 0
for ruledim in ruledims:
    rects[ruledim] = ax.bar(np.arange(nbars)+widthcount, bar_avgs[ruledim].values(), width,align='center',
                     yerr=bar_sems[ruledim].values(), color=colors[colorcount], error_kw=dict(ecolor='black'))
    widthcount += width
    colorcount += 1

ax.set_title('Network RSA of CPRO rules (FDR)',
             y=1.04, fontsize=16)
ax.set_ylabel('Match V. Mismatch Difference (Rho)',fontsize=12)
ax.set_xlabel('Rule types by Networks', fontsize=12)
ax.set_xticks(np.arange(nbars)+width)
ax.set_xticklabels(netkeys.values(),rotation=-45)
vmax=0.07
ax.set_ylim([0,vmax])

plt.legend((rects['logic'], rects['sensory'], rects['motor']),
           ('Logic', 'Sensory', 'Motor'), loc=((1.08,.65)))

## Add asterisks
def autolabel(rects,df,ruledim):
    # attach some text labels
    netcount = 0
    for rect in rects:
        height =...


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
'Total Energy Consumed by the Residential Sector'
Can you write Python code for it?

# split by year-- proportion in each year and stack them.
#
def get_max_record(group):
    return group.ix[group.Value.idxmax()]
TERCBUSresult =TERCBUS_yr.groupby('YYYY').apply(get_max_record)

# Plot TERCBUS Values
TERCBUSresult.Value.plot()

# Making a title:
plt.title('Total Energy Consumed by the Residential Sector', weight='bold')
plt.ylabel('Trillion BTU')
plt.xlabel('Year')
plt.legend(['Total Energy'], loc ='best')

# The image can be saved as a pdf for later use:
plt.savefig('TERCBUSresult.pdf')


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Create a 2013 DataFrame
Can you write Python code for it?

#Show 2013 Monthly totals and yearly summary (MM=13)

TFFCRS2013 = Res_Energy_Consumption_1949_2012[Res_Energy_Consumption_1949_2012.YYYY == 2013].sort_index(by='MM',ascending=True)

# Print the dataframe
TFFCRS2013

#birdseye, quick view of the data: as a cross tab-- just 2013 data.
pd.crosstab(TFFCRS2013.Description, TFFCRS2013.Value)

# Need to bin this.There are too many values


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Create a dataframe of Total Fossil Fuels Consumed by the Residential Sector (TFFCRS).
Can you write Python code for it?

#Total Fossil Fuels Consumed by the Residential Sector = TFFCRS
# use the yearly number:
TFFCRS_yr=  Res_Energy_Consumption_yr[Res_Energy_Consumption_yr.Description == 'Total Fossil Fuels Consumed by the Residential Sector']

TFFCRS_yr.head(10)

TFFCRS_yr.tail(10)

# Can just start from here now.
# Have it local, cleaned.
TFFCRS_yr.to_csv("TFFCRS_yr.csv", index=False)

# Summary for each name
# A series method.
TFFCRS_yr.groupby('YYYY')['Value'].describe()

# Summary for each year
# a series method providing the mean.
TFFCRS_yr.groupby('YYYY')['Value'].mean()

# Create a Function to get the maximum Value over the years.
def get_max_record(group):
    return group.ix[group.Value.idxmax()]
TFFCRS_yr_max =TFFCRS_yr.groupby('Value').apply(get_max_record)

TFFCRS_yr_max.ix[TFFCRS_yr_max.Value.idxmax()]


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Finding:

Adj. R-squared: 0.042 indicates a mild positive relationship over time for residential fossil fuel consumption. The relationship is statistically significant at the 90% confidence interval (p = 0.055). However, auto-correlation is an important issue in time-series analyses and the Durbin-Watson test (d = 0.105) suggests autocorrelation and the concern that the level of statistical significance is underestimated.
As indicated in Wikipedia:

"Since d is approximately equal to 2(1 − r), where r is the sample autocorrelation of the residuals,[3] d = 2 indicates no autocorrelation. The value of d always lies between 0 and 4. If the Durbin–Watson statistic is substantially less than 2, there is evidence of positive serial correlation. As a rough rule of thumb, if Durbin–Watson is less than 1.0, there may be cause for alarm. Small values of d indicate successive error terms are, on average, close in value to one another, or positively correlated. If d > 2, successive error terms are, on average, much different in value from one another, i.e., negatively correlated. In regressions, this can imply an underestimation of the level of statistical significance."

Source: http://en.wikipedia.org/wiki/Durbin%E2%80%93Watson_statistic
Look to this exercise (https://www.udel.edu/htr/Statistics/Notes816/class20.PDF) for an interesting analysis of this energy data, with consideration of the energy crises of the early 1970s. Consider building on that work.
Explore Yearly Energy Data By Fuel Type
The grouping documentation can be found here:
http://pandas.pydata.org/pandas-docs/dev/groupby.html.
Split, Apply, and Combine.
Can you write Python code for it?

Res_Energy_Consumption_yr_desc = Res_Energy_Consumption_yr.groupby(['Description'])
print Res_Energy_Consumption_yr

# This is not working.
#grouped = Res_Energy_Consumption_yr_desc.groupby('MSN')

Res_Energy_Consumption_yr_desc

# Not working
#for Value, group in Res_Energy_Consumption_yr:
 #   print(Description)


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Finding: Still a low Durbin-Watson.
Look for a more interesting and practical regression analysis.
Can you write Python code for it?


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Finding: Total Energy Consumed by the Residential Sector generally increases over time, with some declines in recent years.
Can you write Python code for it?

#Describe the numerical data for Total Energy Consumed by the Residential Sector.
TERCBUSresult.Value.describe()


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Finding: While U.S. residential total energy use has continued to increase since 1970, fossil fuel use has leveled off.
Can you write Python code for it?

# Playing with Regression: Fossil Fuels: TFFCRS by Year
# Though autocorrelation issues may exist.
# See https://www.udel.edu/htr/Statistics/Notes816/class20.PDF for a description

# Increase over time: Adj. R-squared: 0.042
# Statistically significant relationship at 90% Confidence Interval, but not at 95%: P =0.055
# But Durbin-Watson:0.105 suggests autocorrelation (less than 1 on a scale of 0 to 4

model = smf.ols('Value ~ YYYY', data=TFFCRS_yr)
results = model.fit()
results.summary()


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Findings: There is data for 65 years. The average total energy usage is 14,986 trillion BTUs.
Can you write Python code for it?

# Create an interger for the maximum value.
s =  int(max(TERCBUSresult.Value))

print "The maximum value (in trillion BTUs) is: %s " %s   # insert value into a string with the %s placeholder


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Findings:

974,079 Trillion Btu Total Total Energy Consumed by the Residential Sector from 1949 to 2012.

408,070 Trillion Btu Total Fossil Fuels Consumed by the Residential Sector from 1949 to 2012.

Nearly, two-fifths of US energy over this time period is from fossil fuels.
Create a dataframe of Total Energy Consumed by the Residential Sector (TERCBUS).
Can you write Python code for it?

#Total Energy Consumed by the Residential Sector
TERCBUS_yr=  Res_Energy_Consumption_yr[Res_Energy_Consumption_yr.Description == 'Total Energy Consumed by the Residential Sector']


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Look at Natural Gas when I have more time
Can you write Python code for it?

# not working
#NG_result.NaturalGas

# not working
#Res_Energy_Consumption_yr.NaturalGas

#Res_Energy_Consumption_yr.groupby(['Year','NaturalGas']).size().ix[2000]


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Next Steps
Can you write Python code for it?

# Merge in some other data and try to analyze it.
# Such as economic data.
# Perhaps do a state by state analysis of energy use and economic growth.
#  TestData2 = read_csv('TestData2.csv')

# pandas join function is merge
# merge?
# merge(Res_Primary_E_Consumption, TestData2, on= [ 'Year'])
# simple join on year
# inner join is default

#merged = merge(Res_Primary_E_Consumption, TestData2, on= [ 'Year'])

#from __future__ import division
#if needed

#merged['NG'] = np.float32(merged.NaturalGas)

#merged['TTL']  = np.float32(merged.TOTAL)

#merged['RNW']  = np.float32(merged.Renewable)

# Create a new proportion variable
#merged['NG_percent'] = (merged.NG / merged.TTL)

# total test.. Create a new column called TOTALtest,
# need to add the number of persons
#merged['TOTALtest'] = np.floor(merged.TOTAL * merged.Col2)

#decimal numbers so use npl.floor

#merged.groupby(['Year', 'TOTALtest'])['Col3'].sum()
#for each year TOTALtest combo-- the total number of Col3.
#hierarchical labeling.

# Regression Analyses to come. For review
'''
import statsmodels.formula.api as smf
model = smf.ols('merged.TOTAL ~ merged.Col2', data=merged)
results_merged = model.fit()
results_merged.summary()
'''


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Quick Data Analysis with Pandas: Yearly Data
Can you write Python code for it?

# Use only yearly data
# MM = 13.

Res_Energy_Consumption_yr =  Res_Energy_Consumption_1949_2012[Res_Energy_Consumption_1949_2012.MM == 13]

# Use the pivot table function to show the values for each description
report = Res_Energy_Consumption_yr.pivot_table(values=['MSN', 'Value'],index=['Description'], aggfunc=np.sum)
report

# for just the top 10
#.head(n=10)


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Renewables as a Percent of Primary Data
I think it should be based on primary data--For review.
Total Primary Energy Consumed by the Residential Sector 449735.904
Total Renewable Energy Consumed by the Residential Sector   41666.355
(Ideally, pull that directly)
Can you write Python code for it?

s = round(41666/449736*100,2)
print "Overall, the percent renewable out of primary residential is: %s"  %s


I am working on the file "EnergyConsumptionEstimates_1949_2012-Residential_Sector.ipynb".
The first task is:
Total energy data by year regression analysis
Can you write Python code for it?

# Following: http://allendowney.blogspot.com/
# from http://allendowney.blogspot.com/
# Value by year.
###NEEDS TO BE DONE BY EACH DESCRIPTION TYPE. and compared.
# also chart each comparison.

# Total energy data by year.
import statsmodels.formula.api as smf
model = smf.ols('Value ~ YYYY', data=Res_Energy_Consumption_1949_2012)
results = model.fit()
results.summary()


I am working on the file "Engineer.ipynb".
The first task is:
Ages
We'll impute missing values with medians by Title and Sex.
Can you write Python code for it?

df['FilledAge'] = df.groupby(['Sex', 'Title'])['Age'].apply(lambda x: x.fillna(x.median()))


I am working on the file "Engineer.ipynb".
The first task is:
Cabins
We create an indicator variable if the cabin is known, for now.
Can you write Python code for it?


I am working on the file "Engineer.ipynb".
The first task is:
Title
We'll extract title information from the Name feature, and then merge some of the titles together.

Merge 'Mme' into 'Mrs'
Merge 'Mlle' and 'Ms' into 'Miss'
Merge 'Lady', 'the Countess', and 'Dona' into 'fNoble'
Merge 'Don', 'Sir', and 'Jonkheer' into 'mNoble'
Merge 'Col', 'Capt', and 'Major' into 'mil'
Can you write Python code for it?

df['Title'] = df['Name'].str.extract('\,\s(.*?)[.]', expand=False)
df['Title'].replace('Mme', 'Mrs', inplace=True)
df['Title'].replace('Mlle', 'Miss', inplace=True)
df['Title'].replace('Ms', 'Miss', inplace=True)
df['Title'].replace('Lady', 'fNoble', inplace=True)
df['Title'].replace('the Countess', 'fNoble', inplace=True)
df['Title'].replace('Dona', 'fNoble', inplace=True)
df['Title'].replace('Don', 'mNoble', inplace=True)
df['Title'].replace('Sir', 'mNoble', inplace=True)
df['Title'].replace('Jonkheer', 'mNoble', inplace=True)
df['Title'].replace('Col', 'mil', inplace=True)
df['Title'].replace('Capt', 'mil', inplace=True)
df['Title'].replace('Major', 'mil', inplace=True)


I am working on the file "English/Optimizing Fortran and Python code/Optimizing Fortran and Python code.ipynb".
The first task is:
Dbscan clustering algorithm
It's a density-based clustering algorithm: given a set of points in some space, it groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions (whose nearest neighbors are too far away).
Each cluster is made up of core and non-core points. Points that are not included in any cluster are considered noise. A point p is a core point if at least min_points points are within distance eps of it, and those points are said to be directly reachable from p.

Can you write Python code for it?

UNCLASSIFIED = False
NOISE = None

def _dist(p,q):
	return math.sqrt(np.power(p-q,2).sum())

def _eps_neighborhood(p,q,eps):
	return _dist(p,q) < eps

def _region_query(m, point_id, eps):
    n_points = m.shape[1]
    seeds = []
    for i in range(0, n_points):
        if not i == point_id:
            if _eps_neighborhood(m[:,point_id], m[:,i], eps):
                seeds.append(i)
    return seeds

def _expand_cluster(m, classifications, point_id, cluster_id, eps, min_points):
    seeds = _region_query(m, point_id, eps)
    if len(seeds) < min_points:
        classifications[point_id] = NOISE
        return False
    else:
        classifications[point_id] = cluster_id
        for seed_id in seeds:
            classifications[seed_id] = cluster_id

        while len(seeds) > 0:
            current_point = seeds[0]
            results = _region_query(m, current_point, eps)
            if len(results) >= min_points:
                for i in range(0, len(results)):
                    result_point = results[i]
                    if classifications[result_point] == UNCLASSIFIED or \
                       classifications[result_point] == NOISE:
                        if classifications[result_point] == UNCLASSIFIED:
                            seeds.append(result_point)
                        classifications[result_point] = cluster_id
            seeds = seeds[1:]
        return True

def dbscan(m, eps, min_points):
    """Implementation of Density Based Spatial Clustering of Applications with Noise
    See https://en.wikipedia.org/wiki/DBSCAN

    scikit-learn probably has a better implementation

    Uses Euclidean Distance as the measure

    Inputs:
    m - A matrix whose columns are feature vectors
    eps - Maximum distance two points can be to be regionally related
    min_points - The minimum number of points to make a cluster

    Outputs:
    An array with either a cluster id number or dbscan.NOISE (None) for each
    column vector in m.
    """
    cluster_id =...


I am working on the file "English/Optimizing Fortran and Python code/Optimizing Fortran and Python code.ipynb".
The first task is:
Dbscan with F2Py
Can you write Python code for it?

print dbscanf2py.dist.__doc__ #F2Py automatically generates a docstring for our functions

def _eps_neighborhood(p,q,eps):
	return dbscanf2py.dist(p,q) < eps

%timeit dbscan(m, eps, min_points)


I am working on the file "English/Optimizing Fortran and Python code/Optimizing Fortran and Python code.ipynb".
The first task is:
Dbscan with Numba
Can you write Python code for it?

@jit
def _dist(p,q):
	return math.sqrt(np.power(p-q,2).sum())

%timeit dbscan(m, eps, min_points)


I am working on the file "English/Optimizing Fortran and Python code/Optimizing Fortran and Python code.ipynb".
The first task is:
Factorial functions
The speed of different factorial functions is compared.
Can you write Python code for it?

# Pure Python
def factorial_0(n):
    Ans = 1
    for i in range(1, n+1):
        Ans = Ans * i
    return Ans

# NumPy function
def factorial_1(n):
    return np.math.factorial(n)

# Numba-optimized pure Python function
@jit
def factorial_2(n):
    Ans = 1
    for i in range(1, n+1):
        Ans = Ans * i
    return Ans

# Numba-optimized NumPy function
@jit
def factorial_3(n):
    return np.math.factorial(n)

# Recursive Fortran function, compiled with F2Py
import factorial1f2py

# Non-recursive Fortran function, compiled with F2Py
import factorial2f2py

%timeit factorial_0(12)
%timeit factorial_1(12)
%timeit factorial_2(12)
%timeit factorial_3(12)
%timeit factorial1f2py.factorial(12)
%timeit factorial2f2py.factorial(12)


I am working on the file "English/Optimizing Fortran and Python code/Optimizing Fortran and Python code.ipynb".
The first task is:
Let's %timeit
Can you write Python code for it?


I am working on the file "English/Optimizing Fortran and Python code/Optimizing Fortran and Python code.ipynb".
The first task is:
Sum function
The speed of different sum functions is compared.
Can you write Python code for it?

# A pure python funcion
def sum_0(arr):
    M, N = arr.shape
    result = 0
    for i in range(M):
        for j in range(N):
            result += arr[i,j]
    return result

# A NumPy version of the previous funcion
def sum_1(arr):
    return np.sum(arr)

# The numba-optimized version of the first function
@jit
def sum_2(arr):
    M, N = arr.shape
    result = 0
    for i in range(M):
        for j in range(N):
            result += arr[i,j]
    return result

# The numba-optimized version of the second function
@jit
def sum_3(arr):
    return np.sum(arr)

# Fortran function, brought to you by F2Py
import array_sum

# an array to test our functions
a = np.arange(65536).reshape(256,256)

%timeit sum_0(a) # Python
%timeit sum_1(a) # NumPy
%timeit sum_2(a) # Numba
%timeit sum_3(a) # NumPy + Numba
%timeit array_sum.array_sum(a) # F2Py


I am working on the file "Enrich & Prepare MovieLens Dataset.ipynb".
The first task is:
Enrich movie data with TMDB metadata
Can you write Python code for it?

with open("data/ml-latest-small/movies.csv") as f:
    reader = csv.reader(f)
    cols = reader.next()
    raw_movies = sqlContext.createDataFrame([l for l in reader], cols)
with open("data/ml-latest-small/links.csv") as f:
    reader = csv.reader(f)
    cols = reader.next()
    link_data = sqlContext.createDataFrame([l for l in reader], cols)
movie_data = raw_movies.join(link_data, raw_movies.movieId == link_data.movieId)\
    .select(raw_movies.movieId, raw_movies.title, raw_movies.genres, link_data.tmdbId)
num_movies = movie_data.count()
movie_data.show(5)
data = movie_data.collect()

import tmdbsimple as tmdb
tmdb.API_KEY = 'YOUR_KEY'
# base URL for TMDB poster images
IMAGE_URL = 'https://image.tmdb.org/t/p/w500'
import csv
from requests import HTTPError

enriched = []
i = 0
for row in data:
    try:
        m = tmdb.Movies(row.tmdbId).info()
        poster_url = IMAGE_URL + m['poster_path'] if 'poster_path' in m and m['poster_path'] is not None else ""
        movie = {
            "movieId": row.movieId,
            "title": m['title'],
            "originalTitle": row.title,
            "genres": row.genres,
            "overview": m['overview'],
            "release_date": m['release_date'],
            "popularity": m['popularity'],
            "original_language": m['original_language'],
            "image_url": poster_url
        }
        enriched.append(movie)
    except HTTPError as e:
        print "Encountered error: %s for movieId=%d title=%s" % (e, row.movieId, row.title)
        movie = {
            "movieId": row.movieId,
            "title": row.title,
            "originalTitle": row.title,
            "genres": row.genres,
            "overview": "",
            "release_date": "",
            "popularity": 0,
            "original_language": "",
            "image_url": ""
        }
        enriched.append(movie)
    i += 1
    if i % 1 == 0: print "Enriched movie %s of %s" % (i, num_movies)


I am working on the file "Enron Spam Solution.ipynb".
The first task is:
Python 3 Encodings: http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html
Can you write Python code for it?

# Write a function , that when passed in words, will return a dictionary of the form

# {Word1: True, Word2: True, Words3: True}

# Removing stop words is optional

def create_word_features(words):
    my_dict = dict( [ (word, True) for word in words] )
    return my_dict

create_word_features(["the", "quick", "brown", "quick", "a", "fox"])

ham_list = []
spam_list = []

# Same as before, but this time:

# 1. Break the sentences into words using word_tokenize
# 2. Use the create_word_features() function you just wrote
for directories, subdirs, files in os.walk(rootdir):
    if (os.path.split(directories)[1]  == 'ham'):
        for filename in files:
            with open(os.path.join(directories, filename), encoding="latin-1") as f:
                data = f.read()

                # The data we read is one big string. We need to break it into words.
                words = word_tokenize(data)

                ham_list.append((create_word_features(words), "ham"))

    if (os.path.split(directories)[1]  == 'spam'):
        for filename in files:
            with open(os.path.join(directories, filename), encoding="latin-1") as f:
                data = f.read()

                # The data we read is one big string. We need to break it into words.
                words = word_tokenize(data)

                spam_list.append((create_word_features(words), "spam"))
print(ham_list[0])
print(spam_list[0])

combined_list = ham_list + spam_list
print(len(combined_list))

random.shuffle(combined_list)

# Create a test and train section.

# 70% of the data is training. 30% is test

training_part = int(len(combined_list) * .7)

print(len(combined_list))

training_set = combined_list[:training_part]

test_set =  combined_list[training_part:]

print (len(training_set))
print (len(test_set))

# Create the Naive Bayes filter

classifier = NaiveBayesClassifier.train(training_set)

# Find the accuracy, using the test data

accuracy =...


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Bagging -- Building an ensemble of classifiers from bootstrap samples
Bagging is an ensemble learning technique that is closely related to the MajorityVoteClassifier,however, instead of using the same training set to fit the individual classifiers in the ensemble, we draw bootstrap samples (random samples with replacement) from the initial training set, which is why bagging is also known as bootstrap aggregating.
Can you write Python code for it?

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=None)

bag = BaggingClassifier(base_estimator=tree,
                        n_estimators=500,
                        max_samples=1.0,
                        max_features=1.0,
                        bootstrap=True,
                        bootstrap_features=False,
                        n_jobs=-1,
                        random_state=1)

from sklearn.metrics import accuracy_score

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

bag = bag.fit(X_train, y_train)
y_train_pred = bag.predict(X_train)
y_test_pred = bag.predict(X_test)

bag_train = accuracy_score(y_train, y_train_pred)
bag_test = accuracy_score(y_test, y_test_pred)
print('Bagging train/test accuracies %.3f/%.3f'
      % (bag_train, bag_test))


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Combining different algorithms for classification with majority vote
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
import numpy as np

clf1 = LogisticRegression(penalty='l2',
                          C=0.001,
                          random_state=0)

clf2 = DecisionTreeClassifier(max_depth=5,
                              min_samples_leaf=5,
                              min_samples_split=1,
                              criterion='entropy',
                              random_state=0)

clf3 = KNeighborsClassifier(n_neighbors=1,
                            p=2,
                            metric='minkowski')

pipe1 = Pipeline([['sc', StandardScaler()],
                  ['clf', clf1]])
pipe3 = Pipeline([['sc', StandardScaler()],
                  ['clf', clf3]])

clf_labels = ['Logistic Regression', 'Decision Tree', 'KNN']

print('10-fold cross validation:\n')
for clf, label in zip([pipe1, clf2, pipe3], clf_labels):
    scores = cross_val_score(estimator=clf,
                             X=X_train,
                             y=y_train,
                             cv=10,
                             scoring='roc_auc')
    print("ROC AUC: %0.2f (+/- %0.2f) [%s]"
               % (scores.mean(), scores.std(), label))


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Decision Tree accuracy and time elapsed caculation
Can you write Python code for it?

t0=time()
print "DecisionTree"

#dt = DecisionTreeClassifier(min_samples_split=1,random_state=99)
dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)

clf_dt=dt.fit(X_train,y_train)

print "Acurracy: ", clf_dt.score(X_test,y_test)
t1=time()
print "time elapsed: ", t1-t0


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Ensemble Learning
Can you write Python code for it?

from sklearn.base import BaseEstimator
from sklearn.base import ClassifierMixin
from sklearn.preprocessing import LabelEncoder
from sklearn.externals import six
from sklearn.base import clone
from sklearn.pipeline import _name_estimators
import numpy as np
import operator

class MajorityVoteClassifier(BaseEstimator,
                             ClassifierMixin):
    """ A majority vote ensemble classifier

    Parameters
    ----------
    classifiers : array-like, shape = [n_classifiers]
      Different classifiers for the ensemble

    vote : str, {'classlabel', 'probability'} (default='label')
      If 'classlabel' the prediction is based on the argmax of
        class labels. Else if 'probability', the argmax of
        the sum of probabilities is used to predict the class label
        (recommended for calibrated classifiers).

    weights : array-like, shape = [n_classifiers], optional (default=None)
      If a list of `int` or `float` values are provided, the classifiers
      are weighted by importance; Uses uniform weights if `weights=None`.

    """
    def __init__(self, classifiers, vote='classlabel', weights=None):

        self.classifiers = classifiers
        self.named_classifiers = {key: value for key, value
                                  in _name_estimators(classifiers)}
        self.vote = vote
        self.weights = weights

    def fit(self, X, y):
        """ Fit classifiers.

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape = [n_samples, n_features]
            Matrix of training samples.

        y : array-like, shape = [n_samples]
            Vector of target class labels.

        Returns
        -------
        self : object

        """
        if self.vote not in ('probability', 'classlabel'):
            raise ValueError("vote must be 'probability' or 'classlabel'"
                             "; got (vote=%r)"
                             % self.vote)

        if self.weights and len(self.weights) != len(self.classifiers):
           ...


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Feature transformations with ensembles of trees
Can you write Python code for it?

import numpy as np
np.random.seed(10)

import matplotlib.pyplot as plt

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomTreesEmbedding, RandomForestClassifier,
                              GradientBoostingClassifier)
from sklearn.preprocessing import OneHotEncoder
from sklearn.cross_validation import train_test_split
from sklearn.metrics import roc_curve
from sklearn.pipeline import make_pipeline

n_estimator = 10
# It is important to train the ensemble of trees on a different subset
# of the training data than the linear regression model to avoid
# overfitting, in particular if the total number of leaves is
# similar to the number of training samples
X_train, X_train_lr, y_train, y_train_lr = train_test_split(X_train,
                                                            y_train,
                                                            test_size=0.5)

# Unsupervised transformation based on totally random trees
rt = RandomTreesEmbedding(max_depth=3, n_estimators=n_estimator,
	random_state=0)

rt_lm = LogisticRegression()
pipeline = make_pipeline(rt, rt_lm)
pipeline.fit(X_train, y_train)
y_pred_rt = pipeline.predict_proba(X_test)[:, 1]
fpr_rt_lm, tpr_rt_lm, _ = roc_curve(y_test, y_pred_rt)

# Supervised transformation based on random forests
rf = RandomForestClassifier(max_depth=3, n_estimators=n_estimator)
rf_enc = OneHotEncoder()
rf_lm = LogisticRegression()
rf.fit(X_train, y_train)
rf_enc.fit(rf.apply(X_train))
rf_lm.fit(rf_enc.transform(rf.apply(X_train_lr)), y_train_lr)

y_pred_rf_lm = rf_lm.predict_proba(rf_enc.transform(rf.apply(X_test)))[:, 1]
fpr_rf_lm, tpr_rf_lm, _ = roc_curve(y_test, y_pred_rf_lm)

grd = GradientBoostingClassifier(n_estimators=n_estimator)
grd_enc = OneHotEncoder()
grd_lm = LogisticRegression()
grd.fit(X_train, y_train)
grd_enc.fit(grd.apply(X_train)[:, :, 0])
grd_lm.fit(grd_enc.transform(grd.apply(X_train_lr)[:, :, 0]), y_train_lr)

y_pred_grd_lm = grd_lm.predict_proba(
   ...


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Leveraging weak learners via adaptive boosting
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=1)

ada = AdaBoostClassifier(base_estimator=tree,
                         n_estimators=500,
                         learning_rate=0.1,
                         random_state=0)

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

ada = ada.fit(X_train, y_train)
y_train_pred = ada.predict(X_train)
y_test_pred = ada.predict(X_test)

ada_train = accuracy_score(y_train, y_train_pred)
ada_test = accuracy_score(y_test, y_test_pred)
print('AdaBoost train/test accuracies %.3f/%.3f'
      % (ada_train, ada_test))


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
PCA Decomposition
Can you write Python code for it?

target_names = ['Shares > 1400' , 'Shares < 1400']

X.values

%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

pca = PCA(n_components=2)
reduced_X = pca.fit_transform(X)

for a in [red_x, red_y,blue_x,blue_y]:
    print len(a)

red_x, red_y = [], []
blue_x, blue_y = [], []

for i in range(len(reduced_X)):
    if y[i] == 0:
        red_x.append(reduced_X[i][0])
        red_y.append(reduced_X[i][1])
    elif y[i] == 1:
        blue_x.append(reduced_X[i][0])
        blue_y.append(reduced_X[i][1])

plt.scatter(red_x, red_y, c='r', marker='x')
plt.scatter(blue_x, blue_y, c='b', marker='.')
plt.show()

%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
"""
pca = PCA(n_components=2)
X_r = pca.fit(X.values).transform(X.values)

lda = LinearDiscriminantAnalysis(n_components=2)
X_r2 = lda.fit(X.values, y.values).transform(X.values)
"""
pca = PCA(n_components=2)
X_r = pca.fit(X).transform(X)

lda = LinearDiscriminantAnalysis(n_components=2)
X_r2 = lda.fit(X, y).transform(X)

# Percentage of variance explained for each components
print('explained variance ratio (first two components): %s'
      % str(pca.explained_variance_ratio_))

plt.figure()

colors = ['blue','red']
for i in xrange(len(colors)):
    px = X_r[:, 0][y == i]
    py = X_r[:, 1][y == i]
    plt.scatter(px, py, c=colors[i])
plt.legend(target_names)
plt.title('PCA')
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')

plt.figure()

colors = ['blue','red']
for i in xrange(len(colors)):
    px = X_r2[:, 0][y == i]
    py = X_pca[:, 1][y == i]
    plt.scatter(px, py, c=colors[i])
plt.legend(target_names)
plt.title('LDA')
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')

plt.show()

"""

for c, i, target_name in zip("rb", [0, 1], target_names):
    plt.scatter(X_r[y == i, 0],...


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Plot class probabilities calculated by the VotingClassifier
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier

clf1 = LogisticRegression(random_state=123)
clf2 = RandomForestClassifier(random_state=123)
clf3 = GaussianNB()

eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2), ('gnb', clf3)],
                        voting='soft',
                        weights=[1, 1, 5])

# predict class probabilities for all classifiers
probas = [c.fit(X, y).predict_proba(X) for c in (clf1, clf2, clf3, eclf)]

# get class probabilities for the first sample in the dataset
class1_1 = [pr[0, 0] for pr in probas]
class2_1 = [pr[0, 1] for pr in probas]

# plotting

N = 4  # number of groups
ind = np.arange(N)  # group positions
width = 0.35  # bar width

fig, ax = plt.subplots()

# bars for classifier 1-3
p1 = ax.bar(ind, np.hstack(([class1_1[:-1], [0]])), width, color='green')
p2 = ax.bar(ind + width, np.hstack(([class2_1[:-1], [0]])), width, color='lightgreen')

# bars for VotingClassifier
p3 = ax.bar(ind, [0, 0, 0, class1_1[-1]], width, color='blue')
p4 = ax.bar(ind + width, [0, 0, 0, class2_1[-1]], width, color='steelblue')

# plot annotations
plt.axvline(2.8, color='k', linestyle='dashed')
ax.set_xticks(ind + width)
ax.set_xticklabels(['LogisticRegression\nweight 1',
                    'GaussianNB\nweight 1',
                    'RandomForestClassifier\nweight 5',
                    'VotingClassifier\n(average probabilities)'],
                   rotation=40,
                   ha='right')
plt.ylim([0, 1])
plt.title('Class probabilities for sample 1 by different classifiers')
plt.legend([p1[0], p2[0]], ['class 1', 'class 2'], loc='upper left')
plt.show()


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Plot classification probability
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LogisticRegression

#PCA
pca = PCA(n_components=2)
X_r = pca.fit(X).transform(X)

n_features = X_r.shape[1]

C = 1.0

# Create different classifiers. The logistic regression cannot do
# multiclass out of the box.
classifiers = {'L1 logistic': LogisticRegression(C=C, penalty='l1'),
               'L2 logistic': LogisticRegression(C=C, penalty='l2'),
               'Decision Tree': DecisionTreeClassifier(max_depth=5,min_samples_leaf=5,min_samples_split=1,random_state=99),
               'K Nearest Neighbors': KNeighborsClassifier(n_neighbors=3),
               'Random Forest' : RandomForestClassifier(max_depth=25,min_samples_leaf=2,
                                                        min_samples_split=10,n_estimators=100,n_jobs=-1)
              }

n_classifiers = len(classifiers)

plt.figure(figsize=(2 * 2, n_classifiers * 2))
plt.subplots_adjust(bottom=.2, top=.95)

xx = np.linspace(3, 9, 100)
yy = np.linspace(1, 5, 100).T
xx, yy = np.meshgrid(xx, yy)
Xfull = np.c_[xx.ravel(), yy.ravel()]

for index, (name, classifier) in enumerate(classifiers.items()):
    classifier.fit(X_r, y)

    y_pred = classifier.predict(X_r)
    classif_rate = np.mean(y_pred.ravel() == y.ravel()) * 100
    print("classif_rate for %s : %f " % (name, classif_rate))

    # View probabilities=
    probas = classifier.predict_proba(Xfull)
    n_classes = np.unique(y_pred).size
    for k in range(n_classes):
        plt.subplot(n_classifiers, n_classes, index * n_classes + k + 1)
        plt.title("Class %d" % k)
        if k == 0:
            plt.ylabel(name)
        imshow_handle = plt.imshow(probas[:, k].reshape((100, 100)),
                                   extent=(3, 9, 1, 5), origin='lower')
        plt.xticks(())
        plt.yticks(())
        idx = (y_pred == k)
        if idx.any():
            plt.scatter(X_r[idx, 0], X_r[idx, 1], marker='o', c='k')

ax = plt.axes([0.15, 0.04, 0.7,...


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Probability Calibration curves
Can you write Python code for it?

import matplotlib.pyplot as plt

from sklearn import datasets
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
                             f1_score)
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.cross_validation import train_test_split

def plot_calibration_curve(est, name, fig_index):
    """Plot calibration curve for est w/o and with calibration. """
    # Calibrated with isotonic calibration
    isotonic = CalibratedClassifierCV(est, cv=2, method='isotonic')

    # Calibrated with sigmoid calibration
    sigmoid = CalibratedClassifierCV(est, cv=2, method='sigmoid')

    # Logistic regression with no calibration as baseline
    lr = LogisticRegression(C=1., solver='lbfgs')

    fig = plt.figure(fig_index, figsize=(10, 10))
    ax1 = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
    ax2 = plt.subplot2grid((3, 1), (2, 0))

    ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
    for clf, name in [(lr, 'Logistic'),
                      (est, name),
                      (isotonic, name + ' + Isotonic'),
                      (sigmoid, name + ' + Sigmoid')]:
        clf.fit(X_train, y_train)
        y_pred = clf.predict(X_test)
        if hasattr(clf, "predict_proba"):
            prob_pos = clf.predict_proba(X_test)[:, 1]
        else:  # use decision function
            prob_pos = clf.decision_function(X_test)
            prob_pos = \
                (prob_pos - prob_pos.min()) / (prob_pos.max() - prob_pos.min())

        clf_score = brier_score_loss(y_test, prob_pos, pos_label=y.max())
        print("%s:" % name)
        print("\tBrier: %1.3f" % (clf_score))
        print("\tPrecision: %1.3f" % precision_score(y_test, y_pred))
        print("\tRecall: %1.3f" % recall_score(y_test, y_pred))
        print("\tF1: %1.3f\n" % f1_score(y_test, y_pred))

        fraction_of_positives,...


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Random Forest accuracy and time elapsed caculation
Can you write Python code for it?

t2=time()
print "RandomForest"
rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)
clf_rf = rf.fit(X_train,y_train)
print "Acurracy: ", clf_rf.score(X_test,y_test)
t3=time()
print "time elapsed: ", t3-t2


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Receiver Operating Characteristic (ROC) curve
Can you write Python code for it?

roc_auc_score(y_test,rf.predict(X_test))

%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc

predictions = rf.predict_proba(X_test)

false_positive_rate, recall, thresholds = roc_curve(y_test, predictions[:, 1])
roc_auc = auc(false_positive_rate, recall)
plt.title('Receiver Operating Characteristic')
plt.plot(false_positive_rate, recall, 'b', label='AUC = %0.2f' % roc_auc)
plt.legend(loc='lower right')

plt.plot([0, 1], [0, 1], 'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.ylabel('Recall')
plt.xlabel('Fall-out')
plt.show()


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Recursive feature elimination with cross-validation
Can you write Python code for it?

import matplotlib.pyplot as plt
from sklearn.cross_validation import StratifiedKFold
from sklearn.feature_selection import RFECV
from sklearn.datasets import make_classification

# Create the RFE object and compute a cross-validated score.

rf = RandomForestClassifier(max_depth=25,min_samples_leaf=2,min_samples_split=10,n_estimators=100,n_jobs=-1)

# The "accuracy" scoring is proportional to the number of correct classifications
rfecv = RFECV(estimator=rf, step=1, cv=StratifiedKFold(y, 2),
              scoring='accuracy')
rfecv.fit(X, y)

print("Optimal number of features : %d" % rfecv.n_features_)

# Plot number of features VS. cross-validation scores
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
plt.show()


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Tuning Models using GridSearch
Can you write Python code for it?

pipeline2 = Pipeline([
('clf', RandomForestClassifier(criterion='entropy'))
])

parameters = {
    'clf__n_estimators': (25, 50, 100),
    'clf__max_depth': (5, 25 , 50),
    'clf__min_samples_split': (1, 5, 10),
    'clf__min_samples_leaf': (1, 2, 3)
}

grid_search = GridSearchCV(pipeline2, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)

grid_search.fit(X_train, y_train)

print 'Best score: %0.3f' % grid_search.best_score_

print 'Best parameters set:'
best_parameters = grid_search.best_estimator_.get_params()

for param_name in sorted(parameters.keys()):
    print '\t%s: %r' % (param_name, best_parameters[param_name])

predictions = grid_search.predict(X_test)
print 'Accuracy:', accuracy_score(y_test, predictions)
print classification_report(y_test, predictions)


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Tuning our hyperparameters using GridSearch
Can you write Python code for it?

from sklearn.metrics import classification_report

pipeline = Pipeline([
    ('clf', DecisionTreeClassifier(criterion='entropy'))
])

parameters = {
    'clf__max_depth': (5, 25 , 50),
    'clf__min_samples_split': (1, 5, 10),
    'clf__min_samples_leaf': (1, 2, 3)
}

grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='f1')
grid_search.fit(X_train, y_train)

print 'Best score: %0.3f' % grid_search.best_score_
print 'Best parameters set:'

best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
    print '\t%s: %r' % (param_name, best_parameters[param_name])

predictions = grid_search.predict(X_test)

print classification_report(y_test, predictions)


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
Two-class AdaBoost
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_gaussian_quantiles

# Create and fit an AdaBoosted decision tree
bdt = AdaBoostClassifier(DecisionTreeClassifier(max_depth=5),
                         algorithm="SAMME",
                         n_estimators=200)

bdt.fit(X, y)

plot_colors = "br"
plot_step = 0.02
class_names = "AB"

plt.figure(figsize=(10, 5))

# Plot the decision boundaries
plt.subplot(121)
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                     np.arange(y_min, y_max, plot_step))

Z = bdt.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cs = plt.contourf(xx, yy, Z, cmap=plt.cm.Paired)
plt.axis("tight")

# Plot the training points
for i, n, c in zip(range(2), class_names, plot_colors):
    idx = np.where(y == i)
    plt.scatter(X[idx, 0], X[idx, 1],
                c=c, cmap=plt.cm.Paired,
                label="Class %s" % n)
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.legend(loc='upper right')
plt.xlabel('x')
plt.ylabel('y')
plt.title('Decision Boundary')

# Plot the two-class decision scores
twoclass_output = bdt.decision_function(X)
plot_range = (twoclass_output.min(), twoclass_output.max())
plt.subplot(122)
for i, n, c in zip(range(2), class_names, plot_colors):
    plt.hist(twoclass_output[y == i],
             bins=10,
             range=plot_range,
             facecolor=c,
             label='Class %s' % n,
             alpha=.5)
x1, x2, y1, y2 = plt.axis()
plt.axis((x1, x2, y1, y2 * 1.2))
plt.legend(loc='upper right')
plt.ylabel('Samples')
plt.xlabel('Score')
plt.title('Decision Scores')

plt.tight_layout()
plt.subplots_adjust(wspace=0.35)
plt.show()


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
cross validation for DT
Can you write Python code for it?

tt0=time()
print "cross result========"
scores = cross_validation.cross_val_score(dt, X,y, cv=5)
print scores
print scores.mean()
tt1=time()
print "time elapsed: ", tt1-tt0
print "\n"


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
cross validation for KNN
Can you write Python code for it?

tt6=time()
print "cross result========"
scores = cross_validation.cross_val_score(knn, X,y, cv=5)
print scores
print scores.mean()
tt7=time()
print "time elapsed: ", tt7-tt6
print "\n"

from sklearn.cross_validation import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn import grid_search

knn = KNeighborsClassifier()

parameters = {'n_neighbors': (10, 15, 25)}

grid = grid_search.GridSearchCV(knn, parameters, n_jobs=-1, verbose=1, scoring='accuracy')

grid.fit(X_train, y_train)

print 'Best score: %0.3f' % grid.best_score_

print 'Best parameters set:'
best_parameters = grid.best_estimator_.get_params()

for param_name in sorted(parameters.keys()):
    print '\t%s: %r' % (param_name, best_parameters[param_name])

predictions = grid.predict(X_test)
print classification_report(y_test, predictions)


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
cross validation for RF
Can you write Python code for it?

tt2=time()
print "cross result========"
scores = cross_validation.cross_val_score(rf, X,y, cv=5)
print scores
print scores.mean()
tt3=time()
print "time elapsed: ", tt3-tt2
print "\n"


I am working on the file "Ensemble Learning/Classifying Default of Credit Card Clients.ipynb".
The first task is:
cross-validation for NB
Can you write Python code for it?

tt4=time()
print "cross result========"
scores = cross_validation.cross_val_score(nb, X,y, cv=5)
print scores
print scores.mean()
tt5=time()
print "time elapsed: ", tt5-tt4
print "\n"


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
After the grid search has completed, we can print the different hyperparameter
value combinations and the average ROC AUC scores computed via 10-fold
cross-validation. The code is as follows:
Can you write Python code for it?

print('Best parameters: %s' % grid.best_params_)
print('Accuracy: %.2f' % grid.best_score_)


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Although the training accuracies of the decision tree and bagging classifier are similar on the training set (both 1.0), we can see that the bagging classifier has a slightly better generalization performance as estimated on the test set.
In practice, more complex classification tasks and datasets' high dimensionality can easily lead to overfitting in single decision trees and this is where the bagging algorithm can really play out its strengths. Finally, we shall note that the bagging algorithm can be an effective approach to reduce the variance of a model. However, bagging is ineffective in reducing model bias, which is why we want to choose an ensemble of classifiers with low bias, for example, unpruned decision trees.
Leveraging weak learners via adaptive boosting
In this section about ensemble methods, we will discuss boosting with a special focus on its most common implementation, AdaBoost (short for Adaptive Boosting).
In boosting, the ensemble consists of very simple base classifiers, also often referred to as weak learners, that have only a slight performance advantage over random guessing. A typical example of a weak learner would be a decision tree stump. The key concept behind boosting is to focus on training samples that are hard to classify, that is, to let the weak learners subsequently learn from misclassified training samples to improve the performance of the ensemble. In contrast to bagging, the initial formulation of boosting, the algorithm uses random subsets of training samples drawn from the training dataset without replacement. The original boosting procedure is summarized in four key steps as follows:

Draw a random subset of training samples d1 without replacement from the training set D to train a weak learner C1.
Draw second random training subset d2 without replacement from the training set and add 50 percent of the samples that were previously misclassified to train a weak learner C2.
Find the training samples d3 in the training set D on which C1 and C2 disagree to train a third...
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=1)

ada = AdaBoostClassifier(base_estimator=tree,
                         n_estimators=500,
                         learning_rate=0.1,
                         random_state=0)

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
As we can see in the resulting ROC, the ensemble classifier also performs well on
the test set (ROC AUC = 0.86).
Before we tune the individual classifier parameters for ensemble classification, let's call the get_params method to get a basic idea of how we can access the individual parameters inside a GridSearch object:
Can you write Python code for it?


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
As we can see, the decision tree stump seems to overfit the training data in contrast with the unpruned decision tree that we saw in the previous section.
Can you write Python code for it?

ada = ada.fit(X_train, y_train)
y_train_pred = ada.predict(X_train)
y_test_pred = ada.predict(X_test)

ada_train = accuracy_score(y_train, y_train_pred)
ada_test = accuracy_score(y_test, y_test_pred)
print('AdaBoost train/test accuracies %.3f/%.3f'
      % (ada_train, ada_test))


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
As we can see, we get the best cross-validation results when we choose a higher n_neighbors (n = 20) whereas the tree depth does not seem to affect the performance at all, suggesting that a decision stump is sufficient to separate the data. To remind ourselves that it is a bad practice to use the test dataset more than once for model evaluation, we are not going to estimate the generalization performance of the tuned hyperparameters in this section. We will move on swiftly to an alternative approach for ensemble learning: bagging.
Bagging -- Building an ensemble of classifiers from bootstrap samples
Bagging is an ensemble learning technique that is closely related to the MajorityVoteClassifier that we implemented in the previous section,
as illustrated in the following diagram:
<img src = "images/bagging.png">
However, instead of using the same training set to fit the individual classifiers in the ensemble, we draw bootstrap samples (random samples with replacement) from the initial training set, which is why bagging is also known as bootstrap aggregating.
Can you write Python code for it?

X = df[features]
y = df['num']

X.shape , y.shape


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
As you can see the accuracies o our individual classifiers are almost same and are on the high side. Now let's move on to the more exciting part and combine the individual classifiers for majority rule voting in our MajorityVoteClassifier:
Can you write Python code for it?

# Majority Rule (hard) Voting

mv_clf = MajorityVoteClassifier(
                classifiers=[clf2, pipe3])

clf_labels += ['Majority Voting']
all_clf = [clf2, pipe3, mv_clf]

for clf, label in zip(all_clf, clf_labels):
    scores = cross_val_score(estimator=clf,
                             X=X_train,
                             y=y_train,
                             cv=10,
                             scoring='roc_auc')
    print("ROC AUC: %0.2f (+/- %0.2f) [%s]"
               % (scores.mean(), scores.std(), label))


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Based on the values returned by the get_params method, we now know how to access the individual classifier's attributes. Let's now tune the decision tree depth via a grid search for demonstration purposes. The code is as follows:
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV

params = {'decisiontreeclassifier__max_depth': [1,2],
          'pipeline__clf__n_neighbors': [5,15,20]}

grid = GridSearchCV(estimator=mv_clf,
                    param_grid=params,
                    cv=10,
                    scoring='roc_auc')
grid.fit(X_train, y_train)

for params, mean_score, scores in grid.grid_scores_:
    print("%0.3f+/-%0.2f %r"
            % (mean_score, scores.std() / 2, params))


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Decision Tree accuracy and time elapsed caculation
Can you write Python code for it?

from time import time
t0=time()
print ("DecisionTree")

dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)
# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)

clf_dt=dt.fit(X_train,y_train)

print ("Acurracy: ", clf_dt.score(X_test,y_test))
t1=time()
print ("time elapsed: ", t1-t0)

tt0=time()
print ("cross result========")
scores = cross_validation.cross_val_score(dt, X, y, cv=5)
print (scores)
print (scores.mean())
tt1=time()
print ("time elapsed: ", tt1-tt0)


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Feature importances with forests of trees
This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability.
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

from sklearn.ensemble import ExtraTreesClassifier

# Build a classification task using 3 informative features

# Build a forest and compute the feature importances
forest = ExtraTreesClassifier(n_estimators=250,
                              random_state=0)

forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

for f in range(5):
    print("%d. feature %d - %s (%f)" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))

best_features = []
for i in indices[:5]:
    best_features.append(features[i])

# Plot the top 5 feature importances of the forest
plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')
plt.title("Feature importances")
plt.bar(range(5), importances[indices][:5],
       color="r",  yerr=std[indices][:5], align="center")
plt.xticks(range(5), best_features)
plt.xlim([-1, 5])
plt.show()


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
KNN accuracy and time elapsed caculation
Can you write Python code for it?

t6=time()
print ("KNN")
knn = KNeighborsClassifier()
clf_knn=knn.fit(X_train, y_train)
print ("Acurracy: ", clf_knn.score(X_test,y_test) )
t7=time()
print ("time elapsed: ", t7-t6)

tt6=time()
print ("cross result========")
scores = cross_validation.cross_val_score(knn, X, y, cv=5)
print (scores)
print (scores.mean())
tt7=time()
print ("time elapsed: ", tt7-tt6)


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Naive Bayes accuracy and time elapsed caculation
Can you write Python code for it?

t4=time()
print ("NaiveBayes")
nb = BernoulliNB()
clf_nb=nb.fit(X_train,y_train)
print ("Acurracy: ", clf_nb.score(X_test,y_test))
t5=time()
print ("time elapsed: ", t5-t4)

tt4=time()
print ("cross result========")
scores = cross_validation.cross_val_score(nb, X, y, cv=5)
print (scores)
print (scores.mean())
tt5=time()
print ("time elapsed: ", tt5-tt4)


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Next we will calculate the accuracy score of the prediction on the training and test dataset to compare the performance of the bagging classifier to the performance of a single unpruned decision tree. Based on the accuracy values, the unpruned decision tree predicts all class labels of the training samples correctly; however, the substantially lower test accuracy indicates high variance (overfitting) of the model:
Can you write Python code for it?

from sklearn.metrics import accuracy_score

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

bag = bag.fit(X_train, y_train)
y_train_pred = bag.predict(X_train)
y_test_pred = bag.predict(X_test)

bag_train = accuracy_score(y_train, y_train_pred)
bag_test = accuracy_score(y_test, y_test_pred)
print('Bagging train/test accuracies %.3f/%.3f'
      % (bag_train, bag_test))


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Now we can move on to classification of our data.
Can you write Python code for it?

features = df.columns[:-1]

features

X = df[features]
y = df['num']

y.unique()


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Our data contains 6 row with missing values. These values are represented by "?". So first we replace these "?" with NaN and then drop all rows which contain NaNs. We can simply achive this by doing the following:
Can you write Python code for it?

df.replace("?",np.NaN,inplace=True)

df.dropna(axis=0, inplace=True, how='any')

df = df.reset_index(drop=True)

df.head()


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
Putting everything together, let's now implement a MajorityVoteClassifier
in Python:
Can you write Python code for it?

from sklearn.base import BaseEstimator
from sklearn.base import ClassifierMixin
from sklearn.preprocessing import LabelEncoder
from sklearn.externals import six
from sklearn.base import clone
from sklearn.pipeline import _name_estimators
import numpy as np
import operator

class MajorityVoteClassifier(BaseEstimator,
                             ClassifierMixin):
    """ A majority vote ensemble classifier

    Parameters
    ----------
    classifiers : array-like, shape = [n_classifiers]
      Different classifiers for the ensemble

    vote : str, {'classlabel', 'probability'} (default='label')
      If 'classlabel' the prediction is based on the argmax of
        class labels. Else if 'probability', the argmax of
        the sum of probabilities is used to predict the class label
        (recommended for calibrated classifiers).

    weights : array-like, shape = [n_classifiers], optional (default=None)
      If a list of `int` or `float` values are provided, the classifiers
      are weighted by importance; Uses uniform weights if `weights=None`.

    """
    def __init__(self, classifiers, vote='classlabel', weights=None):

        self.classifiers = classifiers
        self.named_classifiers = {key: value for key, value
                                  in _name_estimators(classifiers)}
        self.vote = vote
        self.weights = weights

    def fit(self, X, y):
        """ Fit classifiers.

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape = [n_samples, n_features]
            Matrix of training samples.

        y : array-like, shape = [n_samples]
            Vector of target class labels.

        Returns
        -------
        self : object

        """
        if self.vote not in ('probability', 'classlabel'):
            raise ValueError("vote must be 'probability' or 'classlabel'"
                             "; got (vote=%r)"
                             % self.vote)

        if self.weights and len(self.weights) != len(self.classifiers):
           ...


I am working on the file "Ensemble Learning/Ensemble Learning to Classify Patients with Heart Disease.ipynb".
The first task is:
SVM accuracy and time elapsed caculation¶
Can you write Python code for it?

t7=time()
print ("SVM")

svc = SVC()
clf_svc=svc.fit(X_train, y_train)
print ("Acurracy: ", clf_svc.score(X_test,y_test) )
t8=time()
print ("time elapsed: ", t8-t7)

tt7=time()
print ("cross result========")
scores = cross_validation.cross_val_score(svc, X,y, cv=5)
print (scores)
print (scores.mean())
tt8=time()
print ("time elapsed: ", tt7-tt6)


I am working on the file "Ensemble learning - stacked generalization.ipynb".
The first task is:
Cross validation will be carried out by means of stratified k-fold.
Can you write Python code for it?

skf = StratifiedKFold(y_test, n_folds=3)


I am working on the file "Ensemble learning - stacked generalization.ipynb".
The first task is:
Models
I'm using decision trees (cart) as level-0 classifiers
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier

n_models = 3
clfs = [DecisionTreeClassifier()] * n_models


I am working on the file "Ensemble learning - stacked generalization.ipynb".
The first task is:
Parallel version
Cross validation does not require any form of comunication between the models being trained. This makes stacked generalization a good candidate for parallelization.
In this section I'll be using joblib, a frontend to the multiprocessing  framework, to paralleize the training/testing of level-0 models as well as the generation of level 1 data. The results of parallel computations are written to shared, mem-mapped, ndarrays. In general this is not a good idea; numpy does not provide atomic operations and writes to shared segments can lead to data corruption. However, in this specific case we can rely on the fact that each classifier $k$ and fold $j$ are allocated exclusive segments of the shared ndarrays.
Can you write Python code for it?

from joblib import Parallel, delayed
from joblib import load, dump, cpu_count

import tempfile
import shutil
import os
import numpy as np

mmap_dir = tempfile.mkdtemp()


I am working on the file "Ensemble.ipynb".
The first task is:
---------------------------------------------
Ensemble ALL the submissions
---------------------------------------------
Find the average probability for all IDs
Can you write Python code for it?

from collections import defaultdict

aggregates = defaultdict(list)
averages   = defaultdict(list)

# 1. collect the probabilities for each ID from all the submission files
# ======================================================================
for file_name in file_list:
    with open(file_name, 'r') as f:
        lines = f.read().splitlines()
        for line in lines:
            ID,prob = line.split(',')
            if ID == '': continue
            aggregates[ID].append(prob)

# 2. find the average of all the probabilities for each ID
# ========================================================
averages.update((ID, np.mean(map(float, probs))) for ID, probs in aggregates.items())

aggregates['1'],averages['1']

len(aggregates),len(averages)


I am working on the file "Ensemble.ipynb".
The first task is:
---------------------------------------------------------------
Ensemble the least-correlated submissions
---------------------------------------------------------------
Create a dataframe with one column per submission
Can you write Python code for it?

from os.path import split
corr_table = pd.read_csv(file_list_all[0],names=['id',split(file_list_all[0])[1][11:-4]],header=0,index_col=0)
corr_table.head()

for file_path in file_list_all[1:]:
    temp = pd.read_csv(file_path,names=['id',split(file_path)[1][11:-4]],header=0,index_col=0)
    corr_table[temp.columns[0]] = temp[[temp.columns[0]]]
corr_table.head()


I am working on the file "Ensemble.ipynb".
The first task is:
AFTER
Can you write Python code for it?

# why do it more than once? For some reason it doesn't work if only run once. Who knows?
# ======================================================================================
for i in range(3):
    for file_name in file_list:
        if 'Format'   in file_name: file_list.remove(file_name)
        if 'Ensemble' in file_name: file_list.remove(file_name)
        if 'ensemble' in file_name: file_list.remove(file_name)

file_list.sort(key=lambda x: x[26:32])

from copy import copy
file_list_all = copy(file_list)

file_list


I am working on the file "Ensemble.ipynb".
The first task is:
Create a submission file of the ensemble of averages
Can you write Python code for it?

# f = open("../submissions/submission_EnsembleOfAveragesALL.csv", "w")

# f.write(",Made Donation in March 2007\n")
# for ID in id_list:
#     f.write("{},{}\n".format(ID, averages[ID]))

# f.close()


I am working on the file "Ensemble.ipynb".
The first task is:
Display the correlations among the submissions
Can you write Python code for it?

import seaborn as sns

# Compute the correlation matrix
corr_matrix = corr_table.corr()

# Generate a mask for the upper triangle
mask = np.zeros_like(corr_matrix, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(220, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr_matrix, mask=mask, cmap=cmap, vmax=.9,
            square=True, xticklabels=4, yticklabels=3,
            linewidths=.5, cbar_kws={"shrink": .5}, ax=ax)
plt.show()


I am working on the file "Ensemble.ipynb".
The first task is:
Find the least-correlated pairs of submissions
Can you write Python code for it?

corr_threshold = 0.20

indices = np.where(corr_matrix < corr_threshold)
indices = [(corr_matrix.index[x], corr_matrix.columns[y], corr_matrix.ix[x,y]) for x, y in zip(*indices)
                                        if x != y and x < y]

from operator import itemgetter
indices.sort(key=itemgetter(2))
len(indices),indices

least_corr = set(set(['../submissions/submission_'+a+'.csv' for a,b,c in indices]).\
             union(set(['../submissions/submission_'+b+'.csv' for a,b,c in indices])))

len(least_corr), least_corr

from collections import defaultdict

aggregates = defaultdict(list)
averages   = defaultdict(list)

# 1. collect the probabilities for each ID from all the submission files
# ======================================================================
for file_name in least_corr:
    with open(file_name, 'r') as f:
        lines = f.read().splitlines()
        for line in lines:
            ID,prob = line.split(',')
            if ID == '': continue
            aggregates[ID].append(prob)

# 2. find the average of all the probabilities for each ID
# ========================================================
averages.update((ID, np.mean(map(float, probs))) for ID, probs in aggregates.items())

aggregates['1'],averages['1']

# f = open("../submissions/submission_EnsembleOfAveragesLeastCorr.csv", "w")

# f.write(",Made Donation in March 2007\n")
# for ID in id_list:
#     f.write("{},{}\n".format(ID, averages[ID]))

# f.close()


I am working on the file "Ensemble.ipynb".
The first task is:
Get the list of submission files

remove the example file

and all ensembles

BEFORE
Can you write Python code for it?

file_list = get_filepaths('../submissions')
file_list


I am working on the file "Ensemble.ipynb".
The first task is:
The test set has duplicates so we get the list of IDs in the sample file in order
Can you write Python code for it?

id_list = []
with open('../submissions/Submission_Format.csv', 'r') as f:
    lines = f.read().splitlines()
    for line in lines:
        ID,prob = line.split(',')
        if ID == '': continue
        id_list.append(ID)

def get_filepaths(directory):
    """
    This function will generate the file names in a directory
    tree by walking the tree either top-down or bottom-up. For each
    directory in the tree rooted at directory top (including top itself),
    it yields a 3-tuple list (dirpath, dirnames, filenames).
    """
    import os

    file_paths = []  # List which will store all of the full filepaths.

    # Walk the tree.
    for root, directories, files in os.walk(directory):
        for filename in files:
            # Join the two strings in order to form the full filepath.
            filepath = os.path.join(root, filename)
            file_paths.append(filepath)  # Add it to the list.

    return file_paths


I am working on the file "Ensemble_learning.ipynb".
The first task is:
Dynamic voting
Another idea is as follows
1. Take the image to classify
2. Find its K nearest neighbors
3. Classify the neighbor using all classifiers
4. Use weighted voting
Adaboost
In case of our classiers we cannot use boosting techniques.
However, one can try with svm.SVC(kernel='linear', probability=True)
in order to boost the model.
```python
base_clf = clf['resnet50-keras']
from sklearn import svm
base_clf = svm.SVC(probability=True,kernel='linear')
n_training = 1000 # TODO: try with bigger testing data
X_train = data['resnet50-keras']['features_training'][:n_training]
y_train = data['resnet50-keras']['labels_training'][:n_training]
base_clf.fit( X_train, y_train )
base_clf.score( data['resnet50-keras']['features_testing'], data['resnet50-keras']['labels_testing'] )
from sklearn.ensemble import AdaBoostClassifier
boosted_model = AdaBoostClassifier(base_estimator = base_clf)
boosted_model.fit( X_train, y_train )
boosted_model.score( data['resnet50-keras']['features_testing'], data['resnet50-keras']['labels_testing'] )
```
Adaboost did not boost.
<pre>
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
  max_iter=-1, probability=True, random_state=None, shrinking=True,
  tol=0.001, verbose=False)

0.83030000000000004

AdaBoostClassifier(algorithm='SAMME.R',
          base_estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
  max_iter=-1, probability=True, random_state=None, shrinking=True,
  tol=0.001, verbose=False),
          learning_rate=1.0, n_estimators=50, random_state=None)

0.78369999999999995
</pre>

Model stacking
Here we present the reults with stacking our two best models.
One can try with stacking more models.
Can you write Python code for it?


I am working on the file "Ensemble_learning.ipynb".
The first task is:
Stacking model accuracy is 90.61%, which is not better than the accuracy gained by the base models.
Let us remind the accuracies of the base models:
Can you write Python code for it?

model1.score( X1_testing, y_testing ),  model2.score( X2_testing, y_testing )


I am working on the file "Ensemble_learning.ipynb".
The first task is:
This is the moment to look at the testing data from CIFAR10 database.
Can you write Python code for it?

y_testing = data[model2_name]['labels_testing']  # one can take any model

np.sum( y_predictions == y_testing )


I am working on the file "Ensemble_learning.ipynb".
The first task is:
We are not looking at the testing data from CIFAR10 database, yet.
Can you write Python code for it?

X1_training = data[model1_name]['features_training']
y1_training = data[model1_name]['labels_training']

X2_training = data[model2_name]['features_training']
y2_training = data[model2_name]['labels_training']

n_training = X1_training.shape[0]


I am working on the file "Ensemble_learning.ipynb".
The first task is:
We build training data for the stacked model
Can you write Python code for it?

X_cross_training = np.zeros( (n_training, n_models) )

X1_training.shape,  X2_training.shape

# y_cross_training[ [3,4,5], 0 ] = [ 6,7,2 ]


I am working on the file "Entity Resolution Workshop.ipynb".
The first task is:
Similarity Scores
Links to information about distance metrics:

Implementing the Five Most Popular Similarity Measures in Python
Scikit-Learn Distance Metric
Python Distance Library

Numeric distances are fairly easy, but can be record specific (e.g. phone numbers can compare area codes, city codes, etc. to determine similarity). We will compare text similarity in this section:
Can you write Python code for it?

# Typographic Distances

print distance.levenshtein("lenvestein", "levenshtein")
print distance.hamming("hamming", "hamning")

# Compare glyphs, syllables, or phonemes
t1 = ("de", "ci", "si", "ve")
t2 = ("de", "ri", "si", "ve")
print distance.levenshtein(t1, t2)

# Sentence Comparison
sent1 = "The quick brown fox jumped over the lazy dogs."
sent2 = "The lazy foxes are jumping over the crazy Dog."

print distance.nlevenshtein(sent1.split(), sent2.split(), method=1)

# Normalization
print distance.hamming("fat", "cat", normalized=True)
print distance.nlevenshtein("abc", "acd", method=1)  # shortest alignment
print distance.nlevenshtein("abc", "acd", method=2)  # longest alignment

# Set measures
print distance.sorensen("decide", "resize")
print distance.jaccard("decide", "resize")


I am working on the file "Entity Resolution Workshop.ipynb".
The first task is:
Similarity Vectors
Can you write Python code for it?

def similarity(prod1, prod2):
    """
    Returns a similarity vector of match scores:
    [name_score, description_score, manufacturer_score, price_score]
    """
    pair  = (prod1, prod2)
    names = [r.get('name', None) or r.get('title', None) for r in pair]
    descr = [r.get('description') for r in pair]
    manuf = [r.get('manufacturer') for r in pair]
    price = [float(r.get('price')) for r in pair]

    return [
        normalized_jaccard(*names),
        normalized_jaccard(*descr),
        normalized_jaccard(*manuf),
        abs(1.0/(1+ (price[0] - price[1]))),
    ]

print similarity(X, Y)


I am working on the file "EntityExtractor/EntityExtractor.ipynb".
The first task is:
Entity Extraction
Parts of Speech Tagging
Can you write Python code for it?

# Viewing the parts of speech tag for each token in the given text
pos_tags =  [nltk.pos_tag(token) for token_list in tokens for token in token_list]
pos[:20]        # View top few pos tags


I am working on the file "EntityExtractor/EntityExtractor.ipynb".
The first task is:
NNP - Noun Proper Plural tag with respec to NLP library. The below command provides description of each POS tag followed by the nltk library
Can you write Python code for it?


I am working on the file "EntityExtractor/EntityExtractor.ipynb".
The first task is:
Name Entity Chunks
The below command assigns name entity to each token in the given document. The commonly used type of named entities are
ORGANIZATION, PERSON, LOCATION, DATE, TIME, MONEY, PERCENT, FACILITY, GPE
Can you write Python code for it?

chunks = ne_chunk(pos)
print (chunks.__repr__())


I am working on the file "EntityExtractor/EntityExtractor.ipynb".
The first task is:
Task 2: Entity Extraction
Tokenizing Words
Can you write Python code for it?

tokens = [nltk.word_tokenize(sent[0]) for sent in sentences]
tokens[:6]         # View the first few tokens

tokens[0]


I am working on the file "Enums.ipynb".
The first task is:
Alternative way to create an Enum
There is an alternative way to create and Enum, that matches Python's NamedTuple:
python
Colour = Enum('Colour', 'red, green')
Try it below:
Can you write Python code for it?

SecondEnum = Enum('SecondEnum', 'first, second, third')
print(SecondEnum.first)