I am working on the file "Final.ipynb".
The first task is:
Here we run the regression on 1/interarrival times using the various grids generated. Once more we take advantage of parallel computing method to speed up significantly the algorithm. The best classifier, according to the coefficient of determination measure, is once more added into results_dic dictionary.
Can you write Python code for it?

# ------------------------------------------
# GRID 1/INTERARRIVAL REGRESSION
# THIS CELL TAKES ABOUT 10-20 MIN TO RUN
# ------------------------------------------

best_grid_prior = []
best_grid_post = []

# define the intervals
# intervals = [0.05, 0.1,0.2, 0.3, 0.4,0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.5, 2.0]
intervals = [0.8,0.9, 1.0, 1.5]

   # split randomely the letters in batch for the various threads
all_batch = split_in_batch(intervals)

# Vary the standardization and find optimum
for standardization in ['None','scaler','MinMaxScaler'] :
	# parallelize the loop of interval
	threads = []
	lock = threading.Lock()
	for thread_id in range(num_threads):
		interval = all_batch[thread_id]
		t = threading.Thread(target = do_grid_interarrival_regression, \
			args = (eq_df, welldf, interval, lock ,5,standardization))
		threads.append(t)

	map(lambda t:t.start(), threads)
	map(lambda t: t.join(), threads)

prior_dic, post_dic = print_best_score_append_dictionary('grid')
regression_grid_interarrival = {'prior': prior_dic, 'post': post_dic }
results_dic['regression_grid_interarrival'] =  regression_grid_interarrival
print results_dic


I am working on the file "Final.ipynb".
The first task is:
Here we run the ridge regression on a grid. We loop over a range of grid sizes. Oklahoma state is splitted into 10 to 3100 cells. In each case we do the ridge regression and find the best classifier using the coefficient of determination as a metric. Generating and training the classifier on various grids is an embarrassingly parrallel process. Therefore we use the multithreading module from python to do this. Each thread has a couple of grids to generate and do the regression for. We use a lock to append the results into the global lists best_grid_prior and best_grid_post. This ensure that each thread is not writing at the same time into these shared two lists.
Can you write Python code for it?

# ------------------------------------------
# GRID REGRESSION
# RUNNING THIS CELL TAKES A WHILE ~ 15 MIN
# ------------------------------------------

best_grid_prior = []
best_grid_post = []

# define the intervals
intervals = [0.05, 0.1,0.2, 0.3, 0.4,0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.5, 2.0]
# intervals = [0.4, 0.5, 0.8,0.9, 1.0, 1.5]

   # split randomely the letters in batch for the various threads
all_batch = split_in_batch(intervals)

# Vary the standardization and find optimum
for standardization in ['None','scaler','MinMaxScaler'] :
	# parallelize the loop of interval
	threads = []
	lock = threading.Lock()
	for thread_id in range(num_threads):
		interval = all_batch[thread_id]
		t = threading.Thread(target = do_grid_regression, \
			args = (eq_df, welldf, interval, lock ,5,standardization))
		threads.append(t)

	map(lambda t:t.start(), threads)
	map(lambda t: t.join(), threads)

prior_dic, post_dic = print_best_score_append_dictionary('grid')
grid_regression_dic = {'prior': prior_dic, 'post': post_dic }
results_dic['grid_regression'] = grid_regression_dic
print results_dic


I am working on the file "Final.ipynb".
The first task is:
Here we show the distribution of earthquakes in Oklahoma before 2010.
Can you write Python code for it?

b42010 = eq_df[(eq_df.year_float<2010)]
plot_earthquakes(b42010)


I am working on the file "Final.ipynb".
The first task is:
Here we use a slightly more intelligent split of the labeled data (with StratifiedShuffleSplit) to attempt to achieve a higher accuracy
Can you write Python code for it?

ss = cross_validation.StratifiedShuffleSplit(classes, 5, test_size = 1./3.)
scores = cross_validation.cross_val_score(clf, labeled_svm, classes, cv=ss)
print("Accuracy = {} +- {}".format(scores.mean(),scores.std()))


I am working on the file "Final.ipynb".
The first task is:
However, the above plot shows that while the model does better on the pre-2010 data, the post-2010 is still not well fit with a Poisson Process.
4. Regression Models
In this section we compare various regression models that try to predict the number of earthquakes based on the number of wells and the volume injected in disposal wells. First we filter the earthquakes catalog and keep only the events of magnitude higher than 3 (the ones felt by people). We also initialize a dictionary results_dic in which we will store the results of all the regressions.
Can you write Python code for it?

# ------------------------------------------
# THE REGRESSION SECTION HAS SOME CODES THAT TAKE MORE THAN 40-45 MINUTES TO RUN,
# EVEN WITH THE MULTITHREADING TECHNIQUE USED
# YOU CAN RUN THE WHOLE SECTION BY RUNNING THE FOLLWING COMMAND:
# run grid_regression_sklearn.py
# however this won't plot the results from the clustering algorithm
# (see details later)
# ------------------------------------------

# ------------------------------------------
# LOAD DATAFRAMES
# ------------------------------------------
# Load the earthquakes dataframe
eq_df = pd.DataFrame.from_csv('./tempdata/earthquakes_catalog_treated.csv',sep = '|')
# filter to keep only magnitude >= 3
eq_df  = eq_df[eq_df.prefmag >= 3.0]
# for ease add column year
eq_df['year'] = map(lambda x: int(x), eq_df['year_float'])

# Load the wells dataframe.
welldf = pd.DataFrame.from_csv('tempdata/wells_data.csv')

# define the number of threads for the computation
# please optimize according to your number of cores
num_threads = 4

# initialize the dictionary in which we store the results of all  the regressions
results_dic= {}


I am working on the file "Final.ipynb".
The first task is:
If anything, we would expect the type 3 class (beta Lyrae) to be classified as a type 1 so it is interesing to see that they split evenly. However, we have such low numbers that it's probably not worth further analysis at this point.
Sample plots
Plot an example of the different types (1 and 2) for three features. Incorrectly classified types are highlighted
Can you write Python code for it?

# Get our indices for the correctly and incorrectly classified types

correct_idx = test_all[:,1] == test_all[:,2]
incorrect_idx = test_all[:,1] != test_all[:,2]

# Separate the correctly classified types
type_1_correct = test_all[test_all[correct_idx,1] == 1]
type_2_correct = test_all[test_all[correct_idx,1] == 2]
type_3_correct = test_all[test_all[correct_idx,1] == 3]

# Separate the incorrectly classified types
type_1_incorrect = test_all[test_all[incorrect_idx,1] == 1]
type_2_incorrect = test_all[test_all[incorrect_idx,1] == 2]
type_3_incorrect = test_all[test_all[incorrect_idx,1] == 3]

print len(type_1_correct)
print len(type_2_correct)
print len(type_3_correct)

print len(type_1_incorrect)
print len(type_2_incorrect)
print len(type_3_incorrect)

pl.clf()
fig = pl.figure(figsize = (10, 10))
ax = fig.gca(projection='3d')

plot_list = ['Skew', 'Median', 'IQR']

# Correct class type indexes
type_1_idx = np.where(test_all[correct_idx, 1] == 1)
type_2_idx = np.where(test_all[correct_idx, 1] == 2)
type_3_idx = np.where(test_all[correct_idx, 1] == 3)

# Incorrect class type indexes
type_1_idx_x = np.where(test_all[incorrect_idx, 1] != 1)
type_2_idx_x = np.where(test_all[incorrect_idx, 1] != 2)
type_3_idx_x = np.where(test_all[incorrect_idx, 1] != 3)

# Type 1 Correct
param1 = test_all[type_1_idx,features_dict.get(plot_list[0])]
param2 = test_all[type_1_idx,features_dict.get(plot_list[1])]
param3 = test_all[type_1_idx,features_dict.get(plot_list[2])]
ax.scatter(param1, param2, param3, marker='.', c='r', edgecolors='None', alpha=0.35, s=40, label="Type 1 Correct")

# Type 2 Correct
param1 = test_all[type_2_idx,features_dict.get(plot_list[0])]
param2 = test_all[type_2_idx,features_dict.get(plot_list[1])]
param3 = test_all[type_2_idx,features_dict.get(plot_list[2])]
ax.scatter(param1, param2, param3, marker='.', c='b', edgecolors='None', alpha=0.35, s=40, label="Type 2 Correct")

# Type 1 Not Correct
param1 = test_all[type_1_idx_x,features_dict.get(plot_list[0])]
param2 =...


I am working on the file "Final.ipynb".
The first task is:
In the first plot above, the number of water disposal wells in Oklahoma are plotted by year. In the second plot, the total amount of water gallons disposed is plotted by year. The largest amount of water was disposed between 2006 and 2010. However, we believe that data in the few past years have not been reported or is not currently available to public use.
1.3 Visualization of the earthquakes
Can you write Python code for it?

#REQUIRES INSTALLATION conda install basemap
#received assistance from https://peak5390.wordpress.com/2012/12/08/matplotlib-basemap-tutorial-plotting-points-on-a-simple-map/
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np

def plot_earthquakes(df):
    map = Basemap(llcrnrlon=int(min(df.longitude))-1, llcrnrlat=int(min(df.latitude))-1,
                  urcrnrlon=int(max(df.longitude))+1, urcrnrlat=int(max(df.latitude))+1)
    map.plot(df.longitude.values, df.latitude.values, 'ro', markersize=2)
    map.drawmapboundary(fill_color='white')
    map.drawcoastlines()
    map.drawstates()
    map.drawcounties()
    plt.plot()
    plt.show()


I am working on the file "Final.ipynb".
The first task is:
ROC Curve
Here we determine which classifier was the "best" via an ROC Curve.
NOTE: We could have just performed the exhaustive grid search above and it would have automatically compared all of these and output the absolute best, but because we wanted to generate an ROC we performed them separately.
Can you write Python code for it?

def plot_roc(classifier_list=None):
    # Plot the ROC for each kernel type
    pl.clf()
    fig = pl.figure()
    ax = fig.add_subplot(1,1,1)

    # Add a line for each kernel
    for classifier in classifier_list:
        fpr = classifier['fpr']
        tpr = classifier['tpr']
        auc = classifier['auc']
        ax.plot (fpr, tpr, label='{} (area = {:0.2f})'.format(classifier.get('kernel')[0], auc))

    plt.legend(loc=4)
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic Curve')
    pl.show ()

plot_roc(classifier_list=best_svcs)


I am working on the file "Final.ipynb".
The first task is:
See if there is a better training size that we can use
Can you write Python code for it?

step = 100
stop = int(len(labeled_svm) * (2./3.))
Ns = np.arange(500, stop , step)

#print "Attempted N sizes = {}".format(Ns)
scores = np.zeros(len(Ns))
stds = np.zeros(len(Ns))
for i in range(len(Ns)):
    N = Ns[i]
    ss = cross_validation.StratifiedShuffleSplit(classes, 5, test_size = 1./3., train_size = N)
    scores_i = cross_validation.cross_val_score(clf, labeled_svm, classes, cv=ss)
    scores[i] = scores_i.mean()
    stds[i] = scores_i.std()

# Get our optimal n
optimal_n = Ns[np.argmax(scores)]

# Plot our results, including optimal n
pl.clf()
fig = pl.figure()
ax = fig.add_subplot(1,1,1)
ax.errorbar (Ns, scores, yerr = stds)
ax.set_xlabel("N")
ax.set_ylabel("Accuracy")
pl.title("Optimal: $n={}$".format(optimal_n))
pl.show()


I am working on the file "Final.ipynb".
The first task is:
Separate Training and Test Data
We use a poor-man's version of the holdout method to split the initial set of data into training and test. During the training below, we use various methods to further split the training data into a training and validating set. Once we have trained and validated the data, we perform a prediction on the unlabeled test data.
Can you write Python code for it?

# Start to build the table we will use

# For Training data, we want to get a certain amount of our
# table, we slice off a percentage
row_start = 0
row_end = len(features) - int(.25 * len(features))

# Our features table also contains meta information, so we slice
feature_column_start = 5

# Our collections of data to train and test
labeled_svm = features[row_start:row_end,feature_column_start:]
unlabeled_svm = features[row_end:,feature_column_start:]
test_all = features[row_end:]

# The correct classes corresponding to the training and test
classes = features[row_start:row_end,1]
unlabeled_classes = features[row_end:,1]

#labeled_svm, classes, unlabeled_svm, unlabeled_classes = cross_validation.train_test_split(
#    features[:,feature_column_start], features[:,1], test_size=1./3.)


I am working on the file "Final.ipynb".
The first task is:
Setup our Features
Can you write Python code for it?

def get_irq(rows):
    mags = rows['Mag']
    return np.percentile(mags, 75) - np.percentile(mags, 25)

def get_skew(rows):
    skew = stats.skew(rows['Mag'])
    return skew

def get_median(rows):
    return np.median(rows['Mag'])

def build_features(save_file='features.txt', limit=False):
    """
    Builds our feature array. Loops over entire loaded catalog
    and creates one row per object. Current features include:

        * Period (from catalog)
        * Magnitude
        * Median
        * Amplitude
        * Skew
        * IQR

    If you want a limited set (for faster building), pass
    limit=True
    """

    obj_id_array = np.unique(catalog['ID'])

    features = np.zeros( (len(obj_id_array), 11) )

    # For each object
    for i in range(len(obj_id_array)):
        obj_id = obj_id_array[i]

        rows = catalog[(catalog['ID'] == obj_id)]

        if limit & i > 500:
            break

        features[i,0] = obj_id # Object ID
        features[i,1] = rows['Var_Type'][0] # Class
        features[i,2] = 0. # Predicted Class
        features[i,3] = rows['RA'][0] # RA_J2000
        features[i,4] = rows['Decl'][0] # Dec
        features[i,5] = rows['Period_days'][0]  # Period
        features[i,6] = rows['Mag'][0]  # Mag
        features[i,7] = get_median(rows)  # Median
        features[i,8] = rows['Amplitude'][0]  # Amplitude
        features[i,9] = get_skew(rows)  # Skew
        features[i,10] = get_irq(rows)  # IQR

        # Save the files
        np.savetxt('features.txt', features, delimiter=',',
           header="Object ID, Class, RAJ2000, Dec, Period, Mag, Amplitude, Median, Skew, IQR")

    return features

# Get our features. Either build them from scratch or load from file
# If we build them from scratch, we also pass in the file name to save.

features = None

features_file = 'features.txt'
if os.path.isfile(features_file):
    features = np.loadtxt(features_file, dtype=None, delimiter=",")
else:
    features =...


I am working on the file "Final.ipynb".
The first task is:
Support Vector Classification (SVC)
We use the sklean.svm.svc module with a few different kernels to attempt some fits.
The equation governing this classification is:
$K(\bf{x}, \bf{x}') = e^{-\gamma||\bf{x}-\bf{x}'||^2}$
Threfore we need to obatin both $C$ and $\gamma$, so we test a range of values and create a heat-map to show optimal.
Determine Best Kernel Classifier
Run through the different kernels for an SVC to determine the best. Possible kernels are:

linear
poly
rbf
sigmoid
precomputed

We perform an exhaustive grid search across all the kernels with a range of different values. We compute these separately so that we can then compare each with an ROC. The grid search would handle this internally if we ran them all together but we want to generate an ROC plot so we do it separate.
Get the optimal SVC
Here we define the parameters for each of the kernel types and then run them through an exhaustive grid search. This is similar to the examples above where we ran through a list of $C$ and $\gamma$ values but here the GridSearchCV takes care of all the details. This gets us our optimal classifier based on this exhaustive search.
One area of uncertainty is the selection of our ranges below. We lifted the $C$ and $\gamma$ ranges from Guillermo's Hands-On (todo: get reference) and just copied that over for the coefficient range. We totall guessed at the degree range and would need to explore that more.
Can you write Python code for it?

def get_optimal_svc(param_grid=list(), plot=False):

    # Do the grid search
    grid = GridSearchCV(svm.SVC(), param_grid=param_grid, cv=ss)
    grid.fit (labeled_svm, classes)

    if plot:
        score_dict = grid.grid_scores_

        # We extract the scores
        scores = [x[1] for x in score_dict]
        scores = np.array(scores).reshape(len(C_range), len(gamma_range))

        # Make a nice figure
        pl.figure(figsize=(8, 6))
        pl.subplots_adjust(left=0.15, right=0.95, bottom=0.15, top=0.95)
        pl.imshow(scores, interpolation='nearest', cmap=pl.cm.gist_heat)
        pl.xlabel('$\gamma$')
        pl.ylabel('$C$')
        pl.title('Heatmap for optimal: $C={}$ and $\gamma={}$\n'.format(grid.best_estimator_.C, grid.best_estimator_.gamma))
        pl.colorbar()
        pl.xticks(np.arange(len(gamma_range)), gamma_range, rotation=45)
        pl.yticks(np.arange(len(C_range)), C_range)
        pl.show()

    print "The best classifier for this run is: {}".format(grid.best_estimator_)
    return grid.best_estimator_

# Setup our ranges
C_range = 10. ** np.arange(-4, 4)
gamma_range = 10. ** np.arange(-4, 4)
deg_range = np.arange(1, 3)
coef_range = 10. ** np.arange(-4, 4)

param_grids = [
    #{ 'C': C_range, 'kernel': ['linear'] },
    #{ 'C': C_range, 'gamma': gamma_range, 'degree': deg_range, 'coef0': coef_range, 'kernel': ['poly'] },
    { 'C': C_range, 'gamma': gamma_range, 'kernel': ['rbf'] },
    #{ 'C': C_range, 'gamma': gamma_range, 'coef0': coef_range, 'kernel': ['sigmoid'] },

    { 'kernel': ['linear'] },
    { 'kernel': ['poly'] },
    { 'kernel': ['sigmoid'] },
]

best_svcs = list()

# Run the param_grid one at a time (to compare ROC below)
# Get the best clf for each kernel type
for param_grid in param_grids:
    print "{}...".format(param_grid.get('kernel'))
    clf = get_optimal_svc(param_grid=param_grid, plot=False)

    param_grid['clf'] = clf

    best_svcs.append(param_grid)


I am working on the file "Final.ipynb".
The first task is:
The above plot shows that the fit is not too good. The exponential curve like the histogram does decrease at an increasing rate, but overall the curve does not fit too well. Further the QQ Plot shows that the data does not appear to be well approximated by an exponential distribution.
Can you write Python code for it?

#get the 5th quartile
int((float(len(deltas1))/100)*5)
#get the 9th quartile
int((float(len(deltas1))/100)*95)


I am working on the file "Final.ipynb".
The first task is:
The constant $b$ is calculated by using least squares and maximum likelihood. The calculated value of $b$ is 1.53, which means there are more earthquakes of lower magnitude than higher magnitude.
2.2 Time of Next Earthquake
There are reasons to believe that the number of earthquakes per year, hour, other unit of time can be approximated fairly well by a Poisson distribution, specifically that Q~Pois( $\lambda t$) where Q is the number of earthquakes, t is some unit of time, and $ \lambda $ is a rate parameter. The reasons for this as follows.
The Binomial distribution describes the total number of "successes" out of n trials with a probability of success p. The Poisson distribution arises as n goes to infinite and p goes to 0 while np remains constant. Thus, it is a good approximation when there are many, many trials, but the probability of success is quite local for any given trial. Thus, the Poisson distribution is likely a good approximation for earthquakes. For a given year, hour, etc. there is a low probability that any particular very small region in oklahoma will experience a magnitude>3 earthquake. However, there are many very small regions in Oklahoma.
It can be further shown that if Q~Pois$(\lambda* t)$, D~Expo($\lambda$) where D is the amount of time between any two successive earthquakes.
Thus, as an initial baseline, we plot the amount of time between earthquakes for the whole state of Oklahoma and see how well it can be modelled with an exponential distribution.
Can you write Python code for it?

import datetime
#function to create list of interarrival times in hours from df
def get_hours_between(df):
    dates=[]
    origintimes = df.origintime.values
    for date in origintimes:
        year, month, day = date.split('-')
        day, hour = day.split(' ')
        hour, minute, second = hour.split(':')
        if len(second.split('.'))==2:
            second, microsecond = second.split('.')
        elif len(second.split('.'))==1:
            microsecond=0
        dates.append(datetime.datetime(int(year), int(month), int(day), int(hour), int(minute),
                                       int(second), int(microsecond)))
    dates=sorted(dates)
    deltas=[]
    for i in range(1,len(dates)):
        delta = dates[i] - dates[i-1]
        delta = delta.total_seconds()/3600
        deltas.append(delta)
    deltas = np.array(deltas)
    return deltas

deltas = get_hours_between(eq_df[eq_df.prefmag>=3])

#http://statsmodels.sourceforge.net/devel/generated/statsmodels.graphics.gofplots.ProbPlot.html
#fit and plot exponential to data
def fit_expo(deltas, plot=True, bins=100, xlab1='Hours', ylab1='Probability Density',
             xlab2='Theoretical Quantiles', ylab2='Sample Quantiles',
             xmax=100, ymax=0.02, k=1., force_lambd=None, summary_stats=True):
    #If we are inherting a previous model for comparison, use previous lambda
    if pd.notnull(force_lambd):
        #Calculate Summary Statistics
        lambd=force_lambd
        mean = np.mean(deltas) #sample mean

    #If we are fitting the model on this dataset separately, calculate lambda from data
    elif pd.isnull(force_lambd):
        #Use Maxmimum Likelihood Estimator to get exponential fit
        expo_fit = sp.stats.expon.fit(deltas, floc=0)
        #Calculate Summary Statistics
        lambd = expo_fit[1] #fitted mean
        #mean = lambd

    #Calculate more summary statistics
    k=float(k)
    n = float(len(deltas))
    ssr = np.sum((deltas-lambd)**2)
    aic = n*np.log(ssr/n) + 2*k
    mse = np.mean((deltas-lambd)**2)...


I am working on the file "Final.ipynb".
The first task is:
The function print_best_score_append_dictionary is used to print the best results of the trained classifier. We also append into a dictionary results_dic these results.
Can you write Python code for it?

def print_best_score_append_dictionary(reg_type):

	best_score_prior = [[c[1],c[2]] for c in best_grid_prior]
	best_score_prior = np.array(best_score_prior)
	best_index_prior = np.where(best_score_prior[:,0] == max(best_score_prior[:,0]))[0][0]
	print('Best classifier <2010 is for alpha ={}'.format(best_grid_prior[best_index_prior][0].alpha_))
	print('Coefs <2010 are ={}'.format(best_grid_prior[best_index_prior][0].coef_))
	print('R^2 <2010 = {}'.format(best_grid_prior[best_index_prior][1]))

	prior_dic = {'coefs':best_grid_prior[best_index_prior][0].coef_ , 'r2': best_grid_prior[best_index_prior][1]}

	if reg_type == 'grid':
		prior_dic['interval'] = best_grid_prior[best_index_prior][2]
		print('Best interval <2010 is {}'.format(best_grid_prior[best_index_prior][2]))
	elif reg_type == 'cluster':
		prior_dic['eps'] = best_grid_prior[best_index_prior][2]
		print('Best eps <2010 is {}'.format(best_grid_prior[best_index_prior][2]))

	best_score_post = [[c[1],c[2]] for c in best_grid_post]
	best_score_post = np.array(best_score_post)
	best_index_post = np.where(best_score_post[:,0] == max(best_score_post[:,0]))[0][0]
	print('Best classifier >= 2010 is for alpha ={}'.format(best_grid_post[best_index_post][0].alpha_))
	print('Coefs >= 2010 are ={}'.format(best_grid_post[best_index_post][0].coef_))
	print('R^2 >= 2010 = {}'.format(best_grid_post[best_index_post][1]))

	post_dic = {'coefs':best_grid_post[best_index_post][0].coef_ , 'r2': best_grid_post[best_index_post][1]}

	if reg_type == 'grid':
		post_dic['interval'] = best_grid_post[best_index_post][2]
		print('Best interval >= 2010 is {}'.format(best_grid_post[best_index_post][2]))
	elif reg_type == 'cluster':
		post_dic['eps'] = best_grid_post[best_index_post][2]
		print('Best eps >= 2010 is {}'.format(best_grid_post[best_index_post][2]))

	return prior_dic, post_dic


I am working on the file "Final.ipynb".
The first task is:
The lines are getting thicker as the bifurcations increase at an increasing rate. It is now not immediately apparent how many asymptotic values x approaches from the x vs t graph.
c = 5.7
Can you write Python code for it?

ros57 = p1.Rossler(5.7)
ros57.run()

ros57.plotx()

ros57.plotxy()

ros57.plotxyz()


I am working on the file "Final.ipynb".
The first task is:
The period doubling cascade has given rise to a chaotic attractor with a single lobe. This is an example of spiral-type chaos and exhibits the characteristic sensitivity to initial conditions. The oscillations in the x vs t graph are now completely chaotic and irregular in amplitude. The logistic update map also displays the same behavior of a period doubling cascade giving rise to a chaotic system. As we increase c, this system, like the logistic update map from the midterm as ve vary initial conditions, also demonstrates the stretching and folding quality that we discussed in class. The xyz graph is also reminiscent of a mobius strip in that the underside becomes the upperside via the portion of the graph not in the x-y plane.
Problem 3
Can you write Python code for it?

p1.plotmaxima('x')

p1.plotmaxima('y')

p1.plotmaxima('z')


I am working on the file "Final.ipynb".
The first task is:
The period doubling is occuring at an increasing rate. This is demonstrated by the thicker lines in the xy and xyz graphs. This period doubling phase will soon end as the system approaches complete chaos and the our predictive power decreases greatly.
c = 4.2
Can you write Python code for it?

ros42 = p1.Rossler(4.2)
ros42.run()

ros42.plotx()

ros42.plotxy()

ros42.plotxyz()


I am working on the file "Final.ipynb".
The first task is:
This cell loads the results from the algorithm and create an interactive plot in which you can visualize the clusters for various eps and before and after 2010.
Can you write Python code for it?

eq_df = pd.DataFrame.from_csv('./tempdata/earthquakes_catalog_treated.csv',sep = '|')
from IPython.html.widgets import interact
global eq_df
interact(plot_cluster_M, Period = ('before 2010', 'after 2010') ,distance_to_search_for_neighboring_points=(5,29,1));


I am working on the file "Final.ipynb".
The first task is:
We created an interactive visualization tool on the website that can be accessed here.
The first tab maps the distribution of earthquakes in Oklahoma at a given year in red overlayed on the previous earthquakes in green. One can use the slider on the upper right corner to get a sense of the increase of seismic activity during the past 6 years.  talk about apparent clustering
1.2 Visualization of the disposal wells
The data of the disposal wells contains wells dated from 2006 to 2012 with the amount of water disposed underground in gallons.
Can you write Python code for it?

fig, ax = plt.subplots()
plt.bar(np.arange(2006,2013),welldf.groupby('year').count().volume.values, color = "#99D699")
ax.set_xticks(np.arange(2006,2013)+.5)
ax.set_xticklabels(np.arange(2006,2013))
ax.set_xlabel('Year')
ax.set_ylabel('Number of Wells')
ax.grid(False)

fig, ax = plt.subplots()
plt.bar(np.arange(2006,2013),welldf.groupby('year').volume.sum().values, color = "#99D699")
ax.set_xticks(np.arange(2006,2013)+.5)
ax.set_xticklabels(np.arange(2006,2013))
ax.set_xlabel('Year')
ax.set_ylabel('Water disposed in gallons')
ax.grid(False)


I am working on the file "Final.ipynb".
The first task is:
We intentionally ignore some of the data relative to body and surface waves as well as other source of estimations of the magnitude to take less space in memory. A complete description of the columns can be found here.
The column that will be used in this study is the recommended magnitude estimation from column prefmag. Here is a description of the columns we kept.

id:  Internal    identification  number  assigned    by  the OGS
origintime:  Estimated   date    and time    of  the earthquake
latitude:    Estimated latitudinal   coordinate  of the  earthquake epicenter
longitude:   Estimated Longitudinal  coordinate  of  the earthquake epicenter
depth:   Calculated depth    of  the earthquake in   kilometers
county:  County  within  which   the earthquake  was located
origin_src:  Source  of  reported    earthquake  location
prefmag: Preferred   magnitude
pmag_type: Type  of  magnitude   reported
pmag_src: Source of  reported    magnitude
m3hz_ogs: Lg magnitude   estimate    formerly    used    by  the OGS
md_ogs: Earthquake   duration    magnitude   estimated   by  the OGS
geom: A  well-known  binary  (WKB)   Geographic  Information System  GIS format

Note that OGS stands for Oklahoma Geological Survey and USGS for United States  Geological  Survey.
The water disposal wells data was taken from the Oklahoma Corporation Commision, Oil and Gas Department website. Among the numerous files, we opted for a dataset that contained both the year that the well was reported to the Oil and gas department and the location in coordinates. The downloaded data set, which had many observations with missing features, is in wells_reportyear.csv.
Can you write Python code for it?

# Read data
#welldf = pd.read_csv('tempdata/wells_reportyear.csv')
# Clean up data
#welldf = welldf[np.isfinite(welldf['YEAR'])]
# Discard extreme coordinates
#welldf = welldf[(welldf.LONGITUDE >= -102.918) & (welldf.LONGITUDE <= -94.466)]
#welldf = welldf[(welldf.LATITUDE >= 33.811) & (welldf.LATITUDE <= 36.99869)]
# Data set contains futuristic years, we discard those observations
#welldf = welldf[welldf.YEAR <= 2015]
# Save data frame for future use
# welldf.to_csv('tempdata/wells_data.csv')
# Load data for future use
welldf = pd.read_csv('tempdata/wells_data.csv')
# Show first 5 wells
print welldf.shape
welldf.head()


I am working on the file "Final.ipynb".
The first task is:
With a p-value of .0003, we reject the null hypothesis and conclude that the medians differ. In addition, we perform a permutation test with replacement in case the log-transformed data does not follow a normal distribution:
Can you write Python code for it?

# Prepare data
alleq_df_ByYear = eq_df.groupby('year').count().reset_index().prefmag
n = len(eq_df_postByYear)
N = len(eq_df.groupby('year').count().reset_index().prefmag)
# Get observed difference
obsdiff = np.log(eq_df_postByYear.mean())-np.log(eq_df_priorByYear.mean())

# Start list vector for storing difference of means
mediff = []

# Run permutations
for i in range(100000):
    post = np.random.choice(alleq_df_ByYear.values, size = n, replace = True)
    prior = np.random.choice(alleq_df_ByYear.values, size = N-n, replace = True)
    mediff.append(np.log(post.mean())-np.log(prior.mean()))

# Plot results
plt.hist(mediff, bins = 50)
plt.axvline(obsdiff, color='b', linestyle='dashed', linewidth=2, label = 'Observed difference')
plt.xlabel('Difference of earthquakes per year between prior and post 2010')
plt.ylabel('Frequency')
plt.legend()
plt.grid(False)
# Get p.value
count = 0
for diff in mediff:
    if diff > obsdiff:
        count += 1
print count/100000.


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Initialize Spark Context
Can you write Python code for it?

from pyspark.mllib.recommendation import ALS, Rating
from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local[*]").setAppName("MovieRecommendationsALS").set("spark.executor.memory", "2g")
sc = SparkContext(conf = conf)


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Lets find similar movies for Toy Story (Movie ID: 1)
Can you write Python code for it?

scoreThreshold = 0.97
coOccurenceThreshold = 50

inputMovieID = 1  #Toy Story.

# Filter for movies with this sim that are "good" as defined by our quality thresholds.
filteredResults = moviePairSimilarities.filter(lambda((pair,sim)): \
    (pair[0] == inputMovieID or pair[1] == inputMovieID) and sim[0] > scoreThreshold and sim[1] > coOccurenceThreshold)

 #Top 10 by quality score.

results = filteredResults.map(lambda((pair,sim)): (sim, pair)).sortByKey(ascending = False).take(10)

print "Top 10 similar movies for " + nameDict[inputMovieID]

for result in results:
    (sim, pair) = result
    # Display the similarity result that isn't the movie we're looking at
    similarMovieID = pair[0]
    if (similarMovieID == inputMovieID):
        similarMovieID = pair[1]
    print nameDict[similarMovieID] + "\tscore: " + str(sim[0]) + "\tstrength: " + str(sim[1])


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Most popular movies
Can you write Python code for it?

movies = data.map(lambda x: (int(x.split()[1]), 1))
movieCounts = movies.reduceByKey(lambda x, y: x + y)
flipped = movieCounts.map( lambda (x, y) : (y, x))
sortedMovies = flipped.sortByKey(False)
sortedMoviesWithNames = sortedMovies.map(lambda (count, movie) : (nameDict[movie], count))

results = sortedMoviesWithNames.collect()
subset = results[0:10]
popular_movieNm = [str(i[0]) for i in subset]
popularity_strength = [int(i[1]) for i in subset]

popMovplot = plt.barh(range(len(subset)), popularity_strength, align='center')
plt.yticks(range(len(subset)), popular_movieNm)
popMovplot[0].set_color('g')
print "Most Popular Movies from the Dataset"


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Prepare the pySpark Environment
Can you write Python code for it?

spark_home = os.environ.get('SPARK_HOME', None)

if not spark_home:
    raise ValueError("Please set SPARK_HOME environment variable!")

# Add the py4j to the path.
sys.path.insert(0, os.path.join(spark_home, 'python'))
sys.path.insert(0, os.path.join(spark_home, 'C:/spark/python/lib/py4j-0.9-src.zip'))


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Ratings Histogram
Can you write Python code for it?

ratPlot = plt.bar(range(len(sortedResults)), sortedResults.values(), align='center')
plt.xticks(range(len(sortedResults)), list(sortedResults.keys()))
ratPlot[3].set_color('g')
print "Ratings Histogram"


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Recommendations
Can you write Python code for it?

#Lets recommend movies for the user id - 2
userID = 2

print "\nTop 10 recommendations:"
recommendations = model.recommendProducts(userID, 10)
for recommendation in recommendations:
    print nameDict[int(recommendation[1])] + \
        " score " + str(recommendation[2])


I am working on the file "Final/DATA643_Final_Project.ipynb".
The first task is:
Recommender using MLLib
Training the recommendation model
Can you write Python code for it?

ratings = data.map(lambda l: l.split()).map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2]))).cache()
ratings.take(3)

ratings.take(1)[0]

nratings  = ratings.count()
nUsers    = ratings.keys().distinct().count()
nMovies   = ratings.values().distinct().count()

print "We have Got %d ratings from %d users on %d movies." % (nratings, nUsers, nMovies)

# Build the recommendation model using Alternating Least Squares
#Train a matrix factorization model given an RDD of ratings given by users to items, in the form of
#(userID, itemID, rating) pairs. We approximate the ratings matrix as the product of two lower-rank matrices
#of a given rank (number of features). To solve for these features, we run a given number of iterations of ALS.
#The level of parallelism is determined automatically based on the number of partitions in ratings.

#Our ratings are in the form of ==> [userid, (movie id, rating)] ==>  [ (1, (61, 4.0)), (1, (189, 3.0)) etc. ]
start = time()

seed = 5L
iterations = 10
rank = 8

model = ALS.train(ratings, rank, iterations)

duration = time() - start

print "Model trained in %s seconds" % round(duration,3)


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Initialize Spark Context
Can you write Python code for it?


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Lets find similar movies for Toy Story (Movie ID: 1)
Can you write Python code for it?


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Load and Analyse Data
Can you write Python code for it?

def loadMovieNames():
    movieNames = {}
    for line in urllib2.urlopen("https://raw.githubusercontent.com/psumank/DATA643/master/WK5/ml-100k/u.item"):
        fields = line.split('|')
        movieNames[int(fields[0])] = fields[1].decode('ascii', 'ignore')
    return movieNames

print "\nLoading movie names..."
nameDict = loadMovieNames()

print "\nLoading ratings data..."
data = sc.textFile("file:///C:/Users/p_sum/.ipynb_checkpoints/ml-100k/u.data")

ratings = data.map(lambda x: x.split()[2])

#action -- just to trigger the driver [ lazy evaluation ]
rating_results = ratings.countByValue()

sortedResults = collections.OrderedDict(sorted(rating_results.items()))
for key, value in sortedResults.iteritems():
    print "%s %i" % (key, value)


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Most popular movies
Can you write Python code for it?


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Prepare the pySpark Environment
Can you write Python code for it?


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Recommendations
Can you write Python code for it?


I am working on the file "Final/DATA643_pySpark_Final_Project.ipynb".
The first task is:
Recommender using MLLib
Training the recommendation model
Can you write Python code for it?

ratings = data.map(lambda l: l.split()).map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2]))).cache()
ratings.take(3)

nratings  = ratings.count()
nUsers    = ratings.keys().distinct().count()
nMovies   = ratings.values().distinct().count()

print "We have Got %d ratings from %d users on %d movies." % (nratings, nUsers, nMovies)

# Build the recommendation model using Alternating Least Squares
#Train a matrix factorization model given an RDD of ratings given by users to items, in the form of
#(userID, itemID, rating) pairs. We approximate the ratings matrix as the product of two lower-rank matrices
#of a given rank (number of features). To solve for these features, we run a given number of iterations of ALS.
#The level of parallelism is determined automatically based on the number of partitions in ratings.

start = time()

seed = 5L
iterations = 10
rank = 8

model = ALS.train(ratings, rank, iterations)

duration = time() - start

print "Model trained in %s seconds" % round(duration,3)


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
Exploratory Data Analysis
Identifying Missingness in the data
Can you write Python code for it?

def get_heatmap(df):
    #This function gives heatmap of all NaN values
    plt.figure(figsize=(10,6))
    sns.heatmap(df.isnull(), yticklabels=False, cbar=False, cmap='viridis')
    plt.tight_layout()
    return plt.show()

get_heatmap(training_data)

bots.friends_count/bots.followers_count

plt.figure(figsize=(10,5))
plt.subplot(2,1,1)
plt.title('Bots Friends vs Followers')
sns.regplot(bots.friends_count, bots.followers_count, color='red', label='Bots')
plt.xlim(0, 100)
plt.ylim(0, 100)
plt.tight_layout()

plt.subplot(2,1,2)
plt.title('NonBots Friends vs Followers')
sns.regplot(nonbots.friends_count, nonbots.followers_count, color='blue', label='NonBots')
plt.xlim(0, 100)
plt.ylim(0, 100)

plt.tight_layout()
plt.show()


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
Identifying Imbalance in the data
Can you write Python code for it?

bots['friends_by_followers'] = bots.friends_count/bots.followers_count
bots[bots.friends_by_followers<1].shape

nonbots['friends_by_followers'] = nonbots.friends_count/nonbots.followers_count
nonbots[nonbots.friends_by_followers<1].shape

plt.figure(figsize=(10,5))
plt.plot(bots.listed_count, color='red', label='Bots')
plt.plot(nonbots.listed_count, color='blue', label='NonBots')
plt.legend(loc='upper left')
plt.ylim(10000,20000)
print(bots[(bots.listed_count<5)].shape)

bots_listed_count_df = bots[bots.listed_count<16000]
nonbots_listed_count_df = nonbots[nonbots.listed_count<16000]

bots_verified_df = bots_listed_count_df[bots_listed_count_df.verified==False]
bots_screenname_has_bot_df_ = bots_verified_df[(bots_verified_df.screen_name.str.contains("bot", case=False)==True)].shape

plt.figure(figsize=(12,7))

plt.subplot(2,1,1)
plt.plot(bots_listed_count_df.friends_count, color='red', label='Bots Friends')
plt.plot(nonbots_listed_count_df.friends_count, color='blue', label='NonBots Friends')
plt.legend(loc='upper left')

plt.subplot(2,1,2)
plt.plot(bots_listed_count_df.followers_count, color='red', label='Bots Followers')
plt.plot(nonbots_listed_count_df.followers_count, color='blue', label='NonBots Followers')
plt.legend(loc='upper left')

#bots[bots.listedcount>10000]
condition = (bots.screen_name.str.contains("bot", case=False)==True)|(bots.description.str.contains("bot", case=False)==True)|(bots.location.isnull())|(bots.verified==False)

bots['screen_name_binary'] = (bots.screen_name.str.contains("bot", case=False)==True)
bots['location_binary'] = (bots.location.isnull())
bots['verified_binary'] = (bots.verified==False)
bots.shape

condition = (nonbots.screen_name.str.contains("bot", case=False)==False)| (nonbots.description.str.contains("bot", case=False)==False) |(nonbots.location.isnull()==False)|(nonbots.verified==True)

nonbots['screen_name_binary'] = (nonbots.screen_name.str.contains("bot", case=False)==False)
nonbots['location_binary'] =...


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
Implementing Different Models
Decision Tree Classifier
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, roc_curve, auc
from sklearn.model_selection import train_test_split

X = training_data[features].iloc[:,:-1]
y = training_data[features].iloc[:,-1]

dt = DecisionTreeClassifier(criterion='entropy', min_samples_leaf=50, min_samples_split=10)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)

dt = dt.fit(X_train, y_train)
y_pred_train = dt.predict(X_train)
y_pred_test = dt.predict(X_test)

print("Trainig Accuracy: %.5f" %accuracy_score(y_train, y_pred_train))
print("Test Accuracy: %.5f" %accuracy_score(y_test, y_pred_test))

sns.set(font_scale=1.5)
sns.set_style("whitegrid", {'axes.grid' : False})

scores_train = dt.predict_proba(X_train)
scores_test = dt.predict_proba(X_test)

y_scores_train = []
y_scores_test = []
for i in range(len(scores_train)):
    y_scores_train.append(scores_train[i][1])

for i in range(len(scores_test)):
    y_scores_test.append(scores_test[i][1])

fpr_dt_train, tpr_dt_train, _ = roc_curve(y_train, y_scores_train, pos_label=1)
fpr_dt_test, tpr_dt_test, _ = roc_curve(y_test, y_scores_test, pos_label=1)

plt.plot(fpr_dt_train, tpr_dt_train, color='darkblue', label='Train AUC: %5f' %auc(fpr_dt_train, tpr_dt_train))
plt.plot(fpr_dt_test, tpr_dt_test, color='red', ls='--', label='Test AUC: %5f' %auc(fpr_dt_test, tpr_dt_test))
plt.title("Decision Tree ROC Curve")
plt.xlabel("False Positive Rate (FPR)")
plt.ylabel("True Positive Rate (TPR)")
plt.legend(loc='lower right')


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
Our Classifier
Can you write Python code for it?

class twitter_bot(object):
    def __init__(self):
        pass

    def perform_train_test_split(df):
        msk = np.random.rand(len(df)) < 0.75
        train, test = df[msk], df[~msk]
        X_train, y_train = train, train.ix[:,-1]
        X_test, y_test = test, test.ix[:, -1]
        return (X_train, y_train, X_test, y_test)

    def bot_prediction_algorithm(df):
        # creating copy of dataframe
        train_df = df.copy()
        # performing feature engineering on id and verfied columns
        # converting id to int
        train_df['id'] = train_df.id.apply(lambda x: int(x))
        #train_df['friends_count'] = train_df.friends_count.apply(lambda x: int(x))
        train_df['followers_count'] = train_df.followers_count.apply(lambda x: 0 if x=='None' else int(x))
        train_df['friends_count'] = train_df.friends_count.apply(lambda x: 0 if x=='None' else int(x))
        #We created two bag of words because more bow is stringent on test data, so on all small dataset we check less
        if train_df.shape[0]>600:
            #bag_of_words_for_bot
            bag_of_words_bot = r'bot|b0t|cannabis|tweet me|mishear|follow me|updates every|gorilla|yes_ofc|forget' \
                           r'expos|kill|clit|bbb|butt|fuck|XXX|sex|truthe|fake|anony|free|virus|funky|RNA|kuck|jargon' \
                           r'nerd|swag|jack|bang|bonsai|chick|prison|paper|pokem|xx|freak|ffd|dunia|clone|genie|bbb' \
                           r'ffd|onlyman|emoji|joke|troll|droop|free|every|wow|cheese|yeah|bio|magic|wizard|face'
        else:
            # bag_of_words_for_bot
            bag_of_words_bot = r'bot|b0t|cannabis|mishear|updates every'

        # converting verified into vectors
        train_df['verified'] = train_df.verified.apply(lambda x: 1 if ((x == True) or x == 'TRUE') else 0)

        # check if the name contains bot or screenname contains b0t
        condition = ((train_df.name.str.contains(bag_of_words_bot, case=False, na=False)) |
                    ...


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
Performing Feature Extraction
Can you write Python code for it?

training_data['listed_count_binary'] = (training_data.listed_count>20000)==False
features = ['screen_name_binary', 'name_binary', 'description_binary', 'status_binary', 'verified', 'followers_count', 'friends_count', 'statuses_count', 'listed_count_binary', 'bot']


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
ROC Comparison after tuning the baseline model
Can you write Python code for it?

plt.figure(figsize=(14,10))
(X_train, y_train, X_test, y_test) = twitter_bot.perform_train_test_split(df)
#Train ROC
y_pred_train, y_true = twitter_bot.get_predicted_and_true_values(X_train, y_train)
scores = np.linspace(start=0, stop=1, num=len(y_true))
fpr_botc_train, tpr_botc_train, threshold = metrics.roc_curve(y_pred_train, scores, pos_label=0)

#Test ROC
y_pred_test, y_true = twitter_bot.get_predicted_and_true_values(X_test, y_test)
scores = np.linspace(start=0, stop=1, num=len(y_true))
fpr_botc_test, tpr_botc_test, threshold = metrics.roc_curve(y_pred_test, scores, pos_label=0)

#Train ROC
plt.subplot(2,2,1)
plt.plot(fpr_botc_train, tpr_botc_train, label='Our Classifier AUC: %5f' % metrics.auc(fpr_botc_train,tpr_botc_train), color='darkblue')
plt.plot(fpr_rf_train, tpr_rf_train, label='Random Forest AUC: %5f' %auc(fpr_rf_train, tpr_rf_train))
plt.plot(fpr_dt_train, tpr_dt_train, label='Decision Tree AUC: %5f' %auc(fpr_dt_train, tpr_dt_train))
plt.plot(fpr_mnb_train, tpr_mnb_train, label='MultinomialNB AUC: %5f' %auc(fpr_mnb_train, tpr_mnb_train))
plt.title("Training Set ROC Curve")
plt.xlabel("False Positive Rate (FPR)")
plt.ylabel("True Positive Rate (TPR)")
plt.legend(loc='lower right')

#Test ROC
plt.subplot(2,2,2)
plt.plot(fpr_botc_test,tpr_botc_test, label='Our Classifier AUC: %5f' %metrics.auc(fpr_botc_test,tpr_botc_test), color='darkblue')
plt.plot(fpr_rf_test, tpr_rf_test, label='Random Forest AUC: %5f' %auc(fpr_rf_test, tpr_rf_test))
plt.plot(fpr_dt_test, tpr_dt_test, label='Decision Tree AUC: %5f' %auc(fpr_dt_test, tpr_dt_test))
plt.plot(fpr_mnb_test, tpr_mnb_test, label='MultinomialNB AUC: %5f' %auc(fpr_mnb_test, tpr_mnb_test))
plt.title("Test Set ROC Curve")
plt.xlabel("False Positive Rate (FPR)")
plt.ylabel("True Positive Rate (TPR)")
plt.legend(loc='lower right')
plt.tight_layout()


I am working on the file "FinalProjectAndCode/IPython NoteBooks/.ipynb_checkpoints/BotDetection-checkpoint.ipynb".
The first task is:
Result: Decision Tree gives very good performance and generalizes well. But it may be overfitting as AUC is 0.937, so we will try other models.
Multinomial Naive Bayes Classifier
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB

X = training_data[features].iloc[:,:-1]
y = training_data[features].iloc[:,-1]

mnb = MultinomialNB(alpha=0.0009)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)

mnb = mnb.fit(X_train, y_train)
y_pred_train = mnb.predict(X_train)
y_pred_test = mnb.predict(X_test)

print("Trainig Accuracy: %.5f" %accuracy_score(y_train, y_pred_train))
print("Test Accuracy: %.5f" %accuracy_score(y_test, y_pred_test))

sns.set_style("whitegrid", {'axes.grid' : False})

scores_train = mnb.predict_proba(X_train)
scores_test = mnb.predict_proba(X_test)

y_scores_train = []
y_scores_test = []
for i in range(len(scores_train)):
    y_scores_train.append(scores_train[i][1])

for i in range(len(scores_test)):
    y_scores_test.append(scores_test[i][1])

fpr_mnb_train, tpr_mnb_train, _ = roc_curve(y_train, y_scores_train, pos_label=1)
fpr_mnb_test, tpr_mnb_test, _ = roc_curve(y_test, y_scores_test, pos_label=1)

plt.plot(fpr_mnb_train, tpr_mnb_train, color='darkblue', label='Train AUC: %5f' %auc(fpr_mnb_train, tpr_mnb_train))
plt.plot(fpr_mnb_test, tpr_mnb_test, color='red', ls='--', label='Test AUC: %5f' %auc(fpr_mnb_test, tpr_mnb_test))
plt.title("Multinomial NB ROC Curve")
plt.xlabel("False Positive Rate (FPR)")
plt.ylabel("True Positive Rate (TPR)")
plt.legend(loc='lower right')


I am working on the file "FinalProjectAndCode/IPython NoteBooks/BotDetection.ipynb".
The first task is:
Exploratory Data Analysis
Identifying Missingness in the data
Can you write Python code for it?


I am working on the file "FinalProjectAndCode/IPython NoteBooks/BotDetection.ipynb".
The first task is:
Identifying Imbalance in the data
Can you write Python code for it?


I am working on the file "FinalProjectAndCode/IPython NoteBooks/BotDetection.ipynb".
The first task is:
Our Classifier
Can you write Python code for it?


I am working on the file "Final_Project/theAwesome_EnsModel.ipynb".
The first task is:
Step 7: Model Predictions on Test Dataset
Apply your model to input (scaled) columns of Test dataset to obtain the predicted output for Test
dataset. If your model is regression then plot actual output versus predicted output column of Test
dataset. If your model is classification then generate confusion matrix on actual and predicted columns
of Test dataset.

Hint: Matplotlip, Seaborn, Bokeh scatter(), plot() functions
- http://scikit-learn.org/0.15/auto_examples/plot_confusion_matrix.html
- http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
Can you write Python code for it?

# test prediction for test data
out_test_pred = clf.predict(inp_test)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_test, out_test_pred)
print(cm)

sns.heatmap(cm, center=True)
plt.title('Confusion matrix')
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()


I am working on the file "Final_Project/theAwesome_EnsModel.ipynb".
The first task is:
Train data is performing slightly better than the test data however, I believe it is overfitting, as you can see in the Train data ROC curve, it started very high and hit 1.0 sooner.
This model is not yet deployable because I did not check other models and tweaking the parameters.

Step 9: Update the Model
Go back to Step5, and choose random values (use default values) of the model parameters and re-train
the model. Repeat Steps: 6, 7 and 8. Using the same error metric, generate the accuracy of the model on
Training and Test dataset. Did you get a better or worse performance on Training or Test set?
Explain why the new model performs better or worse than the former model. What does
hyperparameter optimization (grid search) on model building enable?
Can you write Python code for it?

# Using default values for RandomForeset Classifier
# Building a RandomForest
clf = RandomForestClassifier()

clf = clf.fit(inp_train, out_train)

# train prediction for train data
out_train_pred = clf.predict(inp_train)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_train, out_train_pred)
print(cm)

# test prediction for test data
out_test_pred = clf.predict(inp_test)
# Compute confusion matrix for prediction of train
cm = confusion_matrix(out_test, out_test_pred)
print(cm)

# Model trained with default values
# ROC curve for train data
fpr,tpr,thresholds = roc_curve(out_train, out_train_pred,pos_label=2)
# plot the curve
plt.plot(fpr, tpr, label="Train Data")
# ROC curve for test data
fpr, tpr, thresholds = roc_curve(out_test, out_test_pred, pos_label=2)
# Plotting the curves
plt.plot(fpr, tpr, label="Test Data")
plt.xlim([-0.05,1.05])
plt.ylim([-0.05,1.05])
plt.title('ROC curve for Cancer classifer')
plt.xlabel('False positive rate (1-specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.legend(loc=4,)
plt.show()


I am working on the file "Final_Report.ipynb".
The first task is:
3. Bilateral filter
Motivation
We think that the hierarchy in Hardware performance for such a filter is the following:

Integrated GPU
CPU with AVX
Not integrated GPU

CPU with AVX can indeed be faster than a GPU that is not integrated in the case of one pass of a filter on an image. Because the data going through the bus from GPU to CPU is much slower than the access to the RAM.
Using the SIMD of AVX allows us to optimize performance on CPU calculation. However, this will be true only if the algorithm is CPU bound. That is why we will pay special care to memory optimization in this project.
3.1 Code implementations and optimization
3.1.1 Introductory remarks
In the following part of the report we will introduce snippets of our code that don't run by themselves but that show the different things we tried to improve performances. All the function are written in seperate files and are called within this Ipython notebook.Furthermore, to emphasize our work on the optimization of the code we highligh in this report different concepts such as the memory behaviour and the parallelism.
We explore some implementations using AVX. However, we could not manage to code the exponential function in AVX. This computer has only SSE and has an AVX emulator. There are exp functions coded for AVX on the internet but they are not working with an emulator. Because finding a hack to make the exponential function works in AVX was taking too much of our time, we decided to move on in order to focus on concepts of the course and simply commenting out the lines where we take the exp of an AVX array. We could still benchmarck the accuracy of our code when we were not using AVX, in a first step with the Gaussian filter that we compared against a scipy function and for a bilateral filter using only cython. We have already showed an output image of our code as the example for the bilateral filter at the beginning of this report (where we used the function r_r1 that we present below).

3.1.2 Cython, AVX and Python multithreading
Cython
You...
Can you write Python code for it?


I am working on the file "Final_Report.ipynb".
The first task is:
Benchmark when we vary the size of the image
Nota Bene, we do not use the designs corresponding to r_r2, r_pr2, r_AVX2. Because these implementation of the bilateral filters are the only ones that don't take the same image input as the others. But this design will be benchmarked in our Gaussian filter validation section thereafter.
We print and plot the data. The print is for us in order to monitor how the computation evolves.
Can you write Python code for it?

functions = [r_r1,r_pr1,r_AVX1,prj_r1,prj_AVX,step_r1,AVX_r,mth_pr]

th = [None,2,2,2,2,2,2,2]

truncate = 4
sigma_r = 2
sigma_s = 2

assert len(functions) == len(th)

iters = 10
im_sizes = [2**k for k in range(3,iters)]

times = [[0 for im_size in im_sizes] for fn in functions]

for k, fn in enumerate(functions):
    print str(fn)
    for l,im_size in enumerate(im_sizes):
        lw = (max(sigma_r, sigma_s)*truncate)

        #create picture
        picture = np.random.random((im_size, im_size))

        #preprocess
        imsize = np.shape(picture)
        input_im = np.zeros((lw + imsize[0] + lw, lw + imsize[1] + lw + 8 - (2 * lw + 1) % 8))
        input_im[:(lw + imsize[0] + lw), :(lw + imsize[1] + lw)] = np.pad(picture, ((lw, lw), (lw, lw)), mode = 'reflect')

        #time
        times[k][l] = time_update(fn, truncate, imsize, picture, input_im, sigma_r, sigma_s, lw, th[k])

df = pd.DataFrame(times, dtype = float)
df.to_csv('times.csv', index=False)


I am working on the file "Final_Report.ipynb".
The first task is:
Discussion
Here AVX proves useful! 4 times faster. We are much more compute bound. The number of cache miss is also one of the smallest which shows that we have a good management of the memory.

mth_pr
Description
Now we create treads to send the rows. Thread i will compute for output pixels in row i modulo the number of threads.
We also do a pranje over the rows. As this computer only has two thread it should not have much effect, however we make sure there is "nogil = True" that way. It is the only way to do actual multi-threading in Python.
The image descibes the case where there are 4 threads (but we have only 2).

A before, the output pixels are computed 8 by 8 using AVX
Prediction
Now we can benefit from the shared input. There won't be much of wasted time in memory movements and a lot of computation per thread. This should be our best implementation.
Nota Bene: we could make the computation even more computation intensive per thread by doing a few steps of 8 per threads until the numbers of pixel per thread equals the width of the context needed to compute a pixel.
Can you write Python code for it?

! perf stat -e cache-misses,instructions ./driver_mth_pr.py 256 2 2 4


I am working on the file "Final_Report.ipynb".
The first task is:
Discussion
Not much to be said, we will use it as a baseline.

r_pr1
Description
Here we do ranges for the loops over output pixels.
We do pranges for the computation of one pixel.
Prediction
Here we are taking advantage of the 2 threads so it should be about twice faster (because I am using only 2 threads).
Can you write Python code for it?

! perf stat -e cache-misses,instructions ./driver_r_pr1.py 256 2 2 4

print("The cache efficiency is {}".format(1./13967240.*124698915537.))


I am working on the file "Final_Report.ipynb".
The first task is:
Discussion
Not the best performance. But lowest number of cache miss sofar. AVX makes it twice faster. We seem to have a better management of the memory and we become more CPU-bound. It matches our prediction!

step_r1
Description
Here we do steps of 8 in the prange as in the image. There is another range within the thread. Now each thread is computing 8 output pixels so it load a slighly larger context.

Nota Bene: Here we don't have AVX yet (as the image says and our AVX are 8 float32). This image was made for the next implementation but it actually also helps understanding this implementation.
Prediction
The performance should be similar to prj_r1. But now we are much more memory effective. We load slightly larger context, but we capitalize on the overlap of the context of neighboring pixels. There is less memory transfert and more computation per thread.
Can you write Python code for it?

! perf stat -e cache-misses,instructions ./driver_step_r1.py 256 2 2 4

print("The cache efficiency is {}".format(95016359478./10116808.))


I am working on the file "Final_Report.ipynb".
The first task is:
Discussion
The performance is indeed similar to prj_r1 but we have good hopes for the AVX implementation.

AVX_r
Description
Instead of doing a range(8) to compute for a pixel in each thread, we do all the 8 pixel computations in parallel using AVX arrays.
Prediction
As our design is becoming ever more computation bound, AVX should be effective.
Can you write Python code for it?

! perf stat -e cache-misses,instructions ./driver_AVX_r.py 256 2 2 4

print("The cache efficiency is {}".format(25993617895./5902434.))


I am working on the file "Final_Report.ipynb".
The first task is:
Discussion
The performance is slightly better but not twice better. Maybe having some thread adds some overhead and also the r_r2 version is already very effective. The cache performance is a little worse.

r_AVX2
Can you write Python code for it?

! perf stat -e cache-misses,instructions ./driver_r_AVX2.py 256 2 2 4

print("The cache efficiency is {}".format(80883074937./17775463.))


I am working on the file "Final_Report.ipynb".
The first task is:
Discussion
We see that r_AVX1 becomes much better than r_r1 and r_pr1. Our assumption that this function was not performing well because the window  - needed to compute one output pixel - was too small and that overhead was hampering the performance seems to be correct.
The rest stays more or less the same. r_pr1 is sort of plateauing at a truncate number equals 10, but it is hard to tell with only one data point.

Benchmark when we vary the truncation parameter
Can you write Python code for it?

functions = [r_pr1,prj_r1,prj_AVX,step_r1,AVX_r,mth_pr]

iters = 8
threads = [k for k in range(1,iters)]
sigma_r = 2
sigma_s = 2
im_size = 128

times3 = [[0 for k in threads] for fn in functions]

truncate = 4
for k, fn in enumerate(functions):
    print str(fn)
    for l,thread in enumerate(threads):
        lw = (max(sigma_r, sigma_s)*truncate)

        th = [thread,thread,thread,thread,thread,thread]
        #create picture
        picture = np.random.random((im_size, im_size))

        #preprocess
        imsize = np.shape(picture)
        input_im = np.zeros((lw + imsize[0] + lw, lw + imsize[1] + lw + 8 - (2 * lw + 1) % 8))
        input_im[:(lw + imsize[0] + lw), :(lw + imsize[1] + lw)] = np.pad(picture, ((lw, lw), (lw, lw)), mode = 'reflect')

        #time
        times3[k][l] = time_update(fn, truncate, imsize, picture, input_im, sigma_r, sigma_s, lw, th[k])

df2 = pd.DataFrame(times3, dtype = float)
df2.to_csv('times3.csv', index=False)

%matplotlib inline
for k in range(len(functions)):
    plt.plot(threads,times3[:][k],  "o-", label = "{}".format(str(functions[k])[19:-1]))
plt.legend(loc="best")


I am working on the file "Final_Report.ipynb".
The first task is:
For each row, AVX loads 8 adjacent pixels of 32 bits of window and kernel. The middle column is not loaded. The loop over the rows in done using a prange. The method is called filter_AVX
4/ a multithreaded implementation. Each thread is takes care of one row of the input image. Thread 1 takes care of the first row, thread 2 takes care of the 2nd row, thread 3 of the 3r row, thread 4 of the 4th row and then thread 1 continues with row 5th etc (see figure below). The function is called filter_cython_threading.
Can you write Python code for it?

HTML('<img src="./fig3.png" width=400 height=400/>')


I am working on the file "Final_Report.ipynb".
The first task is:
Objective
Build a model to make predictions on blighted buildings based on real data from data.detroitmi.gov as given by coursera.
Building demolition is very important for the city to turn around and revive its economy. However, it's no easy task. Accurate predictions can provide guidance on potential blighted buildings and help avoid complications at early stages.
Building List
The buildings were defined as described below:

Building sizes were estimated using parcel info downloaded here at data.detroitmi.gov. Details can be found in this notebook.
A event table was constructed from the 4 files (detroit-311.csv, detroit-blight-violations.csv, detroit-crime.csv, and detroit-demolition-permits.tsv) using their coordinates, as shown here.
Buildings were defined using these coordinates with an estimated building size (median of all parcels). Each building was represented as a same sized rectangle.
Can you write Python code for it?

# The resulted buildings:
Image("./data/buildings_distribution.png")


I am working on the file "Final_Report.ipynb".
The first task is:
Plot the results
Can you write Python code for it?

functions = [r_r1,r_pr1,r_AVX1,prj_r1,prj_AVX,step_r1,AVX_r,mth_pr]
for k in range(len(functions)):
    plt.plot(im_sizes,times[:][k], "o-", label = "{}".format(str(functions[k])[19:-1]))
plt.legend(loc="best")


for k in range(len(functions)):
    plt.plot(truncates,times2[:][k],  "o-", label = "{}".format(str(functions[k])[19:-1]))
plt.legend(loc="best")


I am working on the file "Final_Report.ipynb".
The first task is:
The code is shown in the cell below but won't run because of AVX implementation not supported in ipython notebook.
Can you write Python code for it?

'''
Here is the class Gaussianfilter2D

'''
class Gaussianfilter2D():
    '''
    2D gaussian filter on Black and White images

    The filter image satisfies the relation
    $$ C_{i,j} = \sum_{m=0}^{2 l_w} \sum_{n=0}^{2 lw} = K_{m,n} \, I_{i-lw + m, j-l_w+n}$$
    where C is the new image, K the gaussian kernel and I the original image

    Parameters

    ----------

    sigma : float

        standard deviation for the gaussian filter

    truncate : float, optional

        truncate the filter at this many standard deviations
        Default is 4.0

    mode : here are the various mode supported by the class

    mode       |   Ext   |         Input          |   Ext
    -----------+---------+------------------------+---------
    'reflect'  | 3  2  1 | 1  2  3  4  5  6  7  8 | 8  7  6
    'nearest'  | 1  1  1 | 1  2  3  4  5  6  7  8 | 8  8  8
    'constant' | 0  0  0 | 1  2  3  4  5  6  7  8 | 0  0  0
    'wrap'     | 6  7  8 | 1  2  3  4  5  6  7  8 | 1  2  3

    Attributes

    ----------

    kernel_ : 2d-array

        this is the 2D gaussian filter kernel used in the convolution
        with the provided image

    image_benchmark_: 2d-array

        this is the image filtered with the 2D gaussian filter provided
        by the scipy library

    run_time_benchmark_: float

        this is the run time of the 2D gaussian filter provied by the
        scipy library

    image_: 2d-array

        this is the image filtered by the 2D gaussian filter implemented in
        python

    run_time_: float

        this is the run time of the 2D gaussian filter implemented in python

    error_ : float

        this is the norm-2 error of the python function compared with the scipy
        function

    '''
    def __init__(self, sigma, truncate = 4.0 , mode = 'reflect', cval = 0.0,
                    num_threads = 4):

        self.sigma = sigma
        self.truncate = truncate
        self.mode = mode
        self.cval = cval
        self.num_threads = num_threads

        # lw is the...


I am working on the file "Final_Report.ipynb".
The first task is:
This model resulted in an AUC score of 0.858 on test data. Feature importances are shown below:
Can you write Python code for it?


I am working on the file "Final_Report.ipynb".
The first task is:
To reduce variance of the model, since overfitting was observed during training. I also tried to reduce variance by including in more nonblighted buildings by sampling again multiple times with replacement (bagging).
A final AUC score of 0.8625 was achieved. The resulted ROC Curve on test data is shown below:
Can you write Python code for it?


I am working on the file "Final_Report.ipynb".
The first task is:
We find that in the case of a small window the cython mutlithtreading is the most efficient method. Let us compare the run time for various values of truncate and number of threads. First we show our computer hardware
Can you write Python code for it?

%load_ext watermark
%watermark

time_python = []
time_cython = []
time_AVX = []
time_threading = []

truncate_vector = [4,8,12,16,20]
threads = [4,8]
for t in threads:
    for trunc in truncate_vector:
        # create a instance gaussian filter
        gb = Gaussianfilter2D(sigma = 8.0, truncate = trunc, mode = 'reflect', cval = 0.0, num_threads = t)
        gb.filter_python(f)
        time_python.append(gb.run_time_)
        gb.filter_cython(f)
        time_cython.append(gb.run_time_)
        gb.filter_AVX(f)
        time_AVX.append(gb.run_time_)
        filter_cython_threading(gb,f)
        time_threading.append(gb.run_time_)

plt.rcParams.update({'font.size': 18})
plt.figure(figsize=(15,8))
plt.plot(truncate_vector,time_python[0:5], 'bo-',lw =2,label ='python 4 threads')
plt.plot(truncate_vector,time_python[5::], 'bo--',lw =2,label ='python 8 threads')
plt.plot(truncate_vector,time_cython[0:5], 'ro-',lw =2,label='cython 4 threads')
plt.plot(truncate_vector,time_cython[5::], 'ro--',lw =2,label='cython 8 threads')
plt.plot(truncate_vector, time_AVX[0:5], 'go-',lw =2,label ='AVX 4 threads')
plt.plot(truncate_vector, time_AVX[5::], 'go--',lw =2,label ='AVX 8 threads')
plt.plot(truncate_vector,time_threading[0:5], 'ko-',lw =2,label='cython multithreaded 4 threads')
plt.plot(truncate_vector,time_threading[5::], 'ko--',lw =2,label='cython multithreaded 8 threads')
plt.legend(loc='best')


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
### This will split the training and testing sets into a more useful format for loading into models
Can you write Python code for it?

#names_x = ['Length','Year','Month','lnEngEst','Time', 'Highway', 'District', 'County' ]
names_x = ['Length','Year','Month','lnEngEst','Time', 'NBidders']
names_y = ['MoreOrLessThan10']

df_Train_x = df_train[ names_x ]
df_Train_y = df_train[ names_y ]
df_Test_x = df_test[ names_x ]
df_Test_y = df_test[ names_y ]

df_Train_x.head()

test_X.head()


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
1) Splitting the Data into Training and Testing Sets
Can you write Python code for it?

df_test = df[(df.Year == 2016) & (df.Month == 4)]
print len(df_test) , 'projects in April 2016'

df_train = df[(df.Year != 2016) | (df.Month != 4)]
print len(df_train) ,'projects from Jan 2010 to April 2016'

#df_train[['Year','Month']].tail()

#df_test.columns


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Bid Model Comparison by Graph

Red is Bad  (beyond the 10% threshhold)
Green is Good   (within 10% of actual)

TxDOT Estimates
Can you write Python code for it?

cmap = {'0': 'r', '1': 'g' }
df_test.loc[:,'cWithin10Percent'] = df_test.Within10Percent.apply(lambda x: cmap[str(x)])
print df_test.plot('lnEngEst', 'lnWinBid', kind='scatter', c=df_test.cWithin10Percent)


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Classifying using Decision Tree
Can you write Python code for it?

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf = clf.fit(df_Train_x, df_Train_y)

clf.predict(df_Test_x)

clf.score(df_Test_x, df_Test_y)


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Exploring the Data
We see that Bids are Log Normally Distributed
Can you write Python code for it?

sns.jointplot(x="EngEst", y="WinBid", data=df, kind="reg"); sns.jointplot(x="lnEngEst", y="lnWinBid", data=df, kind="reg");


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Formatting & Identifying the Data
Can you write Python code for it?

df = df[df['Rank'] == 1]
df = df[df['Type'] == 'Construction']

df = df.drop(' From', 1)
df = df.drop('To', 1)
df = df.drop('Contract Description', 1)
df = df.drop('Contractor', 1)
df = df.drop('Contract Category', 1)
df = df.drop('LNG MON', 1)
df = df.drop('MONTH', 1)

df['Award Amount'] = df['Award Amount'].str.lstrip('$')
df['Engineers Estimate'] = df['Engineers Estimate'].str.lstrip('$')
df['Award Amount'] = df['Award Amount'].str.replace(',','').astype(float)
df['Engineers Estimate'] = df['Engineers Estimate'].str.replace(',','').astype(float)

#Renaming Variables
df['EngEst'] = df['Engineers Estimate']
df['NBidders'] = df['Number of Bidders']
df['Date'] = pd.to_datetime(df['Letting Date'])
df.set_index('Date' , inplace=True)
df['Year'] = df.index.year
df['Month'] = df.index.month
df['WinBid'] = df['Award Amount']

# Creating New Varialbes
df['Diff'] = df['EngEst'] - df['WinBid']
df['lnWinBid'] = np.log(df['WinBid'])
df['lnEngEst'] = np.log(df['EngEst'])
df['DiffLn'] = df['lnWinBid'] - df['lnEngEst']
df['Within10Percent'] = 1
df['PercentOff'] = df['Diff'] / df['EngEst']
df['MoreOrLessThan10'] = 0
df['LessThan10'] = 0
df['MoreThan10'] = 0

df.loc[(df.PercentOff > .10) , 'Within10Percent'] = 0
df.loc[(df.PercentOff < -.10) , 'Within10Percent'] = 0
df.loc[(df.PercentOff > .10) , 'MoreOrLessThan10'] = 1
df.loc[(df.PercentOff < -.10) , 'MoreOrLessThan10'] = 2
df.loc[(df.PercentOff > .10) , 'MoreThan10'] = 1
df.loc[(df.PercentOff < -.10) , 'LessThan10'] = 1

print len(df)


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Load in the Data
Can you write Python code for it?

df = pd.read_csv('C:/Users/Collin/Documents/Python_Projects/TxDOT/data/Bid_Data.csv')
len(df)


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Model Predictions
Can you write Python code for it?

predcmap = {'0': 'r', '1': 'g' }
df_test.loc[:,'cPredWithin10Percent'] = df_test.PredWithin10Percent.apply(lambda x: predcmap[str(x)])
print df_test.plot('lnpred', 'lnWinBid', kind='scatter', c=df_test.cPredWithin10Percent)


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Model given Wining Bid is Less Than 10% of TxDot Estimate
Can you write Python code for it?

#subsets the training data to just those who were less than 10% of the TxDot Estimate
df_train_less = df[df.LessThan10 == 1]

model_5 = smf.ols(formula = 'lnWinBid ~ lnEngEst+Year+Month+Year*Month+NBidders+NBidders*Year+Time', data = df_train_less).fit()
model_5.summary()


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Our Model Prediction of Winning Bid will be a Weighted average of Bid predictions with weights being the probability of being classified as such class found from Logistic Regression
Can you write Python code for it?


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
Predicting
Can you write Python code for it?

df_test.loc[:,'Lnprediction_within'] = model_3.predict(test_X)
df_test.Lnprediction_within.head()


I am working on the file "Final_Version_With_Decision_Tree.ipynb".
The first task is:
We now need to build and train a model that can classify a bid as over or under the TxDOT estimate

Here We can see how this classification might look
Can you write Python code for it?

cmap = {'0': 'g', '1': 'r', '2': 'b' }
df['cMoreOrLessThan10'] = df.MoreOrLessThan10.apply(lambda x: cmap[str(x)])
print df.plot('EngEst', 'WinBid', kind='scatter', c=df.cMoreOrLessThan10)

cmap = {'0': 'g', '1': 'r', '2': 'b' }
df['cMoreOrLessThan10'] = df.MoreOrLessThan10.apply(lambda x: cmap[str(x)])
print df.plot('lnEngEst', 'lnWinBid', kind='scatter', c=df.cMoreOrLessThan10)


I am working on the file "Final_ensemble.ipynb".
The first task is:
In here will be the section where we import the relevant python code from each project
Now, we'll import the three csv datasets, merge them on PassengerID, then ensemble
Can you write Python code for it?

titanic_rf=pd.read_csv('./ensemble_rf.csv')
titanic_gboost=pd.read_csv('./ensemble_gboost.csv')
titanic_svm=pd.read_csv('./ensemble_svm.csv')
titanic_nb=pd.read_csv('./ensemble_nb.csv')
titanic_logit=pd.read_csv('./ensemble_logit.csv')

titanic_ensemble=pd.merge(titanic_rf, titanic_gboost, left_index=True, right_index=True)
titanic_ensemble=pd.merge(titanic_ensemble, titanic_svm, left_index=True, right_index=True)
titanic_ensemble=pd.merge(titanic_ensemble, titanic_nb, left_index=True, right_index=True)
titanic_ensemble=pd.merge(titanic_ensemble, titanic_logit, left_index=True, right_index=True)

print titanic_ensemble.head()


I am working on the file "Final_ensemble.ipynb".
The first task is:
Let's see what the correlations are across the learners, to find how much similarity in information they share
Can you write Python code for it?


I am working on the file "Final_ensemble.ipynb".
The first task is:
Now let's perform the ensembling, using the same method for Gradient Boosting we used earlier
Can you write Python code for it?

titanic_ensemble=titanic_ensemble.values

#clf=GradientBoostingClassifier().fit(titanic_ensemble,ensemble_target)

feat_param=['deviance','exponential']
score=0
for feature in feat_param:
    clf = GradientBoostingClassifier(loss=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_ensemble,ensemble_target,cv=5 )
    if score_test.mean()>score:
        loss_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

score=0
for feature in np.linspace(.05,.45,11):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_ensemble,ensemble_target,cv=5 )
    if score_test.mean()>score:
        rate_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

score=0
for feature in range(100,1001,100):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=rate_out, n_estimators=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_ensemble,ensemble_target,cv=5)
    if score_test.mean()>score:
        feat_n_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

score=0

for feature in range(1,21):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=rate_out, n_estimators=feat_n_out,\
                                     max_depth=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_ensemble,ensemble_target,cv=5)
    if score_test.mean()>score:
        depth_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

score=0
for feature in range(1,21):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=rate_out, n_estimators=feat_n_out,\
                                     max_depth=depth_out, min_samples_split=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_ensemble,ensemble_target,cv=5)
    if score_test.mean()>score:
        sample_out=feature
       ...


I am working on the file "Final_ensemble.ipynb".
The first task is:
OK, let's normalize our continous variables
Can you write Python code for it?

cols_to_norm=['Age','Fare']
col_norms=['Age_z','Fare_z']

titanic[col_norms]=titanic[cols_to_norm].apply(lambda x: (x-x.mean())/x.std())

titanic['cabin_clean']=(pd.notnull(titanic.Cabin))

from sklearn import svm, datasets
from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.cross_validation import StratifiedKFold, KFold, cross_val_score, train_test_split
from sklearn.ensemble import  GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression

titanic_target=titanic.Survived.values
features=['Sex','SibSp','Parch','Pclass_1','Pclass_2','Pclass_3','Emb_C','Emb_Q','Emb_S',\
                         'Emb_nan','Age_ct_C','Age_ct_A','Age_ct_S', 'Sp_ct','Age_z','Fare_z',\
                        'Ti_Dr', 'Ti_Master', 'Ti_Mil', 'Ti_Miss', 'Ti_Mr', 'Ti_Mrs', 'Ti_Other', 'Ti_Rev',\
                         'Fl_AB', 'Fl_CD', 'Fl_EFG', 'Fl_nan']
titanic_features=titanic[features].values

titanic_features, ensemble_features, titanic_target, ensemble_target= \
    train_test_split(titanic_features,
                     titanic_target,
                     test_size=.1,
                     random_state=7132016)


I am working on the file "Final_setup_GBoost.ipynb".
The first task is:
2.) Learning Rate
Can you write Python code for it?

score=0
for feature in np.linspace(.05,.45,11):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_features,titanic_target,cv=10 )
    if score_test.mean()>score:
        rate_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

print rate_out


I am working on the file "Final_setup_GBoost.ipynb".
The first task is:
4.) Maximum depth of tree
Can you write Python code for it?

score=0
for feature in range(1,21):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=rate_out, n_estimators=feat_n_out,\
                                     max_depth=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_features,titanic_target,cv=10 )
    if score_test.mean()>score:
        depth_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

print depth_out


I am working on the file "Final_setup_GBoost.ipynb".
The first task is:
6.) The number of samples that must appear in a leaf
Can you write Python code for it?

score=0
for feature in range(1,21):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=rate_out, n_estimators=feat_n_out,\
                                     max_depth=depth_out, min_samples_split=sample_out,\
                                     min_samples_leaf=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_features,titanic_target,cv=10 )
    if score_test.mean()>score:
        sample_leaf_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

print sample_leaf_out


I am working on the file "Final_setup_GBoost.ipynb".
The first task is:
7.) Min required weighted fraction of samples in a leaf or node
Can you write Python code for it?

score=0

for feature in np.linspace(0.0,0.5,10):
    clf = GradientBoostingClassifier(loss=loss_out, learning_rate=rate_out, n_estimators=feat_n_out,\
                                     max_depth=depth_out, min_samples_split=sample_out,\
                                     min_samples_leaf=sample_leaf_out, min_weight_fraction_leaf=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_features,titanic_target,cv=10 )
    if score_test.mean()>score:
        frac_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

print frac_out


I am working on the file "Final_setup_GBoost.ipynb".
The first task is:
Now, what we're going to do is go stepwise through many of the features of the random forest classifier, to figure out which parameters will give us the best fit. We'll go in the following order:

loss
learning_rate
n_estimators
max_depth (depth of tree)
min_samples_split (number of samples you need to be able to create a new branch/node)
min_samples_leaf
min_weight_fraction_leaf
subsample
max_features (number of features considered at each split)
max_leaf_nodes (implicitly, if the best is not none, then we'll be ignoring the max depth parameter
warm_start

1.) Loss Criteria
Can you write Python code for it?

feat_param=['deviance','exponential']
score=0
for feature in feat_param:
    clf = GradientBoostingClassifier(loss=feature, random_state=7112016)
    score_test= cross_val_score(clf,titanic_features,titanic_target,cv=10 )
    if score_test.mean()>score:
        loss_out=feature
        score_diff=score_test.mean()-score
        score=score_test.mean()

print loss_out


I am working on the file "Final_setup_Random_Forest.ipynb".
The first task is:
2.) Loss Criterion
Can you write Python code for it?

crit_param = ['gini','entropy']
score=0
for feature in crit_param:
    clf = RandomForestClassifier(n_estimators=n_out, criterion=feature, oob_score=True, random_state=7112016)
    clf.fit(titanic_features,titanic_target)
    score_test = clf.oob_score_
    if score_test>score:
        crit_out=feature
        score_diff=score_test-score
        score=score_test

print crit_out


I am working on the file "Final_setup_Random_Forest.ipynb".
The first task is:
3.) Number of features considered at each split
Can you write Python code for it?

feat_param = ['sqrt','log2',None]
score=0
for feature in feat_param:
    clf = RandomForestClassifier(n_estimators=n_out, criterion=crit_out, max_features=feature,\
                                 oob_score=True, random_state=7112016)
    clf.fit(titanic_features,titanic_target)
    score_test = clf.oob_score_
    if score_test>score:
        max_feat_out=feature
        score_diff=score_test-score
        score=score_test

print max_feat_out


I am working on the file "Final_setup_Random_Forest.ipynb".
The first task is:
4.) Maximum depth of tree
Can you write Python code for it?

score=0
for feature in range(1,21):
    clf = RandomForestClassifier(n_estimators=n_out, criterion=crit_out, max_features=max_feat_out, \
                                 max_depth=feature, oob_score=True, random_state=7112016)
    clf.fit(titanic_features,titanic_target)
    score_test = clf.oob_score_
    if score_test>score:
        depth_out=feature
        score_diff=score_test-score
        score=score_test

print depth_out


I am working on the file "Final_setup_SVM.ipynb".
The first task is:
22
0.5
0.802584778872
Can you write Python code for it?


I am working on the file "Final_setup_SVM.ipynb".
The first task is:
Our key parameters here are the penalty term, and the best k features from the univariate analysis
Can you write Python code for it?

score=0

for x in range(10,43):
    for y in np.linspace(.1,.5,5):
        var_filter=SelectKBest(f_classif)
        clf=svm.SVC(kernel='rbf')
        pipe_svm = Pipeline([('anova', var_filter), ('svc', clf)])
        pipe_svm.set_params(anova__k=x, svc__C=y)
        score_test = cross_validation.cross_val_score(pipe_svm, titanic_features, titanic_target, n_jobs=1, \
                                                       cv=StratifiedKFold(titanic_target, n_folds=10, shuffle=True, random_state=7132016))
        if score_test.mean()>score:
            score=score_test.mean()
            k_out=x
            C_out=y

print k_out
print C_out
print score


I am working on the file "Final_setup_SVM.ipynb".
The first task is:
Prep the Kaggle test data, as well as the ensembling test data
Can you write Python code for it?

test_data=pd.read_csv('./test.csv')

test_data.Sex.replace(['male','female'],[True,False], inplace=True)
test_data.Age= test_data.groupby(['Sex','Pclass'])[['Age']].transform(lambda x: x.fillna(x.mean()))
test_data.Fare= titanic.groupby(['Pclass'])[['Fare']].transform(lambda x: x.fillna(x.mean()))
titanic_class=pd.get_dummies(test_data.Pclass,prefix='Pclass',dummy_na=False)
test_data=pd.merge(test_data,titanic_class,on=test_data['PassengerId'])
test_data=pd.merge(test_data,pd.get_dummies(test_data.Embarked, prefix='Emb', dummy_na=True), on=test_data['PassengerId'])
titanic['Floor']=titanic['Cabin'].str.extract('^([A-Z])', expand=False)
titanic['Floor'].replace(to_replace='T',value=np.NaN ,inplace=True)
titanic=pd.merge(titanic,pd.get_dummies(titanic.Floor, prefix="Fl", dummy_na=True),on=titanic['PassengerId'])
test_data['Age_cut']=pd.cut(test_data['Age'],[0,17.9,64.9,99], labels=['C','A','S'])
test_data=pd.merge(test_data,pd.get_dummies(test_data.Age_cut, prefix="Age_ct", dummy_na=False),on=test_data['PassengerId'])

test_data['Title']=test_data['Name'].str.extract(', (.*)\.', expand=False)
test_data['Title'].replace(to_replace='Mrs\. .*',value='Mrs', inplace=True, regex=True)
test_data.loc[test_data.Title.isin(['Col','Major','Capt']),['Title']]='Mil'
test_data.loc[test_data.Title=='Mlle',['Title']]='Miss'
test_data.loc[test_data.Title=='Mme',['Title']]='Mrs'
test_data['Title_ct']=test_data.groupby(['Title'])['Title'].transform('count')
test_data.loc[test_data.Title_ct<5,['Title']]='Other'
test_data=pd.merge(test_data,pd.get_dummies(test_data.Title, prefix='Ti',dummy_na=False), on=test_data['PassengerId'])

test_data['NameTest']=test_data.Name
test_data['NameTest'].replace(to_replace=" \(.*\)",value="",inplace=True, regex=True)
test_data['NameTest'].replace(to_replace=", M.*\.",value=", ",inplace=True,...


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
2. Loading Loan Data / Validity Checks
The loan.csv has been loaded and transformed using the lending_club_loan_data-eda.py script. The data types of the all the attributes have been appropriately defined, as well as the categories of all the nominal and ordinal categorical variables. The transformed data set has ~890K observations, 75 attributes, and it is ~420MB in size. Note, that some 'NA' values exist in specific float and str variables, but this is expected.
Can you write Python code for it?

loan_df = pd.read_pickle(loan_df_path)

loan_df.shape

loan_df.info()

loan_df.head()


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
3. Exploratory  Data Analysis
A. Loan Amounts
Can you write Python code for it?

sns.distplot(loan_df.loc[:, 'loan_amnt'], hist=True, kde=True, rug=False,
             norm_hist=True, axlabel='Loan Amount')
plt.title('Loan Amount Distribution')
plt.show()

grouped = loan_df.groupby(by=['issue_d'])
grouped_agg = (grouped['loan_amnt']
               .agg(np.sum)
               .rename('loanbook_amnt'))
grouped_agg_df = grouped_agg.reset_index()
grouped_agg_ts = pd.Series(data=grouped_agg_df['loanbook_amnt'].values,
                           index=grouped_agg_df['issue_d'])
del grouped_agg_df

fig = plt.figure(figsize=(13, 8))
ax = fig.add_subplot(111)
grouped_agg_ts.plot(ax=ax)
ax.set_xlabel('Date Issued [\'issue_d\']')
ax.set_ylabel('Loan Book Amount')
ax.set_title('Loan Book Growth\n[\'loan_df\']')
plt.show()


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
=> 'verification_status_joint': Also Not Important!
Can you write Python code for it?

individual_loans_ix = (loan_df['application_type'] != 'JOINT')

for attrib in ['application_type', 'verification_status', 'verification_status_joint']:
    s = 'Attribute: "%s":' % attrib
    print(s)
    print('-' * (len(s)+3))
    pprint(loan_df.loc[individual_loans_ix, attrib].head(n=3))
    s = '\nValue Counts:'
    print(s)
    print('-' * (len(s)+3))
    pprint(pd.value_counts(loan_df.loc[individual_loans_ix, attrib]))
    print('\n')


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
B. Loan Statuses & Grades
Can you write Python code for it?

s = '\nBad Loans [Flagged as \'0\']:'
print(s)
print('-'*(len(s) + 3))
pprint(bad_loans)

s = '\nGood Loans [Flagged as \'1\']:'
print(s)
print('-'*(len(s) + 3))
pprint(good_loans)

# Countplot of Loan Statuses
sns.countplot(x=response, data=loan_df)
plt.title('Countplot of Loan Statuses\n[\'loan_df\']')
plt.xticks(range(2), ['Bad Loans', 'Good Loans'], fontweight='bold')
plt.xlabel('Loan Status')
plt.ylabel('No. of Records')
plt.show()

fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(13,8), sharey=True)

sns.boxplot(x='loan_status', y='loan_amnt', data=loan_df, ax=ax1)
ax1.set_xticks(range(2))
ax1.set_xticklabels(['Bad Loans', 'Good Loans'], fontweight='bold')
ax1.set_xlabel('Loan Status')
ax1.set_ylabel('Loan Amount')

sns.boxplot(x='loan_status', y='loan_amnt', hue='grade', data=loan_df, ax=ax2)
ax2.set_xticks(range(2))
ax2.set_xticklabels(['Bad Loans', 'Good Loans'], fontweight='bold')
ax2.legend(loc='upper center', framealpha=0.5, prop={'weight': 'bold'})
ax2.legend_.set_title('Grade', prop={'weight': 'bold', 'size': 12})
ax2.set_xlabel('Loan Status')
ax2.set_ylabel('Loan Amount')

plt.suptitle('"Loan Amount" Distribution per "Loan Class" & "Grade"\n[\'loan_df\']',
             fontsize=16, fontweight='bold')
plt.tight_layout(h_pad=2, w_pad=2, rect=[0,0,0.93,0.93])
plt.show()

fig = plt.figure(figsize=(13, 8))
ax = fig.add_subplot(111)

grouped = loan_df.groupby(by=['issue_d', 'grade'])
grouped_agg = (grouped['loan_amnt']
               .agg(np.sum)
               .rename('loanbook_amnt_per_grade'))
grouped_agg_df = grouped_agg.reset_index()
grouped_agg_df1 = grouped_agg_df.pivot_table(values='loanbook_amnt_per_grade',
                                             index = 'issue_d',
                                             columns=['grade'])

grouped_agg_df1.plot(ax=ax)
ax.set_title('Loan Book Growth per "Loan Grade"\n[\'loan_df\']')
ax.legend(loc='best', prop={'weight': 'bold'})
ax.legend_.set_title('Grade', prop={'weight': 'bold', 'size': 12})
ax.set_xlabel('Date Issued...


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
B. Validity of date fields
All date fields are valid:

'issue_d': The month which the loan was funded ('Jun-2007'-'Dec-2015').
'earliest_cr_line': The month the borrower's earliest reported credit line was opened ('Jan-1944'-'Nov-2012').
'last_pymnt_d': Last month payment was received ('Dec-2007'-'Jan-2016').
'next_pymnt_d': Next scheduled payment date ('Dec-2007'-'Mar-2016').
'last_credit_pull_d': The most recent month LC pulled credit for this loan ('May-2007'-'Jan-2016').
Can you write Python code for it?

loan_df.select_dtypes(include=['datetime64']).describe()

dt_attribs = list(loan_df.select_dtypes(include=['datetime64']).columns)
dt_attribs

for attrib in dt_attribs:
    pprint(loan_df.loc[:3,attrib])
    print('\n')


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
C. Loan Book Distribution across the U.S. States (D3 Visualizations by leveraging the "Bokeh" library)
Here, we provide two choropleth maps concerning the Loan Book Value and the Loan Book Volume distribution across the U.S. States. To do so, we have used the "Bokeh" Python library, a GeoJSON file which defines the U.S. States boundaries and it has been produced from a cartographic boundary shapefile that is provided from the official site of the U.S. Census Bureau, and the Pandas DataFrame grouped_agg_df, where we aggregate the number, and the value of loans per U.S. State. "Bokeh" is a Python library for interactive D3 visualizations!
Can you write Python code for it?

# Compute the "Loan Book Amount & Volume" per "US State"
grouped = loan_df.groupby(by=['addr_state'])
grouped_agg = (grouped[['loan_amnt']].agg(np.sum)
               .rename(columns={'loan_amnt': 'loanbook_amnt_per_state'}))
grouped_agg['loanbook_vol_per_state'] = grouped['loan_amnt'].agg(np.count_nonzero)
grouped_agg_df = grouped_agg.reset_index()
grouped_agg_df.head()

# Load the necessary libraries for the D3 Visualization
from bokeh.io import show, output_notebook
from bokeh.palettes import (
    YlOrRd9 as palette1,
    YlGnBu9 as palette2)
from bokeh.plotting import figure
from bokeh.models import (
    GeoJSONDataSource,
    LogColorMapper,
    HoverTool,
    LogTicker,
    ColorBar)

# Load the enriched GeoJSON Data Source, with the loanbook measures of interest
with open(us_states_GeoJSON, 'r') as f:
    geo_source = GeoJSONDataSource(geojson=f.read())

# Output the Choropleth Plots in Notebook
output_notebook()

# PROVIDE THE CHOROPLETH OF "LOAN BOOK AMOUNT PER STATE"
palette1.reverse()
color_mapper = LogColorMapper(palette=palette1,
                              low=grouped_agg_df['loanbook_amnt_per_state'].min(),
                              high=grouped_agg_df['loanbook_amnt_per_state'].max())

# Define the figure "Tools" we want to make available
TOOLS = "pan, wheel_zoom, reset, hover, save"

# Plot the figure
# Define the figure dimensions and its general details
p = figure(title="Loan Book Value by U.S. States", tools=TOOLS,
           plot_width=960, plot_height=500,
           x_range=(0, 960), y_range=(500, 0),
           x_axis_location=None, y_axis_location=None)

# Render the "Bokeh" patches in Glyph
p.patches('xs', 'ys', source=geo_source,
          fill_color={'field': "loanbook_amnt_per_state" ,'transform': color_mapper},
          fill_alpha=0.7, line_color="white", line_width=0.5)

# Add a Hover Tools over the US States
hover = p.select_one(HoverTool)
hover.point_policy = "follow_mouse"
hover.tooltips = [
    ("State", "@state"),
    ("Loan Book Amount",...


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
Consequently, the attributes that we are going to further consider are enlisted below.
Can you write Python code for it?

imp_attribs0 = list(loan_df.head()
                    .select_dtypes(exclude=['object']).columns)

loan_df.loc[:,imp_attribs0].info()


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
E.  Loan Grades
E1. Loan Grades across their Statuses ("Bad ['0']"  /  "Good Loans ['1']")
Can you write Python code for it?

freq_tables(loan_df, ['grade'], response,
            barplot=True, matplotlib_style='seaborn-whitegrid')