I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
E. Integer Attributes
All integer attributes are important for the predictive model we want to build. The 'NA' values where exist have been set to '-9999'.

'delinq_2yrs': The number of 30+ days past-due incidences of delinquency in the borrower's credit file for the past 2 years.
'inq_last_6mths': Number of credit inquiries in past 12 months.
'mths_since_last_delinq': The number of months since the borrower's last delinquency.
'mths_since_last_record': The number of months since the last public record.
'open_acc': The number of open credit lines in the borrower's credit file.
'pub_rec': Number of derogatory public records.
'total_acc': The total number of credit lines currently in the borrower's credit file.
'collections_12_mths_ex_med': Number of collections in 12 months excluding medical collections.
'mths_since_last_major_derog': Months since most recent 90-day or worse rating.
'acc_now_delinq': The number of accounts on which the borrower is now delinquent.
'tot_coll_amt': Total collection amounts ever owed.
'open_acc_6m': Number of open trades in last 6 months.
'open_il_6m': Number of currently active installment trades.
'open_il_12m': Number of installment accounts opened in past 12 months.
'open_il_24m': Number of installment accounts opened in past 24 months.
'mths_since_rcnt_il': Months since most recent installment accounts opened.
'open_rv_12m': Number of revolving trades opened in past 12 months.
'open_rv_24m': Number of revolving trades opened in past 24 months.
'inq_fi': Number of personal finance inquiries.
'total_cu_tl': Number of finance trades.
'inq_last_12m': Number of credit inquiries in past 12 months.
Can you write Python code for it?

loan_df.select_dtypes(include=['int']).info()

pprint(loan_df.select_dtypes(include=['int']).iloc[:,2:]
       .describe())


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
E2. Interest Rate by Loan Grade
Can you write Python code for it?

sns.boxplot(x='grade', y='int_rate', data=loan_df)
plt.xlabel('Loan Grade')
plt.ylabel('Interest Rate')
plt.title('"Interest Rate" by "Loan Grade"\n[\'loan_df\']',
          fontsize=16, fontweight='bold')
plt.xticks(range(7), list(loan_df['grade'].cat.categories),
           fontweight='bold')
plt.show()


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
F. Nominal & Ordinal Categorical Attributes
Can you write Python code for it?

response = 'loan_status'
categ_df = loan_df.select_dtypes(include=['category'])
categ_attribs = list(categ_df.columns)

nominal_categ_attribs = []; ordinal_categ_attribs = []
for attrib in categ_attribs:
    if categ_df[attrib].cat.ordered:
        ordinal_categ_attribs.append(attrib)
    else:
        nominal_categ_attribs.append(attrib)


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
F2. Ordinal Categorical Attributes
We chose to define as ordinal categorical attributes the variables below:

'term': The number of payments on the loan
Categories (2, object): [36 months < 60 months]

'grade': LC assigned loan grade.
Categories (7, object): [G < F < E < D < C < B < A]

'sub_grade': LC assigned loan subgrade.
Categories (35, object): [G5 < G4 < G3 < G2 ... A4 < A3 < A2 < A1]

'emp_length': Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.
Categories (12, object): [n/a < < 1yr < 1yr < 2yrs ... 7yrs < 8yrs < 9yrs < 10+yrs]

'verification_status': Indicates if the borrowers' income was verified by LC, not verified, or if the income source was verified.
Categories (3, object): [Not Verified < Source Verified < Verified]

'verification_status_joint': Indicates if the co-borrowers' joint income was verified by LC, not verified, or if the income source was verified.
Categories (4, object): [Not Known < Not Verified < Source Verified < Verified]

Note
Among them, the 'verification_status_joint' variable appears a plethora of 'Not Known' values, but we choose not to exclude it from our consideration, at least for the time being. This field keeps the verification status of the co-borrowers' joint income, and can provide useful information for the 'JOINT' loan applications. Note also, that the sum of the value counts concerning its remaining, informative values, i.e. 'Not Verified', 'Source Verified' and 'Verified', equals the number of the recorded 'JOINT' loan applications, i.e. 511:
```
Nominal Categorical Attribute: "application_type":

0    INDIVIDUAL
1    INDIVIDUAL
Name: application_type, dtype: category
Categories (2, object): [INDIVIDUAL, JOINT]
Value Counts:
INDIVIDUAL    886868
JOINT            511
Name: application_type, dtype: int64
Ordinal Categorical Attribute: "verification_status_joint":
0    Not Known
1    Not Known
Name:...
Can you write Python code for it?

pprint(ordinal_categ_attribs)

categ_df[ordinal_categ_attribs].info()

for attrib in ordinal_categ_attribs:
    s = 'Ordinal Categorical Attribute: "%s":' % attrib
    print(s)
    print('-' * (len(s)+3))
    pprint(categ_df.loc[:3, attrib])
    s = '\nValue Counts:'
    print(s)
    print('-' * (len(s)+3))
    pprint(pd.value_counts(categ_df[attrib]))
    print('\n')


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
G. Response Variable ('loan_status')
Possible values of the 'loan_status' attribute:

'Default' ['0A']: Loans that have been defaulted.
'Does not meet the credit policy. Status:Charged Off' (Written-off, ['0B']): Loans that do not meet the LC credit policy and has been written-off.
'Charged Off' (Written-off, ['0C']): The loan balance has been reduced to zero by recognizing the recorded value as an expense.
'Late (31-120 days)' (Red Loans, ['0D']): Loan the installment payments of which, have been delayed for 31-120 days.
'Late (16-30 days)' (Yellow Loans, ['0E']): Loan the installment payments of which have been delayed for 16-30 days.
'Issued' ['1A']: Loans that have been just issued.
'In Grace Period' ['1B']: Loans that have been issued and they are currently being in grace period.
'Current' ['1C']: Loans that are currently active and serviced.
'Does not meet the credit policy. Status:Fully Paid' ['1D']: Loans that do not meet the LC credit policy, but they have been fully-paid.
'Fully Paid' ['1E']: Loans that have been fully paid.
Can you write Python code for it?

s = 'Response Variable: "%s":' % attrib
print(s)
print('-' * (len(s)+3))
pprint(loan_df.loc[:3, response])
s = '\nValue Counts:'
print(s)
print('-' * (len(s)+3))
pprint(pd.value_counts(loan_df[response]))
print('\n')


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
G1. Group in "Bad" ['0'] / "Good Loans" ['1']: Class Imbalance
Can you write Python code for it?

# Group the loan classes in "Bad Loans" & "Good Loans"
bad_loans = ['Default', # ['0A']
             'Does not meet the credit policy. Status:Charged Off', # ['0B']
             'Charged Off', # ['0C']
             'Late (31-120 days)', # ['0D']
             'Late (16-30 days)'] # ['0E']

good_loans = ['Issued', # ['1A']
              'In Grace Period', # ['1B']
              'Current', # ['1C']
              'Does not meet the credit policy. Status:Fully Paid', # ['1D']
              'Fully Paid'] # ['1E']

# Denote the "Bad Loans" ("Good Loans") with the "0" ("1") Flag
loan_df.loc[:,response].replace(to_replace=bad_loans, value='0', inplace=True)
loan_df.loc[:,response].replace(to_replace=good_loans, value='1', inplace=True)

# BAD LOANS
num_bad_loans =  len(loan_df.loc[loan_df[response] == '0', response])
print('Bad Loans: %d' % num_bad_loans)

# GOOD LOANS
num_good_loans = len(loan_df.loc[loan_df[response] == '1', response])
print('Good Loans: %d' % num_good_loans)

# BAD to GOOD LOANS RATIO
bad_to_good_loans_ratio = round(num_bad_loans / num_good_loans, ndigits=2)
print('Bad to Good Loans percentage: %.2f ' % (bad_to_good_loans_ratio * 100) + '%')

# Countplot of Loan Statuses
sns.countplot(x=response, data=loan_df)
plt.title('Countplot of Loan Statuses\n[\'loan_df\']')
plt.xticks(range(2), ['Bad Loans', 'Good Loans'], fontweight='bold')
plt.xlabel('Loan Status')
plt.ylabel('No. of Records')
plt.show()


I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
H. Remove unimportant attributes & record lines from "loan_df" DataFrame
According to the previous discussion, the attributes below are not important for the response variable outcome, and we choose to remove them from the loan_df data set, to enhance the predictive performance of our model.

'policy_code': Publicly available (policy_code=1), New products not publicly available (policy_code=2).
'pymnt_plan': Indicates if a payment plan has been put in place for the loan.
'application_type': Indicates whether the loan is an individual application or a joint application with two co-borrowers.
'verified_status_joint': Indicates if the co-borrowers' joint income was verified by LC, not verified, or if the income source was verified.

Furthermore, keeping the record lines of the 'JOINT' loan applications is now meaningless, and we drop them. The same is true for both the attributes 'annual_inc_joint' and 'dti_joint ', which now have 'NA' values only. We also drop them.
Finally, we are going not to take in consideration the few text attributes that exist in loan_df, at least in this first stage of predictive modelling. These text attributes, which have been stored in the str_attribs list above, are:

'desc': Loan description provided by the borrower.
'emp_title': The job title supplied by the Borrower when applying for the loan (Note: employer title replaces Employer Name for all loans listed after 9/23/2013).
'title': The loan title provided by the borrower.
'url': URL for the LC page with listing data.
Can you write Python code for it?

loan_df = loan_df.loc[individual_loans_ix,:]

loan_df.shape

unimp_attribs = ['policy_code',
                 'pymnt_plan',
                 'application_type',
                 'verification_status_joint',
                 'annual_inc_joint',
                 'dti_joint']
for attrib in unimp_attribs:
    del loan_df[attrib]

loan_df.shape


I am working on the file "FinanceDataReader_여러_종목의_가격을_한번에_가져오기의_사본.ipynb".
The first task is:
개별 종목의 2017년 수익률 비교
Can you write Python code for it?

# 정규화: 가격의 시작을 1.0 으로

df_norm = df / df.iloc[0]
df_norm.plot()


I am working on the file "FinanceDataReader_여러_종목의_가격을_한번에_가져오기의_사본.ipynb".
The first task is:
여러 종목의 종가(Close) 가져오기

ticker 리스트의 종가(Close) 가져와 리스트로 만들기
pd.concat()로 합치기
Can you write Python code for it?

df_krx = fdr.StockListing('KRX')
tiker_list = list(df_krx.Symbol)

import os
os.chdir("..")
os.chdir("home")
os.getcwd()

df_krx.head()

# ticker 리스트의 종가(Close) 가져와 리스트로 만들기

import pandas as pd

df_list_volume = [fdr.DataReader(ticker, '2000-01-01', '2018-12-28')['Volume'] for ticker in tiker_list]

len(df_list_volume)

df_krx_name_code_list = []
for i in range(len(df_krx)):
    name_and_code= df_krx.iloc[i].Name + '(' + df_krx.iloc[i].Symbol + ')'
    df_krx_name_code_list.append(name_and_code)

# pd.concat()로 합치기
import pandas as pd
df_stock_data_volume=pd.concat(df_list_volume, axis=1)
df_stock_data_volume.columns = df_krx_name_code_list
#df_stock_data = df_stock_data.dropna()
df_stock_data_volume.head()

df_stock_data.tail()

df_stock_data_volume.to_csv('2000-2018_stock_data_volume.csv')
# df_krx.to_csv('krx_company_list_20181228.csv')

os.listdir()


I am working on the file "FinanceDataReader_여러_종목의_가격을_한번에_가져오기의_사본.ipynb".
The first task is:
한 종목의 가격 가져오기
Can you write Python code for it?

import FinanceDataReader as fdr

# KB Finance(105560)
df = fdr.DataReader('105560', '2017-01-01', '2017-12-31')
df.head(15)


I am working on the file "FindMe.ipynb".
The first task is:
After tuning the hyperparameters, we're getting good face identification over our test images.
Build Dataset
Base Corpus
Now let's use this to build a base corpus of "these faces are not mine" so we can augment it later with the face we want to target.
Can you write Python code for it?

#Creates cropped faces for imgs matching 'test_imgs/group*.jpg'

def crop(img: np.ndarray, x: int, y: int, width: int, height: int) -> np.ndarray:
    """Returns an image cropped to a given bounding box of top-left coords, width, and height"""
    return img[y:y+height, x:x+width]

def pull_faces(glob_in: str, path_out: str) -> int:
    """Pulls faces out of images found in glob_in and saves them as path_out
    Returns the total number of faces found
    """
    i = 0
    for fname in glob(glob_in):
        print(fname)
        img = imread(fname)
        bboxes = find_faces(img)
        for bbox in bboxes:
            cropped = crop(img, *bbox)
            imsave(path_out.format(i), cropped)
            i += 1
    return i

found = pull_faces('test_imgs/initial/group*.jpg', 'test_imgs/corpus/face{}.jpg')

print('Total number of base corpus faces found:', found)
plt.imshow(imread('test_imgs/corpus/face0.jpg'))


I am working on the file "FindMe.ipynb".
The first task is:
And the last image should be of the target:
Can you write Python code for it?

print('Is the target:', labels[-1] == 1)
plt.imshow(features[-1], cmap='gray')


I am working on the file "FindMe.ipynb".
The first task is:
DATA/FILE CHECKPOINT
The notebook can be run from scratch from this point onward.
Can you write Python code for it?

# DATA/FILE CHECKPOINT
from pickle import load
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.image import imread, imsave
%matplotlib inline
from findme.imageutil import crop, draw_boxes, preprocess
from findme.models import find_faces

features = load(open('findme/features.pkl', 'rb'))
labels = load(open('findme/labels.pkl', 'rb'))

features = features[-24:]
labels = labels[-24:]


I am working on the file "FindMe.ipynb".
The first task is:
Let's quickly see how well it trained to the given data. Because the dataset is so small, we didn't want to keep any for a test or validation set. We'll test it on a new image later.
Can you write Python code for it?

preds = model.predict(features)
print('Non-target faces predicted correctly:', np.all(preds[:12] == 0))
print('Non-target faces predicted correctly:', preds[-12:] == 1))


I am working on the file "FindMe.ipynb".
The first task is:
Now we need to define our model architecture one layer at a time. We'll create three convolutional layers, two fully-connected layers, and the output layer.
Can you write Python code for it?

from keras.layers import Activation, Convolution2D, Dense, Dropout, Flatten, MaxPooling2D
from keras.metrics import binary_accuracy
from keras.models import Sequential

SHAPE = features[0].shape
NB_FILTER = 16

def make_model() -> Sequential:
    """Create a Sequential Keras model to boolean classify faces"""
    model = Sequential()
    #First Convolution
    model.add(Convolution2D(NB_FILTER, (3, 3), input_shape=SHAPE))
    model.add(Activation('relu'))
    model.add(MaxPooling2D())
    model.add(Dropout(0.1))
    # Second Convolution
    model.add(Convolution2D(NB_FILTER*2, (2, 2)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D())
    model.add(Dropout(0.2))
    # Third Convolution
    model.add(Convolution2D(NB_FILTER*4, (2, 2)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D())
    model.add(Dropout(0.3))
    # Flatten for Fully Connected
    model.add(Flatten())
    # First Fully Connected
    model.add(Dense(1024))
    model.add(Activation('relu'))
    model.add(Dropout(0.4))
    # Second Fully Connected
    model.add(Dense(1024))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    # Output
    model.add(Dense(2))
    model.compile(loss = 'mean_squared_error', optimizer = 'rmsprop', metrics=[binary_accuracy])
    return model

print(make_model().summary())


I am working on the file "FindMe.ipynb".
The first task is:
Now we need to train the model. Even though we have a large model in terms of its parameters, we can still let the model train for many epochs because our feature set is so small. On a MacBook Air, it takes around 30 seconds to train the model with 500 epochs. To save space, I've disabled the full training printout that Keras provides, but you can watch the accuracy progress yourself by changing verbose from 0 to 1.
We also need to shuffle our data because feeding all of the non-target and target faces into the model in order will lead to a biased model. Scikit-Learn has a convenient function to do this for us. Rather than just calling random, this function preserves the relationship between the feature and label indexes.
Can you write Python code for it?

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.utils import shuffle

model = KerasClassifier(build_fn=make_model, epochs=500, batch_size=len(labels), verbose=0)
X, Y = shuffle(features, labels, random_state=42)
model.fit(X, Y)


I am working on the file "FindMe.ipynb".
The first task is:
That was easy enough. In order to have a large enough corpus of target faces, I included pictures of myself with other people and deleted their faces after the code block ran. It ended up having eleven target faces.
Model Training Data
Now that we have our faces, we need to create the features and labels that will be used to train our facial recognition model. We've already classified our data based on the face's filename; all we need to do is assign a 1 or 0 to each group for our labels. We'll also need to scale each image to a standard size. Thankfully the output for each bounding box is a square, so we don't have to worry about introducing distortions.
Can you write Python code for it?

#Load the two sets of images
from pickle import load

notme = load(open('findme/base_corpus.pkl', 'rb'))
me = [imread(fname) for fname in glob('test_imgs/corpus/me*.jpg')]

#Create features and labels
features = notme + me
labels = [0] * len(notme) + [1] * len(me)

#Preprocess images for the model
def preprocess(img: np.ndarray) -> np.ndarray:
    """Resizes a given image and remove alpha channel"""
    img = cv2.resize(img, (45, 45), interpolation=cv2.INTER_AREA)[:,:,:3]
    return img

features = [preprocess(face) for face in features]


I am working on the file "FindMe.ipynb".
The first task is:
That's it for our data. You'll notice that we only loaded a subset of our dataset. This ensures that the number of target and non-target images matches, which leads to a better model even though it has less data overall. We'll split our data in the next section.
Am I in This?
We've already created all of our data. Now for the model we're going to train. First, we need to convert our labels to one-hot encoding for use in the model. This means our output layer will have two nodes: True and False.
Can you write Python code for it?

from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder()
labels = enc.fit_transform(labels.reshape(-1, 1)).toarray()
print('Not target label:', labels[0])
print('Is target label:', labels[-1])


I am working on the file "FindMe.ipynb".
The first task is:
That's it. While Keras has its own mechanisms for training and validating models, we're using a wrapper around our Keras model so it conforms to the Scikit-Learn model API. We can use fit and predict when working with the model in our code, and it let's us train and use our model with the other helper modules sk-learn provides. For example, we could have evaluated the model using StratifiedKFold and cross_val_score which would look like this:
```python
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
model = KerasClassifier(build_fn=make_model, epochs=5, batch_size=len(labels), verbose=0)
evaluate using 10-fold cross validation
kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
result = cross_val_score(model, features, labels, cv=kfold)
print(result.mean())
```
This method allows us to determine how effective our model is but does not return a trained model for us to use.
Putting It Together
Lastly, let's create a single function that takes in an image and returns if the target was found and where.
First we'll load in our test image. Keep in mind that the model we just trained has never seen this image before and it contains multiple people (and a manatee statue).
Can you write Python code for it?

test_img = imread('test_imgs/evaluate/me1.jpg')
plt.imshow(test_img)


I am working on the file "FindMe.ipynb".
The first task is:
Yeah. That's it. Let's break down the steps:

find_faces returns a list of bounding boxes containing faces
We prepare each face by cropping the image to the bounding box, scaling to 45x45, and removing the alpha channel
The model predicts whether the face is or is not the target
If the target is found (pred == 1), return True and the current bounding box
If there aren't any faces or none of the faces belongs to the target, return False and None

Now let's test it. If it works properly, we should see a bounding bx appear around the target's face.
Can you write Python code for it?

found, bbox = target_in_img(test_img)

print('Target face found in test image:', found)
if found:
    plt.imshow(draw_boxes([bbox], test_img, line_width=20))


I am working on the file "FindSwirlLocs-AstroScriptV1.ipynb".
The first task is:
Now extract objects
Can you write Python code for it?

#creat filter kernal
kern = np.array([[1,2,1], [2,4,2], [1,2,1]])      #Basic default kernal

kern = np.array([[1,2,4,2,1],[2,3,5,3,2],[3,6,8,6,3],[2,3,5,3,2],[1,2,4,2,1]])      #Basic default kernal

from scipy.stats import multivariate_normal as mvnorm

x = np.linspace(-5, 5, 100)
y = mvnorm.pdf(x, mean=0, cov=1)
#plt.plot(x,y)
#mvnorm.pdf(

x = np.mgrid[-1:1:.01]
y = x;
r = (x**2+y**2)**0.5
kern = np.empty(x.shape)
#for i in kern.shape[0]
#    kern[i,:] = mvnorm.pdf(r[i,:], mean=0, cov=1)

#plt.imshow(kern)

#y = mvnorm.pdf(x, mean=0, cov=1)
#pos = np.empty(x.shape + (2,))l
#pos[:, :, 0] = x; pos[:, :, 1] = y

x = np.mgrid[-10:10:1]
x.shape

objects = sep.extract(np.ascontiguousarray(Swirl[:,:,1]), 1.5, err=bkg.globalrms,filter_kernel=kern)


I am working on the file "FindSwirlLocs-AstroScriptV1.ipynb".
The first task is:
np.ascontiguousarray(Swirl[:,:,1]).flags how to make array C contiguous
Can you write Python code for it?

len(objects)

from matplotlib.patches import Ellipse

#fig, ax = plt.subplots()
plt.figure(figsize = [8,3])
plt.pcolor(X,Y,Swirl[:,:,1], cmap='RdBu_r');
ax = plt.gca()
plt.clim([-50, 50])
plt.axis('scaled')
plt.xlim([X.min(), X.max()])
plt.ylim([Y.min(), Y.max()])
plt.colorbar()

scale = (X[1,-1]-X[1,1])/uSize[1]
#plt.plot(objects['x']*scale,objects['y']*scale,'go')

for i in range(len(objects)):
    e = Ellipse(xy=(objects['x'][i]*scale, objects['y'][i]*scale),
                width=6*objects['a'][i]*scale,
                height=6*objects['b'][i]*scale,
                angle=objects['theta'][i] * 180. / np.pi)
    e.set_facecolor('none')
    e.set_edgecolor('red')
    ax.add_artist(e)

#objects['x']
scale = (X[1,-1]-X[1,1])/uSize[1]
objects['x']*scale

X[objects['x'],objects['y']]

objects['x']


I am working on the file "FindSwirlLocs.ipynb".
The first task is:
Create thresholded field
Can you write Python code for it?

import copy

SwirlFilt = copy.copy(Swirl)    #think this should completely copy the list, allowing me to try things

#Swirl must be above a certain background value or it is zeroed
SwirlFilt[np.absolute(SwirlFilt)<7] = 0

#Then only keep those locations where swirls is greater than Thresh*SwirlStd
Thresh = 60
SwirlFilt[np.absolute(SwirlNorm)<Thresh] = 0


I am working on the file "FindSwirlLocs.ipynb".
The first task is:
Find all blobs that agree with the thresholds
Can you write Python code for it?

PIVutils = reload(PIVutils)
[num_features,features_per_frame, labeled_array, cent] = PIVutils.findBlobs(SwirlFilt)

[f, ax] = PIVutils.plotScalarField(X,Y,SwirlFilt[:,:,1],10)

for i in range(features_per_frame[1]):
    plt.plot(cent[1][i][1]*scale+X[1,1],cent[1][i][0]*scale+Y[1,1],'oy',markersize=4,markeredgecolor=None)


I am working on the file "FindSwirlLocs.ipynb".
The first task is:
Plot after normalization
Can you write Python code for it?

PIVutils.plotScalarField(X,Y,Swirl[:,:,1],30)

Swirl[NanLocs] = 0   #Get rid of nans for now


I am working on the file "Finding Lane Lines/notebook/Fiding_Lane_Lines.ipynb".
The first task is:
Here is the youtube link of the video.
Optional Challenge
Can you write Python code for it?

challenge_output = '../test_videos_output/challenge.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
##clip3 = VideoFileClip('test_videos/challenge.mp4').subclip(0,5)
clip3 = VideoFileClip('../test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(process_image)
%time challenge_clip.write_videofile(challenge_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))


I am working on the file "Finding Lane Lines/notebook/Fiding_Lane_Lines.ipynb".
The first task is:
Test on Videos
We can test our solution on two provided videos:
solidWhiteRight.mp4
solidYellowLeft.mp4
Let's try the one with the solid white lane on the right first ...
Can you write Python code for it?

def process_image(image):
    # NOTE: The output you return should be a color image (3 channel) for processing video below
    # you should return the final output (image where lines are drawn on lanes)
    img = grayscale(image)
    # further smoothing to blur image for better result
    img = gaussian_blur(img, kernel_size = 3)
    img = canny(img, low_threshold = 80, high_threshold = 240)

    # This time we are defining a four sided polygon to mask
    imshape = image.shape
    vertices = np.array([[(0,imshape[0]),(460, 320), (500, 320), (imshape[1],imshape[0])]], dtype=np.int32)
    img = region_of_interest(img, vertices)

    # Hough transform
    line_image = hough_lines(img, rho = 2, theta= np.pi/180, threshold = 50, min_line_len = 40, max_line_gap = 20)

    # Draw the lines on the edge image
    final = weighted_img(line_image, image, α = 0.8, β = 1., λ = 0.)
    return final

white_line_output = '../test_videos_output/solidWhiteRight.mp4'
clip1 = VideoFileClip('../test_videos/solidWhiteRight.mp4')
white_line_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_line_clip.write_videofile(white_line_output, audio=False)


I am working on the file "Finding Lane Lines/notebook/Fiding_Lane_Lines.ipynb".
The first task is:
Testing with 1 sampe image
Can you write Python code for it?

#reading in an image
image = mpimg.imread('../test_images/solidWhiteRight.jpg')

img = grayscale(image)
# further smoothing to blur image for better result
img = gaussian_blur(img, kernel_size = 3)
img = canny(img, low_threshold = 80, high_threshold = 240)

# This time we are defining a four sided polygon to mask
imshape = image.shape
vertices = np.array([[(0,imshape[0]),(460, 320), (500, 320), (imshape[1],imshape[0])]], dtype=np.int32)
img = region_of_interest(img, vertices)

# Hough transform
line_image = hough_lines(img, rho = 1, theta= np.pi/180, threshold = 50, min_line_len = 40, max_line_gap = 20)

# Draw the lines on the edge image
img = weighted_img(line_image, image , α=0.8, β=1., λ=0.)
plt.imsave("../test_images_output/solidWhiteRightOutput.jpg", img)
plt.imshow(img)
x = [vertices[0][0][0], vertices[0][1][0], vertices[0][2][0], vertices[0][3][0]]
y = [vertices[0][0][1], vertices[0][1][1], vertices[0][2][1], vertices[0][3][1]]
plt.plot(x, y, 'b--', lw=4)


I am working on the file "Finding a structure in the data/Week 2/PCA.ipynb".
The first task is:
А теперь сравним эти направления с направлениями, которые выбирает метод главных компонент:
Can you write Python code for it?

def plot_principal_components(data, model, scatter=True, legend=True):
    W_pca = model.components_
    if scatter:
        plt.scatter(data[:,0], data[:,1])
    plt.plot(data[:,0], -(W_pca[0,0]/W_pca[0,1])*data[:,0], color="c")
    plt.plot(data[:,0], -(W_pca[1,0]/W_pca[1,1])*data[:,0], color="c")
    if legend:
        c_patch = mpatches.Patch(color='c', label='Principal components')
        plt.legend(handles=[c_patch], loc='lower right')
    # сделаем графики красивыми:
    plt.axis('equal')
    limits = [np.minimum(np.amin(data[:,0]), np.amin(data[:,1]))-0.5,
              np.maximum(np.amax(data[:,0]), np.amax(data[:,1]))+0.5]
    plt.xlim(limits[0],limits[1])
    plt.ylim(limits[0],limits[1])
    plt.draw()

model = PCA(n_components=2)
model.fit(data)

plt.scatter(data[:,0], data[:,1])
# построим истинные компоненты, вдоль которых максимальна дисперсия данных
plt.plot(data[:,0], (W_true[0,0]/W_true[0,1])*data[:,0], color="g")
plt.plot(data[:,0], (W_true[1,0]/W_true[1,1])*data[:,0], color="g")
# построим компоненты, полученные с использованием метода PCA:
plot_principal_components(data, model, scatter=False, legend=False)
c_patch = mpatches.Patch(color='c', label='Principal components')
plt.legend(handles=[g_patch, c_patch])
plt.draw()


I am working on the file "Finding a structure in the data/Week 2/PCA.ipynb".
The first task is:
В этом случае главные компоненты значительно точнее приближают истинные направления данных, вдоль которых наблюдается наибольшая дисперсия.
Статистический взгляд на модель
Как формализовать предположения метода, указанные выше? При помощи вероятностной модели!
Задача, стоящая за любым методом уменьшения размерности: получить из набора зашумлённых признаков $X$ истинные значения $Y$, которые на самом деле определяют набор данных (т.е. сведение датасета с большим количеством признаков к данным, имеющим т.н. "эффективную размерность").
В случае метода главных компонент мы хотим найти направления, вдоль которых максимальна дисперсия, с учётом описанных выше предположений о структуре данных и компонент.
Материал, описанный ниже в данной секции, не обязателен для ознакомления для выполнения следующего задания, т.к. требует некоторых знаний статистики.
Для тех, кто собирается его пропустить: в конце раздела мы получим метрику качества, которая должна определять, насколько данные хорошо описываются построенной моделью при заданном числе компонент. Отбор признаков при этом сводится к тому, что мы выбираем то количество компонент, при котором используемая метрика (логарифм правдоподобия) является максимальной.
С учётом предположений задача метода главных компонент выглядит следующим образом:
$$ x = Wy + \mu + \epsilon$$
где:
- $x$ -- наблюдаемые данные
- $W$ -- матрица главных компонент (каждый стобец -- одна компонента)
- $y$ -- их проекция на главные компоненты
- $\mu$ -- среднее наблюдаемых данных
- $\epsilon \sim \mathcal{N}(0, \sigma^2I)$ -- нормальный шум
Исходя из распределения шума, выпишем распределение на $x$:
$$p(x \mid y) = \mathcal{N}(Wx + \mu, \sigma^2I) $$
Введём априорное распределение на $y$:
$$p(y) = \mathcal{N}(0, 1)$$
Выведем из этого при помощи формулы Байеса маргинальное распределение на $p(x)$:
$$p(x) = \mathcal{N}(\mu, \sigma^2I + WW^T)$$
Тогда правдоподобие набора данных при условии используемой модели выглядит следующим образом:
$$\mathcal{L} = \sum_{i=1}^N \log p(x_i) = -N/2 \Big( d\log(2\pi)...
Can you write Python code for it?

from sklearn.decomposition import PCA
from sklearn.cross_validation import cross_val_score as cv_score

def plot_scores(d_scores):
    n_components = np.arange(1,d_scores.size+1)
    plt.plot(n_components, d_scores, 'b', label='PCA scores')
    plt.xlim(n_components[0], n_components[-1])
    plt.xlabel('n components')
    plt.ylabel('cv scores')
    plt.legend(loc='lower right')
    plt.show()

def write_answer_1(optimal_d):
    with open("pca_answer1.txt", "w") as fout:
        fout.write(str(optimal_d))

data = pd.read_csv('data_task1.csv')

# place your code here
pass

data.head()

data.isnull().any().any()

d_scores = []

for d in xrange(1,len(data.columns)+1):
    model = PCA(n_components = d,svd_solver='full')
    scores = cv_score(model, data)
    d_scores = np.append(d_scores, scores.mean())

plot_scores(d_scores)

ans1 = np.argmax(d_scores,axis = 0) + 1 # отсчет с 0
print ans1
write_answer_1(ans1)


I am working on the file "Finding a structure in the data/Week 2/PCA.ipynb".
The first task is:
Вариационный взгляд на модель
Мы знаем, что каждой главной компоненте соответствует описываемая ей дисперсия данных (дисперсия данных при проекции на эту компоненту). Она численно равна значению диагональных элементов матрицы $\Lambda$, получаемой из спектрального разложения матрицы ковариации данных (смотри теорию выше).
Исходя из этого, мы можем отсортировать дисперсию данных вдоль этих компонент по убыванию, и уменьшить размерность данных, отбросив $q$ итоговых главных компонент, имеющих наименьшую дисперсию.
Делать это можно двумя разными способами. Например, если вы вдальнейшем обучаете на данных с уменьшенной размерностью модель классификации или регрессии, то можно запустить итерационный процесс: удалять компоненты с наименьшей дисперсией по одной, пока качество итоговой модели не станет значительно хуже.
Более общий способ отбора признаков заключается в том, что вы можете посмотреть на разности в дисперсиях в отсортированном ряде $\lambda_{(1)} > \lambda_{(2)} > \dots > \lambda_{(D)}$: $\lambda_{(1)}-\lambda_{(2)}, \dots, \lambda_{(D-1)} - \lambda_{(D)}$, и удалить те компоненты, на которых разность будет наибольшей. Именно этим методом вам и предлагается воспользоваться для тестового набора данных.
Задание 2. Ручное уменьшение размерности признаков посредством анализа дисперсии данных вдоль главных компонент
Рассмотрим ещё один набор данных размерности $D$, чья реальная размерность значительно меньше наблюдаемой (назовём её также $d$). От вас требуется:

Построить модель PCA с $D$ главными компонентами по этим данным.
Спроецировать данные на главные компоненты.
Оценить их дисперсию вдоль главных компонент.
Отсортировать дисперсии в порядке убывания и получить их попарные разности: $\lambda_{(i-1)} - \lambda_{(i)}$.
Найти разность с наибольшим значением и получить по ней оценку на эффективную размерность данных $\hat{d}$.
Построить график дисперсий и убедиться, что полученная оценка на $\hat{d}{opt}$ действительно имеет смысл, после этого внести полученное значение $\hat{d}{opt}$ в файл ответа.

Для...
Can you write Python code for it?

from sklearn.decomposition import PCA
from sklearn.cross_validation import cross_val_score as cv_score

def plot_variances(d_variances):
    n_components = np.arange(1,d_variances.size+1)
    plt.plot(n_components, d_variances, 'b', label='Component variances')
    plt.xlim(n_components[0], n_components[-1])
    plt.xlabel('n components')
    plt.ylabel('variance')
    plt.legend(loc='upper right')
    plt.show()

def write_answer_2(optimal_d):
    with open("pca_answer2.txt", "w") as fout:
        fout.write(str(optimal_d))

data = pd.read_csv('data_task2.csv')

# place your code here
pass

model = PCA(n_components=len(data.columns))   #1
model.fit(data)    #2
#  Доля информации (доля от общей дисперсии)
d_variances = model.explained_variance_ratio_    #3
plot_variances(d_variances)

sort_d_variances = np.sort(d_variances)   #4
difference_d_variances = []
#5
for i in xrange(1,len(sort_d_variances)):
    diff = sort_d_variances[i] - sort_d_variances[i-1]
    difference_d_variances = np.append(difference_d_variances,diff)
# 6
plot_variances(difference_d_variances)

# -1, для задаия одинаковой размерности
ans2 = len(data.columns) - 1 - np.argmax(difference_d_variances)
print ans2
write_answer_2(ans2)


I am working on the file "Finding a structure in the data/Week 2/PCA.ipynb".
The first task is:
Видно, что уже при небольшом количестве данных они отличаются незначительно. Увеличим размер выборки:
Can you write Python code for it?

data_large = np.random.multivariate_normal(mu, C, size=5000)

model = PCA(n_components=2)
model.fit(data_large)
plt.scatter(data_large[:,0], data_large[:,1], alpha=0.1)
# построим истинные компоненты, вдоль которых максимальна дисперсия данных
plt.plot(data_large[:,0], (W_true[0,0]/W_true[0,1])*data_large[:,0], color="g")
plt.plot(data_large[:,0], (W_true[1,0]/W_true[1,1])*data_large[:,0], color="g")
# построим компоненты, полученные с использованием метода PCA:
plot_principal_components(data_large, model, scatter=False, legend=False)
c_patch = mpatches.Patch(color='c', label='Principal components')
plt.legend(handles=[g_patch, c_patch])
plt.draw()


I am working on the file "Finding a structure in the data/Week 2/PCA.ipynb".
The first task is:
Путём диагонализации истинной матрицы ковариаций $C$, мы можем найти преобразование исходного набора данных, компоненты которого наилучшим образом будут описывать дисперсию, с учётом их ортогональности друг другу:
Can you write Python code for it?

v, W_true = np.linalg.eig(C)
print W_true
plt.scatter(data[:,0], data[:,1])
# построим истинные компоненты, вдоль которых максимальна дисперсия данных
plt.plot(data[:,0], (W_true[0,0]/W_true[0,1])*data[:,0], color="g")
plt.plot(data[:,0], (W_true[1,0]/W_true[1,1])*data[:,0], color="g")
g_patch = mpatches.Patch(color='g', label='True components')
plt.legend(handles=[g_patch])
plt.axis('equal')
limits = [np.minimum(np.amin(data[:,0]), np.amin(data[:,1])),
          np.maximum(np.amax(data[:,0]), np.amax(data[:,1]))]
plt.xlim(limits[0],limits[1])
plt.ylim(limits[0],limits[1])
plt.draw()


I am working on the file "Finding a structure in the data/Week 2/PCA.ipynb".
The first task is:
Теория
Любой набор данных представляет собой матрицу $X$.
Метод главных компонент последовательно находит следующие линейные комбинации признаков (компоненты) из $X$:
- каждая компонента ортогональна всем остальным и нормированна: $<w_i, w_j> = 0, \quad ||w_i||=1$,
- каждая компонента описывает максимально возможную дисперсию данных (с учётом предыдущего ограничения).
Предположения, в рамках которых данный подход будет работать хорошо:
- линейность компонент: мы предполагаем, что данные можно анализировать линейными методами,
- большие дисперсии важны: предполагается, что наиболее важны те направления в данных, вдоль которых они имеют наибольшую дисперсию,
- все компоненты ортогональны: это предположение позволяет проводить анализ главных компонент при помощи техник линейной алгебры (например, сингулярное разложение матрицы $X$ или спектральное разложение матрицы $X^TX$).
Как это выглядит математически?
Обозначим следующим образом выборочную матрицу ковариации данных: $\hat{C} \propto Q = X^TX$. ($Q$ отличается от $\hat{C}$ нормировкой на число объектов).
Сингулярное разложение матрицы $Q$ выглядит следующим образом:
$$Q = X^TX = W \Lambda W^T$$
Можно строго показать, что столбцы матрицы $W$ являются главными компонентами матрицы $X$, т.е. комбинациями признаков, удовлетворяющих двум условиям, указанным в начале. При этом дисперсия данных вдоль направления, заданного каждой компонентой, равна соответствующему значению диагональной матрицы $\Lambda$.
Как же на основании этого преобразования производить уменьшение размерности? Мы можем отранжировать компоненты, используя значения дисперсий данных вдоль них.
Сделаем это: $\lambda_{(1)} > \lambda_{(2)} > \dots > \lambda_{(D)}$.
Тогда, если мы выберем компоненты, соответствующие первым $d$ дисперсиям из этого списка, мы получим набор из $d$ новых признаков, которые наилучшим образом описывают дисперсию изначального набора данных среди всех других возможных линейных комбинаций исходных признаков матрицы $X$.
- Если $d=D$, то мы вообще не теряем никакой...
Can you write Python code for it?

from sklearn.decomposition import PCA

mu = np.zeros(2)
C = np.array([[3,1],[1,2]])

data = np.random.multivariate_normal(mu, C, size=50)
plt.scatter(data[:,0], data[:,1])
plt.show()


I am working on the file "Finding a structure in the data/Week 4/edit_CookingLDA_PA.ipynb".
The first task is:
Для такого большого количества классов это неплохая точность. Вы можете попроовать обучать RandomForest на исходной матрице частот слов, имеющей значительно большую размерность, и увидеть, что accuracy увеличивается на 10–15%. Таким образом, LDA собрал не всю, но достаточно большую часть информации из выборки, в матрице низкого ранга.
LDA — вероятностная модель
Матричное разложение, использующееся в LDA, интерпретируется как следующий процесс генерации документов.
Для документа $d$ длины $n_d$:
1. Из априорного распределения Дирихле с параметром alpha сгенерировать распределение над множеством тем: $\theta_d \sim Dirichlet(\alpha)$
1. Для каждого слова $w = 1, \dots, n_d$:
    1. Сгенерировать тему из дискретного распределения $t \sim \theta_{d}$
    1. Сгенерировать слово из дискретного распределения $w \sim \phi_{t}$.
Подробнее об этом в Википедии.
В контексте нашей задачи получается, что, используя данный генеративный процесс, можно создавать новые рецепты. Вы можете передать в функцию модель и число ингредиентов и сгенерировать рецепт :)
Can you write Python code for it?

def generate_recipe(model, num_ingredients):
    theta = np.random.dirichlet(model.alpha)
    for i in range(num_ingredients):
        t = np.random.choice(np.arange(model.num_topics), p=theta)
        topic = model.show_topic(t, topn=model.num_terms)
        topic_distr = [x[1] for x in topic]
        terms = [x[0] for x in topic]
        w = np.random.choice(terms, p=topic_distr)
        print w

generate_recipe(model, 5)

generate_recipe(model2, 5)

generate_recipe(model3, 5)


I am working on the file "Finding a structure in the data/Week 4/edit_CookingLDA_PA.ipynb".
The first task is:
Наша коллекция небольшая, и целиком помещается в оперативную память. Gensim может работать с такими данными и не требует их сохранения на диск в специальном формате. Для этого коллекция должна быть представлена в виде списка списков, каждый внутренний список соответствует отдельному документу и состоит из его слов. Пример коллекции из двух документов:
[["hello", "world"], ["programming", "in", "python"]]
Преобразуем наши данные в такой формат, а затем создадим объекты corpus и dictionary, с которыми будет работать модель.
Can you write Python code for it?

texts = [recipe["ingredients"] for recipe in recipes]
dictionary = corpora.Dictionary(texts)   # составляем словарь
corpus = [dictionary.doc2bow(text) for text in texts]  # составляем корпус документов

print texts[0]
print corpus[0]


I am working on the file "Finding a structure in the data/Week 4/edit_CookingLDA_PA.ipynb".
The first task is:
Таким образом, гиперпараметр alpha влияет на разреженность распределений тем в документах. Аналогично гиперпараметр eta влияет на разреженность распределений слов в темах.
LDA как способ понижения размерности
Иногда, распределения над темами, найденные с помощью LDA, добавляют в матрицу объекты-признаки как дополнительные, семантические, признаки, и это может улучшить качество решения задачи. Для простоты давайте просто обучим классификатор рецептов на кухни на признаках, полученных из LDA, и измерим точность (accuracy).
Задание 5. Используйте модель, построенную по сокращенной выборке с alpha по умолчанию (вторую модель). Составьте матрицу $\Theta = p(t|d)$ вероятностей тем в документах; вы можете использовать тот же метод get_document_topics, а также вектор правильных ответов y (в том же порядке, в котором рецепты идут в переменной recipes). Создайте объект RandomForestClassifier со 100 деревьями, с помощью функции cross_val_score вычислите среднюю accuracy по трем фолдам (перемешивать данные не нужно) и передайте в функцию save_answers5.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import cross_val_score

X = np.zeros((len(recipes),40))
y = []
for r in recipes:
    y.append(r['cuisine'])

for i in range(len(recipes)):
    top = model2.get_document_topics(corpus2[i])
    for t in top:
        X[i,t[0]] = t[1]

RFC = RandomForestClassifier(n_estimators = 100)
estimator = cross_val_score(RFC,X,y,cv = 3).mean()
print estimator

def save_answers5(accuracy):
     with open("cooking_LDA_pa_task5.txt", "w") as fout:
        fout.write(str(accuracy))

save_answers5(estimator)


I am working on the file "Finding a structure in the data/Week 4/edit_CookingLDA_PA.ipynb".
The first task is:
У вас должно получиться, что документ характеризуется небольшим числом тем. Попробуем поменять гиперпараметр alpha, задающий априорное распределение Дирихле для распределений тем в документах.
Задание 4. Обучите третью модель: используйте сокращенный корпус (corpus2 и dictionary2) и установите параметр alpha=1, passes=5. Не забудьте про фиксацию seed! Выведите темы новой модели для нулевого документа; должно получиться, что распределение над множеством тем практически равномерное. Чтобы убедиться в том, что во второй модели документы описываются гораздо более разреженными распределениями, чем в третьей, посчитайте суммарное количество элементов, превосходящих 0.01, в матрицах темы-документы обеих моделей. Другими словами, запросите темы  модели для каждого документа с параметром minimum_probability=0.01 и просуммируйте число элементов в получаемых массивах. Передайте две суммы (сначала для модели с alpha по умолчанию, затем для модели в alpha=1) в функцию save_answers4.
Can you write Python code for it?

np.random.seed(76543)
# здесь код для построения модели:
%time model3 = models.ldamodel.LdaModel(corpus2, id2word=dictionary2, num_topics = 40, passes = 5,alpha = 1)

print model3.get_document_topics(corpus2[0])

def get_doc_topics_summa(model,corpus):
    summa = 0
    for i in corpus:
        top = model.get_document_topics(i, minimum_probability=0.01)
        summa += len(top)
    return summa

count_model2 = get_doc_topics_summa(model2,corpus2)
count_model3 = get_doc_topics_summa(model3,corpus2)

print count_model2, count_model3

def save_answers4(count_model2, count_model3):
    with open("cooking_LDA_pa_task4.txt", "w") as fout:
        fout.write(" ".join([str(el) for el in [count_model2, count_model3]]))

save_answers4(count_model2, count_model3)


I am working on the file "Finding a structure in the data/Week 4/edit_CookingLDA_PA.ipynb".
The first task is:
У объекта dictionary есть полезная переменная dictionary.token2id, позволяющая находить соответствие между ингредиентами и их индексами.
Обучение модели
Вам может понадобиться документация LDA в gensim.
Задание 1. Обучите модель LDA с 40 темами, установив количество проходов по коллекции 5 и оставив остальные параметры по умолчанию.
Затем вызовите метод модели show_topics, указав количество тем 40 и количество токенов 10, и сохраните результат (топы ингредиентов в темах) в отдельную переменную. Если при вызове метода show_topics указать параметр formatted=True, то топы ингредиентов будет удобно выводить на печать, если formatted=False, будет удобно работать со списком программно. Выведите топы на печать, рассмотрите темы, а затем ответьте на вопрос:
Сколько раз ингредиенты "salt", "sugar", "water", "mushrooms", "chicken", "eggs" встретились среди топов-10 всех 40 тем? При ответе не нужно учитывать составные ингредиенты, например, "hot water".
Передайте 6 чисел в функцию save_answers1 и загрузите сгенерированный файл в форму.
У gensim нет возможности фиксировать случайное приближение через параметры метода, но библиотека использует numpy для инициализации матриц. Поэтому, по утверждению автора библиотеки, фиксировать случайное приближение нужно командой, которая написана в следующей ячейке. Перед строкой кода с построением модели обязательно вставляйте указанную строку фиксации random.seed.
Can you write Python code for it?

np.random.seed(76543)
# здесь код для построения модели:
%time model = models.ldamodel.LdaModel(corpus, id2word=dictionary, num_topics = 40, passes = 5)

topics = model.show_topics(num_topics = 40,num_words = 10, formatted = True)
print topics

topics = model.show_topics(num_topics = 40,num_words = 10, formatted = False)

c_salt = 0
c_sugar = 0
c_water = 0
c_mushrooms = 0
c_chicken = 0
c_eggs = 0

for i in range(40):
    check = topics[i][1]
    for j in check:
        word = j[0]
        if word == "salt":
            c_salt += 1
        elif word == "sugar":
            c_sugar += 1
        elif word == "water":
            c_water += 1
        elif word == "mushrooms":
            c_mushrooms += 1
        elif word == "chicken":
            c_chicken += 1
        elif word == "eggs":
            c_eggs += 1

print c_salt, c_sugar, c_water, c_mushrooms, c_chicken, c_eggs

def save_answers1(c_salt, c_sugar, c_water, c_mushrooms, c_chicken, c_eggs):
    with open("cooking_LDA_pa_task1.txt", "w") as fout:
        fout.write(" ".join([str(el) for el in [c_salt, c_sugar, c_water, c_mushrooms, c_chicken, c_eggs]]))

save_answers1(c_salt, c_sugar, c_water, c_mushrooms, c_chicken, c_eggs)


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
2. Deck, Substructure, and Supertructure

Finding mean time interval before any intervention, Intervention can be described as Repair, Reconstruction, or Rehabilitation.

Construction of dictionary consisting of 'structureNumber' as key and a list of condition rating of that particular bridge component as value. This construction of dictionary is carried for all the bridge components such as Deck, Substructure and Superstructure.
Can you write Python code for it?

deckCondition = {k: g["deck"].tolist() for k,g in conditionRatings.groupby("structureNumber")}
substructureCondition = {k: g["substructure"].tolist() for k,g in conditionRatings.groupby("structureNumber")}
superstructureCondition = {k: g["superstructure"].tolist() for k,g in conditionRatings.groupby("structureNumber")}


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
As opposed to the hypothesis the mean time for second interval is greater than first time interval for deck. but not all intervals start from rating 9 and we might not consider that some of the bridges might have be reconstructed before 1992
Calculation of first condition rating of the first time interval and second time intervals.
Can you write Python code for it?

avgStartOfFirstInterval = []
avgStartOfSecondInterval = []
for intervals in deckIntervalGt2:
    avgStartOfFirstInterval.append(intervals[0][0])
    avgStartOfSecondInterval.append(intervals[1][0])
print("Mean of the first condition rating of first time intervals :", np.mean(avgStartOfFirstInterval))
print("Mean of the first condition rating of second time intervals :", np.mean(avgStartOfSecondInterval))


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Calculating the mean time interval for first and second time interval for Deck
Can you write Python code for it?

deckIntervals = []
deckIntervalSize = []
for i in deckCondition.values():
    lst, size = findAllIntervals(i)
    deckIntervalSize.append(size)
    deckIntervals.append(lst)

#
from collections import Counter
keys =list(Counter(deckIntervalSize).keys())
values =list(Counter(deckIntervalSize).values())

plt.figure(figsize=(10,8))
plt.bar(keys,values)
plt.xlabel("No. of Intervals")
plt.ylabel("No. of Records")
plt.title("No. of Records vs No. of Intervals")
plt.show()

## Filter records with only two or more interval
deckIntervalGt2 = []
for index in range(0,len(deckIntervalSize)-1,1):
    if deckIntervalSize[index] > 1:
        deckIntervalGt2.append(deckIntervals[index])

first_interval = []
second_interval = []
for intervals in deckIntervalGt2:
    first_interval.append(len(intervals[0]))
    second_interval.append(len(intervals[1]))
print("Mean of the first time intervals :", np.mean(first_interval))
print("Mean of the second time intervals :", np.mean(second_interval))


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Calculating the mean time interval for first and second time interval for Substructure
Can you write Python code for it?

substructureIntervals = []
substructureIntervalSize = []
for i in substructureCondition.values():
    lst, size = findAllIntervals(i)
    substructureIntervalSize.append(size)
    substructureIntervals.append(lst)

#
from collections import Counter
keys_substructure =list(Counter(substructureIntervalSize).keys())
values_substructure =list(Counter(substructureIntervalSize).values())

plt.figure(figsize=(10,8))
plt.bar(keys_substructure,values_substructure)
plt.xlabel("No. of Intervals")
plt.ylabel("No. of Records")
plt.xticks([1,2,3,4,5])
plt.title("No. of Records vs No. of Intervals")
plt.show()

## Filter records with only two or more interval
substructureIntervalGt2 = []
for index in range(0,len(substructureIntervalSize)-1,1):
    if substructureIntervalSize[index] > 1:
        substructureIntervalGt2.append(substructureIntervals[index])

substructureFirstInterval = []
substructureSecondInterval = []
for intervals in substructureIntervalGt2:
    substructureFirstInterval.append(len(intervals[0]))
    substructureSecondInterval.append(len(intervals[1]))
print("Mean of the first time intervals :", np.mean(substructureFirstInterval))
print("Mean of the second time intervals :", np.mean(substructureSecondInterval))

avgStartOfFirstInterval_sb = []
avgStartOfSecondInterval_sb = []
for intervals in substructureIntervalGt2:
    avgStartOfFirstInterval_sb.append(intervals[0][0])
    avgStartOfSecondInterval_sb.append(intervals[1][0])
print("Mean of the first condition rating of first time intervals :", np.mean(avgStartOfFirstInterval_sb))
print("Mean of the first condition rating of second time intervals :", np.mean(avgStartOfSecondInterval_sb))


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Calculating the mean time interval for first and second time interval for Superstructure
Can you write Python code for it?

superstructureIntervals = []
superstructureIntervalSize = []
for i in superstructureCondition.values():
    lst, size = findAllIntervals(i)
    superstructureIntervalSize.append(size)
    superstructureIntervals.append(lst)

#
from collections import Counter
keys_superstructure =list(Counter(superstructureIntervalSize).keys())
values_superstructure =list(Counter(superstructureIntervalSize).values())

plt.figure(figsize=(10,8))
plt.bar(keys_superstructure,values_superstructure)
plt.xlabel("No. of Intervals")
plt.ylabel("No. of Records")
plt.xticks([1,2,3,4,5])
plt.title("No. of Records vs No. of Intervals")
plt.show()

## Filter records with only two or more interval
superstructureIntervalGt2 = []
for index in range(0,len(superstructureIntervalSize)-1,1):
    if superstructureIntervalSize[index] > 1:
        superstructureIntervalGt2.append(superstructureIntervals[index])

superstructureFirstInterval = []
superstructureSecondInterval = []
for intervals in superstructureIntervalGt2:
    superstructureFirstInterval.append(len(intervals[0]))
    superstructureSecondInterval.append(len(intervals[1]))
print("Mean of the first time intervals :", np.mean(superstructureFirstInterval))
print("Mean of the second time intervals :", np.mean(superstructureSecondInterval))


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Consideration:
1. We might consider bridges built only after 1992 to get an unbaised mean years of first time interval.
Can you write Python code for it?

CR1992 = conditionRatings.loc[conditionRatings['yearBuilt'] >=1992]
CR1992.head()


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Data Exploration
The following attributes are explored in futher detail:
1. Year reconstructed
2. Deck condition, Substructure condition and Superstructure condition
1. Year reconstructed
List of years in which bridges went under reconstruction
Note: The list contains 0 and -1 , these values indicate that year of reconstruction was either recorded incorrectly or was absent
Can you write Python code for it?


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Data filterxation

Deck, Substructure, Superstructure - survery record with 'N', 'NA' are removed
Can you write Python code for it?

before = len(conditionRatings)
print("Total Records before filteration: ",len(conditionRatings))
conditionRatings = conditionRatings.loc[~conditionRatings['deck'].isin(['N','NA'])]
conditionRatings = conditionRatings.loc[~conditionRatings['substructure'].isin(['N','NA'])]
conditionRatings = conditionRatings.loc[~conditionRatings['superstructure'].isin(['N','NA'])]
after = len(conditionRatings)
print("Total Records after filteration: ",len(conditionRatings))
print("Difference: ", before - after)


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Dividing the time interval of deterioration on basis of suspected intervention. if the deterioration of condition rating is abruptly interfered by sudden increase in condition rating, we will consider a possible case of intervention (Repair, Reconstruction and Repair) and hence the time interval will be divided at the point of intervention, which is considered as splitting point, and second interval will be calculated from next point.
A sudden increase of two or more condition rating is considered as possible intervention. Since, An increase in condition rating of 1 is not considered a possible intervention as the rating is given by inspectors which could be subjective.
The Following function will split all possible intervals and then return a list of condition rating and return number of intervals.
Can you write Python code for it?

def findAllIntervals(lst):
    fList = []
    temp = []
    i = 0
    j = 1
    for k in lst:
        if j == len(lst):
            temp.append(int(lst[i]))
            fList.append(temp)
            return (fList,len(fList))
        if lst[i] == lst[j]:
            pass
        temp.append(int(lst[i]))
        if lst[i] < lst[j]:
            diff = int(lst[j]) - int(lst[i])
            if diff > 1:
               #break
               fList.append(temp)
               temp = []
            pass
        i = i + 1
        j = j + 1


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Extraction of Data from MongoDB and Creating DataFrame
Can you write Python code for it?


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
First five rows from the dataframe
Can you write Python code for it?


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Substructure built after 1992
Can you write Python code for it?

substructureIntervals_1992 = []
substructureIntervalSize_1992 = []
for i in substructureCondition_1992.values():
    lst, size = findAllIntervals(i)
    substructureIntervalSize_1992.append(size)
    substructureIntervals_1992.append(lst)

#
from collections import Counter
keys_sb1992 =list(Counter(substructureIntervalSize_1992).keys())
values_sb1992 =list(Counter(substructureIntervalSize_1992).values())

plt.figure(figsize=(10,8))
plt.bar(keys_sb1992,values_sb1992)
plt.xlabel("No. of Intervals")
plt.ylabel("No. of Records")
plt.xticks([1,2,3,4,5])
plt.title("No. of Records vs No. of Intervals")
plt.show()

substructureIntervalGt2_1992 = []
for index in range(0,len(substructureIntervalSize_1992)-1,1):
    if substructureIntervalSize_1992[index] > 1:
        substructureIntervalGt2_1992.append(substructureIntervals_1992[index])

substructureFirstInterval_1992 = []
substructureSecondInterval_1992 = []
for intervals in substructureIntervalGt2_1992:
    substructureFirstInterval_1992.append(len(intervals[0]))
    substructureSecondInterval_1992.append(len(intervals[1]))
print("Mean of the first time intervals :", np.mean(substructureFirstInterval_1992))
print("Mean of the second time intervals :", np.mean(substructureSecondInterval_1992))
print()

avgStartOfFirstInterval_sb_1992 = []
avgStartOfSecondInterval_sb_1992 = []
for intervals in substructureIntervalGt2_1992:
    avgStartOfFirstInterval_sb_1992.append(intervals[0][0])
    avgStartOfSecondInterval_sb_1992.append(intervals[1][0])
print("Mean of the first condition rating of first time intervals :", np.mean(avgStartOfFirstInterval_sb_1992))
print("Mean of the first condition rating of second time intervals :", np.mean(avgStartOfSecondInterval_sb_1992))

## Consider creating a deterioration model and curves


I am working on the file "Finding+Time+Interval+Before+Intervention+of+Bridge+Components.ipynb".
The first task is:
Superstructure built after 1992
Can you write Python code for it?

superstructureIntervals_1992 = []
superstructureIntervalSize_1992 = []
for i in superstructureCondition_1992.values():
    lst, size = findAllIntervals(i)
    superstructureIntervalSize_1992.append(size)
    superstructureIntervals_1992.append(lst)

#
from collections import Counter
keys_sp1992 =list(Counter(superstructureIntervalSize_1992).keys())
values_sp1992 =list(Counter(superstructureIntervalSize_1992).values())

plt.figure(figsize=(10,8))
plt.bar(keys_sp1992,values_sp1992)
plt.xlabel("No. of Intervals")
plt.ylabel("No. of Records")
plt.xticks([1,2,3,4,5])
plt.title("No. of Records vs No. of Intervals")
plt.show()

superstructureIntervalGt2_1992 = []
for index in range(0,len(superstructureIntervalSize_1992)-1,1):
    if superstructureIntervalSize_1992[index] > 1:
        superstructureIntervalGt2_1992.append(superstructureIntervals_1992[index])

superstructureFirstInterval_1992 = []
superstructureSecondInterval_1992 = []
for intervals in superstructureIntervalGt2_1992:
    superstructureFirstInterval_1992.append(len(intervals[0]))
    superstructureSecondInterval_1992.append(len(intervals[1]))
print("Mean of the first time intervals :", np.mean(superstructureFirstInterval_1992))
print("Mean of the second time intervals :", np.mean(superstructureSecondInterval_1992))
print()

avgStartOfFirstInterval_sp_1992 = []
avgStartOfSecondInterval_sp_1992 = []
for intervals in superstructureIntervalGt2_1992:
    avgStartOfFirstInterval_sp_1992.append(intervals[0][0])
    avgStartOfSecondInterval_sp_1992.append(intervals[1][0])
print("Mean of the first condition rating of first time intervals :", np.mean(avgStartOfFirstInterval_sp_1992))
print("Mean of the first condition rating of second time intervals :", np.mean(avgStartOfSecondInterval_sp_1992))


deckCondition_1992 = {k: g["deck"].tolist() for k,g in CR1992.groupby("structureNumber")}
substructureCondition_1992 = {k: g["substructure"].tolist() for k,g in CR1992.groupby("structureNumber")}
superstructureCondition_1992 = {k: g["superstructure"].tolist() for k,g in CR1992.groupby("structureNumber")}

deckIntervals_1992 = []
deckIntervalSize_1992 = []
for i in deckCondition_1992.values():
    lst, size = findAllIntervals(i)
    deckIntervalSize_1992.append(size)
    deckIntervals_1992.append(lst)

#
from collections import Counter
keys_d1992 =list(Counter(deckIntervalSize_1992).keys())
values_d1992 =list(Counter(deckIntervalSize_1992).values())

plt.figure(figsize=(10,8))
plt.bar(keys_d1992,values_d1992)
plt.xlabel("No. of Intervals")
plt.ylabel("No. of Records")
plt.xticks([1,2,3,4,5])
plt.title("No. of Records vs No. of Intervals")
plt.show()

## Filter records with only two or more interval
deckIntervalGt2_1992 = []
for index in range(0,len(deckIntervalSize_1992)-1,1):
    if deckIntervalSize_1992[index] > 1:
        deckIntervalGt2_1992.append(deckIntervals_1992[index])

deckFirstInterval_1992 = []
deckSecondInterval_1992 = []
for intervals in deckIntervalGt2_1992:
    deckFirstInterval_1992.append(len(intervals[0]))
    deckSecondInterval_1992.append(len(intervals[1]))
print("Mean of the first time intervals :", np.mean(deckFirstInterval_1992))
print("Mean of the second time intervals :", np.mean(deckSecondInterval_1992))
print()
avgStartOfFirstInterval_1992 = []
avgStartOfSecondInterval_1992 = []
for intervals in deckIntervalGt2_1992:
    avgStartOfFirstInterval_1992.append(intervals[0][0])
    avgStartOfSecondInterval_1992.append(intervals[1][0])
print("Mean of the first condition rating of first time intervals :", np.mean(avgStartOfFirstInterval_1992))
print("Mean of the first condition rating of second time intervals :", np.mean(avgStartOfSecondInterval_1992))


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
For highly-skewed feature distributions such as 'capital-gain' and 'capital-loss', it is common practice to apply a logarithmic transformation on the data so that the very large and very small values do not negatively affect the performance of a learning algorithm. Using a logarithmic transformation significantly reduces the range of values caused by outliers. Care must be taken when applying this transformation however: The logarithm of 0 is undefined, so we must translate the values by a small amount above 0 to apply the the logarithm successfully.
Run the code cell below to perform a transformation on the data and visualize the results. Again, note the range of values and how they are distributed.
Can you write Python code for it?

# Log-transform the skewed features
skewed = ['capital-gain', 'capital-loss']
features_raw[skewed] = data[skewed].apply(lambda x: np.log(x + 1))

# Visualize the new log distributions
vs.distribution(features_raw, transformed = True)


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
Implementation: Data Exploration
A cursory investigation of the dataset will determine how many individuals fit into either group, and will tell us about the percentage of these individuals making more than \$50,000. In the code cell below, you will need to compute the following:
- The total number of records, 'n_records'
- The number of individuals making more than \$50,000 annually, 'n_greater_50k'.
- The number of individuals making at most \$50,000 annually, 'n_at_most_50k'.
- The percentage of individuals making more than \$50,000 annually, 'greater_percent'.
Hint: You may need to look at the table above to understand how the 'income' entries are formatted.
Can you write Python code for it?

# TODO: Total number of records
n_records = len(data)

# TODO: Number of records where individual's income is more than $50,000
n_greater_50k = 0
for entry in data.income:
    if entry == '>50K':
        n_greater_50k = n_greater_50k+1

# TODO: Number of records where individual's income is at most $50,000
n_at_most_50k = 0

for entry in data.income:
    if entry == '<=50K':
        n_at_most_50k = n_at_most_50k + 1

# TODO: Percentage of individuals whose income is more than $50,000
greater_percent = (float(n_greater_50k)/n_records)*100

# Print the results
print "Total number of records: {}".format(n_records)
print "Individuals making more than $50,000: {}".format(n_greater_50k)
print "Individuals making at most $50,000: {}".format(n_at_most_50k)
print "Percentage of individuals making more than $50,000: {:.2f}%".format(greater_percent)


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
Implementation: Data Preprocessing
From the table in Exploring the Data above, we can see there are several features for each record that are non-numeric. Typically, learning algorithms expect input to be numeric, which requires that non-numeric features (called categorical variables) be converted. One popular way to convert categorical variables is by using the one-hot encoding scheme. One-hot encoding creates a "dummy" variable for each possible category of each non-numeric feature. For example, assume someFeature has three possible entries: A, B, or C. We then encode this feature into someFeature_A, someFeature_B and someFeature_C.
|   | someFeature |                    | someFeature_A | someFeature_B | someFeature_C |
| :-: | :-: |                            | :-: | :-: | :-: |
| 0 |  B  |  | 0 | 1 | 0 |
| 1 |  C  | ----> one-hot encode ----> | 0 | 0 | 1 |
| 2 |  A  |  | 1 | 0 | 0 |
Additionally, as with the non-numeric features, we need to convert the non-numeric target label, 'income' to numerical values for the learning algorithm to work. Since there are only two possible categories for this label ("<=50K" and ">50K"), we can avoid using one-hot encoding and simply encode these two categories as 0 and 1, respectively. In code cell below, you will need to implement the following:
 - Use pandas.get_dummies() to perform one-hot encoding on the 'features_raw' data.
 - Convert the target label 'income_raw' to numerical entries.
   - Set records with "<=50K" to 0 and records with ">50K" to 1.
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder
import pandas as pd
# TODO: One-hot encode the 'features_raw' data using pandas.get_dummies()
features = pd.get_dummies(features_raw)

le = LabelEncoder()
le.fit(income_raw)

# TODO: Encode the 'income_raw' data to numerical values
income = le.transform(income_raw)

# Print the number of features after one-hot encoding
encoded = list(features.columns)
print "{} total features after one-hot encoding.".format(len(encoded))

# Uncomment the following line to see the encoded feature names
print encoded


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
Implementation: Initial Model Evaluation
In the code cell, you will need to implement the following:
- Import the three supervised learning models you've discussed in the previous section.
- Initialize the three models and store them in 'clf_A', 'clf_B', and 'clf_C'.
  - Use a 'random_state' for each model you use, if provided.
  - Note: Use the default settings for each model — you will tune one specific model in a later section.
- Calculate the number of records equal to 1%, 10%, and 100% of the training data.
  - Store those values in 'samples_1', 'samples_10', and 'samples_100' respectively.
Note: Dependent on which algorithms you chose, the following implementation may take some time to run!
Can you write Python code for it?

# TODO: Import the three supervised learning models from sklearn
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC

# TODO: Initialize the three models
clf_A = GaussianNB()
clf_B = DecisionTreeClassifier(random_state = 1234)
clf_C = SVC(random_state = 1234)

# TODO: Calculate the number of samples for 1%, 10%, and 100% of the training data
samples_1 = len(X_train)/100
samples_10 = len(X_train)/10
samples_100 = len(X_train)/1

# Collect results on the learners
results = {}
for clf in [clf_A, clf_B, clf_C]:
    clf_name = clf.__class__.__name__
    results[clf_name] = {}
    for i, samples in enumerate([samples_1, samples_10, samples_100]):

        results[clf_name][i] = \
        train_predict(clf, samples, X_train, y_train, X_test, y_test)

# Run metrics visualization for the three supervised learning models chosen
vs.evaluate(results, accuracy, fscore)


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
Normalizing Numerical Features
In addition to performing transformations on features that are highly skewed, it is often good practice to perform some type of scaling on numerical features. Applying a scaling to the data does not change the shape of each feature's distribution (such as 'capital-gain' or 'capital-loss' above); however, normalization ensures that each feature is treated equally when applying supervised learners. Note that once scaling is applied, observing the data in its raw form will no longer have the same original meaning, as exampled below.
Run the code cell below to normalize each numerical feature. We will use sklearn.preprocessing.MinMaxScaler for this.
Can you write Python code for it?

# Import sklearn.preprocessing.StandardScaler
from sklearn.preprocessing import MinMaxScaler

# Initialize a scaler, then apply it to the features
scaler = MinMaxScaler()
numerical = ['age', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']
features_raw[numerical] = scaler.fit_transform(data[numerical])

# Show an example of a record with scaling applied
display(features_raw.head(n = 1))


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
Preparing the Data
Before data can be used as input for machine learning algorithms, it often must be cleaned, formatted, and restructured — this is typically known as preprocessing. Fortunately, for this dataset, there are no invalid or missing entries we must deal with, however, there are some qualities about certain features that must be adjusted. This preprocessing can help tremendously with the outcome and predictive power of nearly all learning algorithms.
Transforming Skewed Continuous Features
A dataset may sometimes contain at least one feature whose values tend to lie near a single number, but will also have a non-trivial number of vastly larger or smaller values than that single number.  Algorithms can be sensitive to such distributions of values and can underperform if the range is not properly normalized. With the census dataset two features fit this description: 'capital-gain' and 'capital-loss'.
Run the code cell below to plot a histogram of these two features. Note the range of the values present and how they are distributed.
Can you write Python code for it?

# Split the data into features and target label
income_raw = data['income']
features_raw = data.drop('income', axis = 1)

# Visualize skewed continuous features of original data
vs.distribution(data)


I am working on the file "Finding_Donors_For_Charity/finding_donors.ipynb".
The first task is:
Supverised Learning Models
The following supervised learning models are currently available in scikit-learn that you may choose from:
- Gaussian Naive Bayes (GaussianNB)
- Decision Trees
- Ensemble Methods (Bagging, AdaBoost, Random Forest, Gradient Boosting)
- K-Nearest Neighbors (KNeighbors)
- Stochastic Gradient Descent Classifier (SGDC)
- Support Vector Machines (SVM)
- Logistic Regression
Question 2 - Model Application
List three of the supervised learning models above that are appropriate for this problem that you will test on the census data. For each model chosen
- Describe one real-world application in industry where the model can be applied. (You may need to do research for this — give references!)
- What are the strengths of the model; when does it perform well?
- What are the weaknesses of the model; when does it perform poorly?
- What makes this model a good candidate for the problem, given what you know about the data?
Answer:
As this is a classification problem, the models that I think are appropriate for this are
1. Gaussian Naive Bayes
2. Decision Tree
3. Support Vector Machine
Applications:
1.Naive Bayes for spam detection of emails
2.Tree based classification has been used recently for recognizing three dimensional objects
3.SVM for Protein Fold and Remote Homology Detection
References:
Quora.com, http://www.cbcb.umd.edu/~salzberg/docs/murthy_thesis/survey, http://www.clopinet.com/SVM.applications.html
Strengths and Weaknesses:
Naive Bayes is fast to train, so it performs well on discrete data.
Weakness is that it assumes independence of features.
Decision Trees give you a lot of possibilities, so it will perform well when you have to make more than one decision.
Weakness is overfitting i.e, if you consider all possibilities you might be overfitting the data.
SVM has many kernel tricks and regularisation parameter.
SVMs cannot represent all features as a simple parametric function  since its dimension may be very high.
Why this model is a good candidate:
A naive Bayes classifier...
Can you write Python code for it?

# TODO: Import two metrics from sklearn - fbeta_score and accuracy_score
from sklearn.metrics import fbeta_score, accuracy_score

def train_predict(learner, sample_size, X_train, y_train, X_test, y_test):
    '''
    inputs:
       - learner: the learning algorithm to be trained and predicted on
       - sample_size: the size of samples (number) to be drawn from training set
       - X_train: features training set
       - y_train: income training set
       - X_test: features testing set
       - y_test: income testing set
    '''

    results = {}

    # TODO: Fit the learner to the training data using slicing with 'sample_size'
    start = time() # Get start time
    learner.fit(X_train[:sample_size], y_train[:sample_size])
    end = time() # Get end time

    # TODO: Calculate the training time
    results['train_time'] = end-start

    # TODO: Get the predictions on the test set,
    #       then get predictions on the first 300 training samples
    start = time() # Get start time
    predictions_test = learner.predict(X_test)
    predictions_train = learner.predict(X_train[:300])
    end = time() # Get end time

    # TODO: Calculate the total prediction time
    results['pred_time'] = end-start

    # TODO: Compute accuracy on the first 300 training samples
    results['acc_train'] = accuracy_score(y_train[:300],predictions_train)

    # TODO: Compute accuracy on test set
    results['acc_test'] = accuracy_score(y_test,predictions_test)

    # TODO: Compute F-score on the the first 300 training samples
    results['f_train'] = fbeta_score(y_train[:300],predictions_train, beta = 0.5)

    # TODO: Compute F-score on the test set
    results['f_test'] = fbeta_score(y_test,predictions_test,beta =0.5)

    # Success
    print "{} trained on {} samples.".format(learner.__class__.__name__, sample_size)

    # Return the results
    return results


I am working on the file "FiniteSizeScaling/plotScript.ipynb".
The first task is:
Load the data and perform the linear fit for each BC and interaction strength
Can you write Python code for it?

S2scaled = {}
for cBC in BC:
    for cV in V:

        # load raw data
        data = np.loadtxt('N1%sn1u_%3.1f.dat'%(cBC[0],cV))

        # Raises each N to the power of the leading finite size correction γ=(4g+1)
        x = data[:,0]**(-(1.0+4*g(cV)))

        # perform the linear fit
        p = np.polyfit(x,data[:,3],1)

        # rescale the data
        y = (data[:,3]-p[1])/p[0]

        # store
        S2scaled['%s%3.1f'%(cBC[0],cV)] = np.stack((x,y),axis=1)


I am working on the file "FiniteSizeScaling/plotScript.ipynb".
The first task is:
Plot the scaled finite-size data collapse
Can you write Python code for it?

colors = ['#4173b3','#e95c47','#7dcba4','#5e4ea2','#fdbe6e','#808080','#2e8b57','#b8860b','#87ceeb']
markers = ['o','s','h','D','H','>','^','<','v']

plt.style.reload_library()
with plt.style.context('../IOP_large.mplstyle'):

    # Create the figure
    fig1 = plt.figure()
    ax2 = fig1.add_subplot(111)
    ax3 = fig1.add_subplot(111)
    ax2.set_xlabel(r'$N^{-(1+4g)}$')
    ax2.set_ylabel(r'$(S_2(n=1)-{\rm{ln}}(N)-a)/b$')
    ax2.set_xlim(0,0.52)
    ax2.set_ylim(0,0.52)
    ax3 = ax2.twinx()
    ax3.set_xlim(0,0.52)
    ax3.set_ylim(0,0.52)
    ax1 = fig1.add_subplot(111)
    ax1.set_xlim(0,0.52)
    ax1.set_ylim(0,0.52)

    # Plots (S2(n=1)-ln(N)-a)/b vs. N^-(4g+1)
    # anti-periodic boundary conditions
    for i,cV in enumerate(V):
        data = S2scaled['A%3.1f'%cV]
        if cV > 0:
            ax = ax2
        else:
            ax = ax3

        ax.plot(data[:,0],data[:,1], marker=markers[i], color=colors[i], mfc='None', mew='1.0',
                linestyle='None', label=r'$%3.1f$'%cV)

    # U/t > 0 legend
    ax2.legend(loc=(0.01,0.29),frameon=False,numpoints=1,ncol=1,title=r'$V/t$')

    # periodic boundary conditions
    for i,cV in enumerate(V):
        data = S2scaled['P%3.1f'%cV]

        ax1.plot(data[:,0],data[:,1], marker=markers[i], color=colors[i], mfc='None', mew=1.0,
                 linestyle='None', label='None')

    # U/t < 0 legend
    ax3.legend(loc=(0.65,0.04),frameon=False,numpoints=1,ncol=1,title=r'$V/t$')
    ax3.tick_params(
       axis='both',          # changes apply to the x-axis
       which='both',      # both major and minor ticks are affected
       right='off',      # ticks along the bottom edge are off
       top='off',         # ticks along the top edge are off
       labelright='off') # labels along the bottom edge are off
    ax2.tick_params(
       axis='both',          # changes apply to the x-axis
       which='both',      # both major and minor ticks are affected
       right='on',
       top='on')         # ticks...


I am working on the file "First Class Functions.ipynb".
The first task is:
Anonymous Functions
The example above, where we've utilized map and filter combined with a lambda function leads us into our next example.
The lambda keyword created an anonymous function within a Python expression. However the syntax limits the lambda to be pure expressions. This means that the body of a lambda function can't use other Python statements such as while or try, etc.
These are typically limited in their use because of the lack of the ability to use more complex control structures within the lambda functions. This can lead to unreadable or unworkable results.
However, they can still prove useful in certain contexts, such as list arguments.
Can you write Python code for it?

food = ['eggplant', 'carrots', 'celery',
        'potatoes', 'tomatoes', 'rhubarb',
        'strawberry', 'blueberry', 'raspberry',
        'banana', 'cherry']

print(sorted(food, key=lambda word: word[::-1]))


I am working on the file "First Class Functions.ipynb".
The first task is:
Any single argument function can be used in the key argument of the sorted() method.
as a trivial example, we may want to use the reversed order of the characters to sort of words, as this will cause certain clustering of character strings together, such as -berry, and -toes.
Can you write Python code for it?

def reverse(word):
    '''
    Reverse the order of the letters in a given string
    '''
    return word[::-1]

print(sorted(food, key=reverse))


I am working on the file "First Class Functions.ipynb".
The first task is:
First class analysis
We can show the first class nature of this function object using a few examples.
We can assign the function to a variable, which will invoke the function when calling that variable.
Can you write Python code for it?


I am working on the file "First Class Functions.ipynb".
The first task is:
Higher-Order Functions
A higher order function is a bit....meta. It can take, as an argument, a function and then returns a function as a result.
The map() example used above is a great example of this.
The built-in sorted() function is another great example of this, within Python. We can pass it an iterable, along with a key that can then be applied in succession to the items in the list.
Can you write Python code for it?

food = ['eggplant', 'carrots', 'celery',
        'potatoes', 'tomatoes', 'rhubarb',
        'strawberry', 'blueberry', 'raspberry',
        'banana', 'cherry']

print(sorted(food, key=len))


I am working on the file "First Class Functions.ipynb".
The first task is:
Replacements for map and filter
Map, filter, and reduce are typically offered in functional languages as higher order functions. However, the introduction of list comprehensions and generator expressions have downplayed the value of the map and filter functions, as listcomp's and genexp's combine the job of map and filter.
Can you write Python code for it?

# Build a list of factorials from 0! to 5!
list(map(fact, range(6)))

# Build a list of factorials from 0! to 5!
# but using list comprehension
[fact(n) for n in range(6)]

# Build a list of factorials of odd numbers up to 5!, using `map` and `filter`
list(map(factorial, filter(lambda n: n % 2, range(6))))


I am working on the file "First Class Functions.ipynb".
The first task is:
We can also use the map function, and pass our function as the first argument, allowing that function to be evaluated against the second argument, which is an iterable. Allowing this function to be applied in a successive fashion to all elements of this iterable.
Can you write Python code for it?


I am working on the file "First Example.ipynb".
The first task is:
As you can see in the above equation, with the $x$ variable representing our genome list of integers, the $f(x)$ shows our evaluation function, which is the sum of ‘0’ values in the list. For example, if we have a list with 10 elements like this:
Can you write Python code for it?


I am working on the file "First Example.ipynb".
The first task is:
From the second generation on, we use the step function. We will display the best individual and its score, and plot the scores of the population, for each individual, and the histogram.
You may repeat the cell below many times by pressing Ctrl+Enter.
Can you write Python code for it?

ga.step()
print("Generation: %d" % ga.currentGeneration)
best = ga.bestIndividual()
print('Best individual: %s' % str(best.genomeList))
print('Best score: %.0f' % best.score)
population = ga.getPopulation()
it.plotPopScore(population)
it.plotHistPopScore(population)


I am working on the file "First Example.ipynb".
The first task is:
It is important to note that in Pyevolve, we have raw score and fitness score, the raw score is the return of the evaluation function and the fitness score is the scaled score.
The next step is the creation of one sample genome for the Genetic Algorithm. We can define our genome as this:
Can you write Python code for it?

# Genome instance
genome = G1DList.G1DList(20)

# The evaluator function (objective function)
genome.evaluator.set(eval_func)


I am working on the file "First Example.ipynb".
The first task is:
Ready! Simple, isn't it? We simple create our GA Engine with the created genome. You can ask: “Where is the selector method? The number of generations? Mutation rate?“. Again: we have defaults. By default, the GA will evolve for 100 generations with a population size of 80 individuals, it will use the mutation rate of 2% and a crossover rate of 80%, the default selector is the Ranking Selection (Selectors.GRankSelector()) method. Those default parameters was not random picked, they are all based on the commom used properties.
We need to import the Interaction module, which includes some plot functions. We will plot the score of all the individuals of the population after each generation.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import pyevolve.Interaction as it


I am working on the file "First Example.ipynb".
The first task is:
Right, now we have our evaluation function and our first genome ready, the next step is to create our Genetic Algorithm Engine, the GA Core which will do the evolution, control statistics, etc... The GA Engine which we will use is the GSimpleGA.GSimpleGA which resides in the GSimpleGA module, this GA Engine is the genetic algorithm described by Goldberg. So, let’s create the engine:
Can you write Python code for it?

from pyevolve import GSimpleGA
ga = GSimpleGA.GSimpleGA(genome)


I am working on the file "First Example.ipynb".
The first task is:
You can change everything with the API, for example, you can pass the ranges to the genome, like this:
Can you write Python code for it?


I am working on the file "First Five.ipynb".
The first task is:
Problem 1.
If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.  Find the sum of all the multiples of 3 or 5 below 1000.
Can you write Python code for it?

x = range(1,10)
x

def sum_number_mode(a,b,maxbarrier):
    x = 0
    for i in range(1,maxbarrier-1):
        if i%a == 0 or i%b == 0:
            x += i
    return x

sum_number_mode(3,5,1001)


I am working on the file "First Five.ipynb".
The first task is:
Problem 2.
Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:
1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...
By considering the terms in the Fibonacci sequence whose values do not exceed four million, find the sum of the even-valued terms.
Can you write Python code for it?

x = [3,4]
x[1]

def sum_even_fibonacci(mod, maxfibonacci):
    x = [1,2]
    sum_fib = 0

    while x[1] < maxfibonacci:
        if x[1]%mod == 0:
            sum_fib += x[1]
        y = x[1]
        x[1] += x[0]
        x[0] = y

    return sum_fib

sum_even_fibonacci(2,40000000000)


I am working on the file "First Five.ipynb".
The first task is:
Problem 3.
The prime factors of 13195 are 5, 7, 13 and 29.
What is the largest prime factor of the number 600851475143 ?
Can you write Python code for it?

def greatest_factor(prime):
    x = 2  # initiate first factor
    while x != prime:
        if prime%x == 0:
            result = prime/x
            prime = result

        else:
            x += 1
    else:
        return prime

greatest_factor(81)


I am working on the file "First Five.ipynb".
The first task is:
Problem 5.
2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder.
What is the smallest positive number that is evenly divisible by all of the numbers from 1 to 20?
Can you write Python code for it?

def smallest_divided(start, max_number):
    factors = range(start, max_number + 1)
    x = (len(factors)) - 1
    factors_result = factors[0]
    while (factors[x]-1) >= factors[0]:
        if factors_result%factors[x-1] !=0 :
            factors_result *= factors[x-1]
            print factors_result
        x -= 1
    #return factors_result

smallest_divided(1,20)

def factoring(prime):
    x = 2  # initiate first factor
    factors = []
    result = 0
    while x != prime:
        while (prime%x == 0):
            result = prime/x
            prime = result
            factors.append(x)
            if prime == 1:
                break
        if prime == 1:
                break
        else:
            x += 1
    else:
        factors.append(prime)
    return factors

factoring(20)

def smallest_divided(start, max_number):
    x = range(start, max_number + 1)
    factors = []
    for i in x:
        z = factoring(i)
        factors.append(z)
    return factors

smallest_divided(1,20)
faktor = smallest_divided(1,20)

set_factor = faktor
def add_set(set_factor):
    set_all = []
    intersect = []
    join = []
    for i in set_factor:
        intersect = i and set_all
        i = i - intersect
        set_all = i + set_all

    return set_all

set_all_simulate = []
for i in set_factor:
    set_all_simulate += i

print set_all_simulate

factor_set = set(set_all_simulate)
factor_set
z = 1
for i in factor_set:
    z *= i
print z

set_factor


I am working on the file "First Five.ipynb".
The first task is:
Problem 6.
The sum of the squares of the first ten natural numbers is,
1^2 + 2^2 + ... + 10^2 = 385
The square of the sum of the first ten natural numbers is,
(1 + 2 + ... + 10)^2 = 552 = 3025
Hence the difference between the sum of the squares of the first ten natural numbers and the square of the sum is 3025 − 385 = 2640.
Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.
Can you write Python code for it?

def sum_of_squares(start, end):
    sum_all = 0
    for i in range(start, end +1):
        sum_all += (i)**2
    return sum_all
su_sq = sum_of_squares(1,100)

def square_of_sum(start, end):
    sum_all_i = 0
    b = 0
    for i in range(start, end + 1):
        sum_all_i += i
    b = sum_all_i * sum_all_i
    return b

sq_su = square_of_sum(1,100)

sq_su - su_sq


I am working on the file "First Steps.ipynb".
The first task is:
1) Instantiate an object and set the parameters
Can you write Python code for it?


I am working on the file "First Steps.ipynb".
The first task is:
2) Fit the model
Can you write Python code for it?


I am working on the file "First Steps.ipynb".
The first task is:
And again
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=50)

rf.fit(X_train, y_train)

rf.score(X_test, y_test)

%load https://raw.githubusercontent.com/scikit-learn/scikit-learn/master/examples/classification/plot_classifier_comparison.py


I am working on the file "First Steps.ipynb".
The first task is:
Exercises
Load the iris dataset from the sklearn.datasets module using the load_iris function.
Split it into training and test set using train_test_split.
Then train an evaluate a classifier of your choice.
Can you write Python code for it?


I am working on the file "First Steps.ipynb".
The first task is:
Get some data to play with
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(digits.data,
                                                    digits.target)

X_train.shape


I am working on the file "First look at the Data.ipynb".
The first task is:
1. Read the data
Can you write Python code for it?

df = pd.read_csv("../MC1 2015 Data/park-movement-Fri.csv")


I am working on the file "First look at the Data.ipynb".
The first task is:
3. Single guest
Now lets plot the locations for a single random person
Can you write Python code for it?

guest_id = np.random.choice(df["id"])

guest_df = df.loc[df["id"]==guest_id]

guest_df.shape

guest_df.plot("X","Y","scatter")


I am working on the file "First look at the Data.ipynb".
The first task is:
4. Single time frame
Where were the guests between 12:00 and 12:05 ?
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
5. Time analysis
Now lets try to ask some simple questions about time data

At what time do guests arrive?
At what time do they leave?
How long they stay?
How does park ocupacy vary during the day?

To answer the first questions we needd to transform the data
Can you write Python code for it?

grouped_times = df.groupby("id")["time"]

arrivals = grouped_times.min()

departures = grouped_times.max()

duration = departures - arrivals

sns.distplot(arrivals.dt.hour+arrivals.dt.minute/60)

sns.distplot(departures.dt.hour+departures.dt.minute/60)

h_duration = duration.dt.seconds/60/60

sns.distplot(h_duration)


I am working on the file "First look at the Data.ipynb".
The first task is:
At what time did he leave?
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
It looks very similar to the pats in the map
Now lets look at just the check-in samples
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
Let's look at the first five rows
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
Lets look at the range of the location data
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
Now for the question of park occupacy, we need to group the dataframe by time
Can you write Python code for it?

time_groups = df.groupby(df.time.dt.hour)

occupancy = time_groups["id"].aggregate(lambda x:len(np.unique(x)))

occupancy.plot()


I am working on the file "First look at the Data.ipynb".
The first task is:
Now lets create a 2d histogram to see which locations are more popular. We will use the hist2d function
Can you write Python code for it?

cnts, xe, ye, img = plt.hist2d(df_small["X"], df_small["Y"],range=((0,100),(0,100)),normed=True)


I am working on the file "First look at the Data.ipynb".
The first task is:
Now the time column contains datetime objects
2. Looking at location data
First, take a random subsample to speed up exploration
Can you write Python code for it?

df_small = df.sample(10000)

df_small.shape


I am working on the file "First look at the Data.ipynb".
The first task is:
They are strings, it would be better if they were dats, lets fix that
with the to_datetime function
Can you write Python code for it?

df["time"] = pd.to_datetime(df.Timestamp, format="%Y-%m-%d %H:%M:%S")

df.tail()

df.dtypes


I am working on the file "First look at the Data.ipynb".
The first task is:
We can increase the number of bins
Can you write Python code for it?

cnts, xe, ye, img = plt.hist2d(df_small["X"], df_small["Y"],range=((0,100),(0,100)),normed=True, bins=20)

df_small.plot("X","Y","hexbin")


I am working on the file "First look at the Data.ipynb".
The first task is:
We will now create a simple scatter plot with all the X and Y values in our subsample
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
What are the types of the data?
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
What are the values of type ?
Can you write Python code for it?


I am working on the file "First look at the Data.ipynb".
The first task is:
What is the type of the timestamps?
Can you write Python code for it?


I am working on the file "First-Contact-With-Data.ipynb".
The first task is:
Header
Can you write Python code for it?

# Command line
!head -1 taxi.csv | tr , \\n
!printf "%d fields" $(head -1 taxi.csv | tr , \\n | wc -l)

# Python
import csv
with open('taxi.csv') as fp:
    fields = next(csv.reader(fp))
print('\n'.join(fields))
print('%d fields' % len(fields))


I am working on the file "First-Contact-With-Data.ipynb".
The first task is:
Number of Lines
Can you write Python code for it?

# Command line
!wc -l taxi.csv

# Python
with open('taxi.csv') as fp:
    print(sum(1 for _ in fp))