I am working on the file "Machine Learning - Hands on Lab - Session #1 - Data Visualisation.ipynb".
The first task is:
Here are the information available on our test users:

id: user ID
date_account_created: date of account creation
timestamp_first_active: timestamp of the first activity, note that it can be earlier than date_account_created or date_first_booking because a user can search before signing up
date_first_booking: date of first booking
gender
age
signup_method: What method was used to signup (Facebook, basic, etc.)
signup_flow: the page a user came to signup up from
language: international language preference
affiliate_channel: what kind of paid marketing
affiliate_provider: where the marketing is e.g. google, craigslist, other
first_affiliate_tracked: whats the first marketing the user interacted with before the signing up
signup_app
first_device_type
first_browser
<span style="color:red">country_destination: this is the target variable that we want to predict </span>

Remark on country_destination =>  12 Possibilities: US, FR, CA, GB, ES, IT, PT, NL,DE, AU, NDF (no destination found), and other
1.2.5 session.csv
We load the csv file and visualise it straight away
Can you write Python code for it?

df_sessions = pd.read_csv("data/sessions.csv")

print("There are " + str(df_sessions.shape[0])+ " rows in the dataset")

df_sessions.head(n=5) # Only display a few lines and not the whole dataframe


I am working on the file "Machine Learning - Hands on Lab - Session #1 - Data Visualisation.ipynb".
The first task is:
Now it appears that we have a file containing demographic information by country, by age and by gender.
For each "age-bucket", "country" and gender, we have the following information:
* Population in thousands
* year of the statistic
Can you write Python code for it?

tmp_sorted.year.unique() # We query the distinct values over the last column "year"


I am working on the file "Machine Learning - Hands on Lab - Session #1 - Data Visualisation.ipynb".
The first task is:
We have a file containing a list of countries with the following features:

country code
latitude
longitude
distance in km
surface in km2
language
levenshtein distance with English Language

=> General information about countries of destination
1.2.3 age_gender_bkts.csv
We load the csv file and visualise it straight away
Can you write Python code for it?

df_age_gender_bkts = pd.read_csv("data/age_gender_bkts.csv")
df_age_gender_bkts.head(n=5) # Only display a few lines and not the whole dataframe


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
2. Loading in the Data
The first step is to load the data from the CSV files using Python.
To do this we will use the Pandas library and load the data from two files train_users_2.csv and test_users.csv.
After loading, we will combine them into one dataset so that any cleaning (and later any other changes) will be done to all the data at once.
Can you write Python code for it?

df_train = pd.read_csv("data/train_users_2.csv")
df_train.sample(n=5) # Only display a few lines and not the whole dataframe

df_test = pd.read_csv("data/test_users.csv")
df_test.sample(n=5) # Only display a few lines and not the whole dataframe

# Combine into one dataset
df_all = pd.concat((df_train, df_test), axis=0, ignore_index=True)
df_all.head(n=5) # Only display a few lines and not the whole dataframe


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
3. Cleaning the timestamp
Once the data has been loaded and combined, the first cleaning step we will undertake is fixing the format of the dates.
Fixing date format is essential in order to simplify date manipulation.
Can you write Python code for it?

df_all['date_account_created'] = pd.to_datetime(df_all['date_account_created'], format='%Y-%m-%d')
df_all['timestamp_first_active'] = pd.to_datetime(df_all['timestamp_first_active'], format='%Y%m%d%H%M%S')

df_all.head(n=5) # Only display a few lines and not the whole dataframe


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
4. Remove booking date field
For every test data the column "date_first_booking" is missing, so we can't rely on this one for our prediction model.
=> We must delete the column
Can you write Python code for it?

# Remove date_first_booking column
df_all.drop('date_first_booking', axis=1, inplace=True)


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
5. Clean the Age column
There are several age values that are clearly incorrect (unreasonably high or too low).
In this step, we replace these incorrect values with "NaN", which literally stands for Not a Number, but implies we do not know the age value. In other words we are changing the incorrect values into missing values.
Can you write Python code for it?

# Remove outliers function
def remove_outlier(x, min_val=15, max_val=90):
    if np.isnan(x):
        return np.nan
    elif np.logical_or(x<=min_val, x>=max_val):
        return np.nan
    else:
        return x

# Fixing age column
df_all['age'] = df_all['age'].apply(lambda x: remove_outlier(x, min_val=15, max_val=90))
df_all['age'].fillna(-1, inplace=True)

df_all.sample(n=5)


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
7. Identify and fill additional columns with missing values
Can you write Python code for it?

def check_NaN_Values_in_df(df):
    # searching for NaN values is all the columns
    for col in df:
        nan_count = df[col].isnull().sum()

        if nan_count != 0:
            print (col + " => "+  str(nan_count) + " NaN Values")

check_NaN_Values_in_df(df_all)


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
As mentioned earlier, there are several more complicated ways to fill in the missing values in the age column.
We are selecting this simple method for two main reasons:

Clarity: this workshop is going to be long enough without adding the complication of a complex methodology for imputing missing ages.

Questionable results: in my testing during the actual competition, I did test several more complex imputation methodologies. However, none of the methods I tested actually produced a better end result than the methodology outlined above.

6. Converting Age Column to Integers
Values inside the age columns are floats, it is better to have them as integers
Can you write Python code for it?

df_all.age = df_all.age.astype(int)
df_all.sample(n=5)


I am working on the file "Machine Learning - Hands on Lab - Session #2 - Data Cleaning.ipynb".
The first task is:
It is absolutely normal that country_destination contains NaN values, it is the output field. NaN Values are test data.
However, we need to fix the column first_affiliate_tracked
Can you write Python code for it?

df_all['first_affiliate_tracked'].fillna(-1, inplace=True)

# We check there is no more NaN except in country_destination column
check_NaN_Values_in_df(df_all)

df_all.sample(n=5)


I am working on the file "Machine Learning - Hands on Lab - Session #3 - Feature Engineering.ipynb".
The first task is:
3. Loading in the Data from Session #2
Can you write Python code for it?

df_all = pd.read_csv(
    "output/cleaned.csv",
    dtype={
        'country_destination': str
    }
)

# We transform again the date column into datetime
df_all['date_account_created'] = pd.to_datetime(df_all['date_account_created'], format='%Y-%m-%d %H:%M:%S')
df_all['timestamp_first_active'] = pd.to_datetime(df_all['timestamp_first_active'], format='%Y-%m-%d %H:%M:%S')

# Check for NaN Values => We must find: country_destination => 62096 NaN Values
check_NaN_Values_in_df(df_all)

df_all.sample(n=5) # Only display a few lines and not the whole dataframe


I am working on the file "Machine Learning - Hands on Lab - Session #3 - Feature Engineering.ipynb".
The first task is:
3. Transforming Categorical Data
Let's go for some One Hot Encoding - replacing the categorical fields in the dataset with multiple columns representing one value from each column.
Can you write Python code for it?

# Home made One Hot Encoding function
def convert_to_binary(df, column_to_convert):
    categories = list(df[column_to_convert].drop_duplicates())

    for category in categories:
        cat_name = str(category).replace(" ", "_").replace("(", "").replace(")", "").replace("/", "_").replace("-", "").lower()
        col_name = column_to_convert[:5] + '_' + cat_name[:10]
        df[col_name] = 0
        df.loc[(df[column_to_convert] == category), col_name] = 1

    return df

columns_to_convert = [
    'gender',
    'signup_method',
    'signup_flow',
    'language',
    'affiliate_channel',
    'affiliate_provider',
    'first_affiliate_tracked',
    'signup_app',
    'first_device_type',
    'first_browser'
]

# One Hot Encoding
for column in columns_to_convert:
    df_all = convert_to_binary(df=df_all, column_to_convert=column)
    df_all.drop(column, axis=1, inplace=True)

df_all.sample(n=5)

# Add new date related fields
df_all['day_account_created'] = df_all['date_account_created'].dt.weekday
df_all['month_account_created'] = df_all['date_account_created'].dt.month
df_all['quarter_account_created'] = df_all['date_account_created'].dt.quarter
df_all['year_account_created'] = df_all['date_account_created'].dt.year
df_all['hour_first_active'] = df_all['timestamp_first_active'].dt.hour
df_all['day_first_active'] = df_all['timestamp_first_active'].dt.weekday
df_all['month_first_active'] = df_all['timestamp_first_active'].dt.month
df_all['quarter_first_active'] = df_all['timestamp_first_active'].dt.quarter
df_all['year_first_active'] = df_all['timestamp_first_active'].dt.year
df_all['created_less_active'] = (df_all['date_account_created'] - df_all['timestamp_first_active']).dt.days

# Drop unnecessary columns
columns_to_drop = ['date_account_created', 'timestamp_first_active', 'date_first_booking', 'country_destination']
for column in columns_to_drop:
    if column in df_all.columns:
        df_all.drop(column, axis=1, inplace=True)

print ("Dataframe Shape:", df_all.shape)
df_all.sample(n=5)


I am working on the file "Machine Learning - Hands on Lab - Session #3 - Feature Engineering.ipynb".
The first task is:
6.2. Cleaning and Transforming the Data
6.2.1. Extract the primary and secondary devices for each user
The first piece of information we are going to extract is the primary and secondary device for each user.
How do we determine what is the user's primary and secondary devices are? We look at how much time they spent on each device.
Can you write Python code for it?

# Determine primary device
sessions_device = df_sessions.loc[:, ['user_id', 'device_type', 'secs_elapsed']]
aggregated_lvl1 = sessions_device.groupby(['user_id', 'device_type'], as_index=False, sort=False).aggregate(np.sum)
idx = aggregated_lvl1.groupby(['user_id'], sort=False)['secs_elapsed'].transform(max) == aggregated_lvl1['secs_elapsed']
df_primary = pd.DataFrame(aggregated_lvl1.loc[idx , ['user_id', 'device_type', 'secs_elapsed']])
df_primary.rename(columns = {'device_type':'primary_device', 'secs_elapsed':'primary_secs'}, inplace=True)
df_primary = convert_to_binary(df=df_primary, column_to_convert='primary_device')
df_primary.drop('primary_device', axis=1, inplace=True)

df_primary.sample(n=5)

# Determine Secondary device
remaining = aggregated_lvl1.drop(aggregated_lvl1.index[idx])
idx = remaining.groupby(['user_id'], sort=False)['secs_elapsed'].transform(max) == remaining['secs_elapsed']
df_secondary = pd.DataFrame(remaining.loc[idx , ['user_id', 'device_type', 'secs_elapsed']])
df_secondary.rename(columns = {'device_type':'secondary_device', 'secs_elapsed':'secondary_secs'}, inplace=True)
df_secondary = convert_to_binary(df=df_secondary, column_to_convert='secondary_device')
df_secondary.drop('secondary_device', axis=1, inplace=True)

df_secondary.sample(n=5)


I am working on the file "Machine Learning - Hands on Lab - Session #3 - Feature Engineering.ipynb".
The first task is:
6.2.2. Determine Counts of Actions
The next thing we are going to do is take counts of how many times each action was taken by each user. This is a two-step process.
To handle the multiple action columns, we repeat these steps for each column individually, effectively creating three separate tables.
Because we have now created tables where each row represents one user, we can now join these three tables together on the basis of the user id.
Can you write Python code for it?

# Count occurrences of value in a column
def convert_to_counts(df, id_col, column_to_convert):
    id_list = df[id_col].drop_duplicates()

    df_counts = df.loc[:,[id_col, column_to_convert]]
    df_counts['count'] = 1
    df_counts = df_counts.groupby(by=[id_col, column_to_convert], as_index=False, sort=False).sum()

    new_df = df_counts.pivot(index=id_col, columns=column_to_convert, values='count')
    new_df = new_df.fillna(0)

    # Rename Columns
    categories = list(df[column_to_convert].drop_duplicates())
    for category in categories:
        cat_name = str(category).replace(" ", "_").replace("(", "").replace(")", "").replace("/", "_").replace("-", "").lower()
        col_name = column_to_convert + '_' + cat_name
        new_df.rename(columns = {category:col_name}, inplace=True)

    return new_df

# Aggregate and combine actions taken columns

session_actions = df_sessions.loc[:,['user_id', 'action', 'action_type', 'action_detail']]
columns_to_convert = ['action', 'action_type', 'action_detail']

session_actions = session_actions.fillna('not provided')
first = True

for column in columns_to_convert:
    print("Converting " + column + " column...")
    current_data = convert_to_counts(df=session_actions, id_col='user_id', column_to_convert=column)

    # If first loop, current data becomes existing data, otherwise merge existing and current
    if first:
        first = False
        actions_data = current_data
    else:
        actions_data = pd.concat([actions_data, current_data], axis=1, join='inner')

actions_data.sample(n=5)


I am working on the file "Machine Learning - Hands on Lab - Session #3 - Feature Engineering.ipynb".
The first task is:
7. Saving the DataFrame to csv
Can you write Python code for it?

# We create the output directory if necessary
if not os.path.exists("output"):
    os.makedirs("output")

# We export to csv
df_all.to_csv("output/enriched.csv", sep=',')


I am working on the file "Machine Learning - Hands on Lab - Session #4 - Model Design.ipynb".
The first task is:
2. Re-Import the Check NaN Function from Part 2
Can you write Python code for it?

def check_NaN_Values_in_df(df):
    # searching for NaN values is all the columns
    for col in df:
        nan_count = df[col].isnull().sum()

        if nan_count != 0:
            print (col + " => "+  str(nan_count) + " NaN Values")


I am working on the file "Machine Learning - Hands on Lab - Session #4 - Model Design.ipynb".
The first task is:
3. Loading in the Data from Part 3
Can you write Python code for it?

df_all = pd.read_csv("output/enriched.csv", index_col=0)

# Check for NaN Values => We must find: country_destination => 62096 NaN Values
check_NaN_Values_in_df(df_all)

df_all.sample(n=5) # Only display a few lines and not the whole dataframe


I am working on the file "Machine Learning - Hands on Lab - Session #4 - Model Design.ipynb".
The first task is:
6. Saving the model
=> loaded_model = joblib.load("models/0001.model") # To load back the model
Can you write Python code for it?

# We create the output directory if necessary
if not os.path.exists("models"):
    os.makedirs("models")

# save model to file
joblib.dump(model, 'models/0001.model')


I am working on the file "Machine Learning - Hands on Lab - Session #4 - Model Design.ipynb".
The first task is:
7. Making the Predictions
Now that we have trained a model based on the best parameters, the next step is to use the model to make predictions for the records in the testing dataset. Again we need to extract the testing data out of the combined dataset we created for the cleaning and transformation steps, and again we need to separate the main components for the model. After these steps, we use the model created in the previous step to make the predictions.
Can you write Python code for it?

df_test = pd.read_csv("data/test_users.csv")
df_test.head(n=5) # Only display a few lines and not the whole dataframe

# Prepare test data for prediction
df_test.set_index('id', inplace=True)
df_test = pd.merge(df_test.loc[:,['date_first_booking']], df_all, how='left', left_index=True, right_index=True, sort=False)
X_test = df_test.drop('date_first_booking', axis=1, inplace=False)
X_test = X_test.fillna(-1)
id_test = df_test.index.values

# Make predictions
y_pred = model.predict_proba(X_test)

#Taking the 5 classes with highest probabilities
ids = []  #list of ids
cts = []  #list of countries
for i in range(len(id_test)):
    idx = id_test[i]
    ids += [idx] * 5
    cts += le.inverse_transform(np.argsort(y_pred[i])[::-1])[:5].tolist()


I am working on the file "Machine Learning - Hands on Lab - Session #4 - Model Design.ipynb".
The first task is:
8. Saving Predictions
Can you write Python code for it?

# We create the output directory if necessary
if not os.path.exists("output"):
    os.makedirs("output")

#Generate submission
sub = pd.DataFrame(np.column_stack((ids, cts)), columns=['id', 'country'])
sub.to_csv('output/submission.csv', index=False, sep=',')


I am working on the file "Machine Learning - Titanic Disaster Solutions.ipynb".
The first task is:
Analyse data
Visualize the first 5 rows with the head() function.
Can you write Python code for it?


I am working on the file "Machine Learning - Titanic Disaster Solutions.ipynb".
The first task is:
Check and dealing wiht null values
Can you write Python code for it?

# Visualize the null values (train)
one_hot_train.isnull().sum().sort_values(ascending=False)

# Fill the null Age values with the mean of all ages
one_hot_train['Age'].fillna(one_hot_train['Age'].mean(), inplace=True)
one_hot_test['Age'].fillna(one_hot_test['Age'].mean(), inplace=True)
one_hot_train.isnull().sum()

# Visualize the null values (test)
one_hot_test.isnull().sum().sort_values(ascending=False)

# Fill the null Fare values with the mean of all Fares
one_hot_test['Fare'].fillna(one_hot_test['Fare'].mean(), inplace=True)
one_hot_test.isnull().sum().sort_values(ascending=False)


I am working on the file "Machine Learning - Titanic Disaster Solutions.ipynb".
The first task is:
Generate one-hot (dummies) variables from categorical data
Using the 'get_dummies' function from Pandas to gerenate the one-hot encoders
Can you write Python code for it?

one_hot_train = pd.get_dummies(train)
one_hot_test = pd.get_dummies(test)

# First five rows from train dataset
one_hot_train.head()

# First five rows from test dataset
one_hot_test.head()


I am working on the file "Machine Learning Basics with Sklearn.ipynb".
The first task is:
Implementing your own KNN
Let's implement a simple knn classifier with k=1.
We have to implement the fit method and the predict method.
Then initialize, train and print error.
Can you write Python code for it?

from scipy.spatial import distance

class ScrappyKNN(object):
    def fit(self, X_train, y_train):
        self.X_train = X_train
        self.y_train = y_train
        return self

    def predict(self, X_test):
        predictions = []
        for row in X_test:
            label = self.closest(row)
            predictions.append(label)
        return predictions

    def closest(self, row):
        best_dist = distance.euclidean(row, self.X_train[0])
        best_index = 0
        for i in range(1, len(self.X_train)):
            dist = distance.euclidean(row, self.X_train[i])
            if dist < best_dist:
                best_dist = dist
                best_index = i

        return self.y_train[best_index]

clf = ScrappyKNN()
clf = clf.fit(train_X,train_y)
print(accuracy_score(test_y, clf.predict(test_X)))


I am working on the file "Machine Learning Basics with Sklearn.ipynb".
The first task is:
KNN-Classifier
Let's try another classifier. Initialize, train, print error.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier()
clf = clf.fit(train_X,train_y)
print(accuracy_score(test_y, clf.predict(test_X)))


I am working on the file "Machine Learning Basics with Sklearn.ipynb".
The first task is:
Splitting the dataset
The dataset needs to be split into a training and test dataset.
This is done to first train our model and then test how good it is on data it has never seen before.
Can you write Python code for it?

test_idx = [0, 50, 100]

train_y = np.delete(iris.target, test_idx)
train_X = np.delete(iris.data, test_idx, axis=0)

test_y = iris.target[test_idx]
test_X = iris.data[test_idx]


I am working on the file "Machine Learning Basics with Sklearn.ipynb".
The first task is:
Visualize the decision tree
Whereas the decision tree is a simple graph we can visualize it quite simple.
Can you write Python code for it?

from sklearn.externals.six import StringIO
import pydot
import matplotlib.image as mpimg

dot_data = StringIO()
tree.export_graphviz(clf, out_file=dot_data, feature_names=iris.feature_names,
        class_names=iris.target_names,
        filled=True, rounded=True, impurity=False)

pydot_graph = pydot.graph_from_dot_data(dot_data.getvalue())
png_str = pydot_graph.create_png(prog='dot')

# treat the dot output string as an image file
sio = StringIO()
sio.write(png_str)
sio.seek(0)
img = mpimg.imread(sio)

# plot the image
f, axes = plt.subplots(1, 1, figsize=(12,12))
imgplot = axes.imshow(img, aspect='equal')
plt.show()


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
Data structures
Can you write Python code for it?

# Lists
numbers = [1, 4, 9, 16, 25]

numbers[:]

numbers[:2]

numbers[2:]

type(numbers)

letters = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
len(letters)

letters[2]

a = [66.25, 333, 333, 1, 1234.5]
a

a.count(333), a.count(66.25), a.count('x')

a.insert(2, -1)
a

a.append(333)
a

a.index(333)

a.remove(333)
a

a.reverse()
a

a.sort()
a

a.pop()

a

# dictionaires
phones = {'Spiderman': 151984858, 'Me': 151234324}

phones['Superman'] = 15104928
phones

phones['Spiderman']

del phones['Me']
phones

phones['Batman'] = 15123545
phones

phones.keys()

'Ken' in phones

# tuples

tuple = 31213, 123453, 'hi Ml!'
tuple

tuple[0]

tuple[2]

tuple[1] = 1234

tupleTheSecond = tuple, (1, 2, 3, 4, 5)
tupleTheSecond

t1, t2 = tupleTheSecond

t1

t2

for i, j in zip (t1, t2):
    print i, j

type(t1)

# sets

basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']

fruit = set(basket)

fruit

'orange' in fruit

'plum' in fruit


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
Histograms - data distributions
Can you write Python code for it?

plt.figure(figsize=(10, 5))
plt.hist(nn['bedrooms'],normed=False)
plt.show()

plt.figure(figsize=(10, 5))
plt.hist(nn['price'],normed=False)
plt.show()

plt.figure(figsize=(10, 5))
plt.hist(nn['sqft_living'],normed=False)
plt.show()

plt.figure(figsize=(10, 5))
plt.hist(nn['sqft_lot'],normed=False)
plt.show()


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
House Prices - Data fields description
Here's a brief version of what you'll find in the data description file.

SalePrice - the property's sale price in dollars. This is the target variable that you're trying to predict.
MSSubClass: The building class
MSZoning: The general zoning classification
LotFrontage: Linear feet of street connected to property
LotArea: Lot size in square feet
Street: Type of road access
Alley: Type of alley access
LotShape: General shape of property
LandContour: Flatness of the property
Utilities: Type of utilities available
LotConfig: Lot configuration
LandSlope: Slope of property
Neighborhood: Physical locations within Ames city limits
Condition1: Proximity to main road or railroad
Condition2: Proximity to main road or railroad (if a second is present)
BldgType: Type of dwelling
HouseStyle: Style of dwelling
OverallQual: Overall material and finish quality
OverallCond: Overall condition rating
YearBuilt: Original construction date
YearRemodAdd: Remodel date
RoofStyle: Type of roof
RoofMatl: Roof material
Exterior1st: Exterior covering on house
Exterior2nd: Exterior covering on house (if more than one material)
MasVnrType: Masonry veneer type
MasVnrArea: Masonry veneer area in square feet
ExterQual: Exterior material quality
ExterCond: Present condition of the material on the exterior
Foundation: Type of foundation
BsmtQual: Height of the basement
BsmtCond: General condition of the basement
BsmtExposure: Walkout or garden level basement walls
BsmtFinType1: Quality of basement finished area
BsmtFinSF1: Type 1 finished square feet
BsmtFinType2: Quality of second finished area (if present)
BsmtFinSF2: Type 2 finished square feet
BsmtUnfSF: Unfinished square feet of basement area
TotalBsmtSF: Total square feet of basement area
Heating: Type of heating
HeatingQC: Heating quality and condition
CentralAir: Central air conditioning
Electrical: Electrical system
1stFlrSF: First Floor square feet
2ndFlrSF: Second floor square feet
LowQualFinSF: Low quality finished square feet (all...
Can you write Python code for it?

# read csv data
data = pd.read_csv('housing_train.csv')
# describe dataset
data.describe()

# show first 5 records in the dataset

data.head()

# show last 5 records in the dataset

data.tail()

# row selection from 10-15 record
dataTemp = data[0:15]

# iteration over rows
for row in dataTemp.iterrows():
    print row[1]['SalePrice']

data['Lambda'] = data['SalePrice'].apply(lambda x: x * 1.1)
dataTemp = data[0:15]
dataTemp[::3]

columns = ['SalePrice', 'LotArea', '1stFlrSF', '2ndFlrSF', 'BedroomAbvGr', 'YrSold']
data = data[columns]

plt.figure(figsize=(10, 5))
plt.hist(data['SalePrice'],normed=False)
plt.show()

plt.figure(figsize=(10, 5))
plt.hist(data['LotArea'],normed=False)
plt.show()

plt.figure(figsize=(10, 5))
plt.hist(data['BedroomAbvGr'],normed=False)
plt.show()

len(data['SalePrice'])

# Data filtering
dataFiltering = data[['SalePrice', 'BedroomAbvGr','LotArea']].copy()

dataFiltering.head()

# Hadling NaN values
original = pd.read_csv('housing_train.csv')
#original.isnull().any()
original.loc[:, original.isnull().any()]

original.dropna(subset=["LotFrontage"])    # option 1

original.drop("LotFrontage", axis=1)       # option 2

median = housing["LotFrontage"].median()
original["LotFrontage"].fillna(median)     # option 3

# Mention ~ operator
count = original[(original["MSZoning"].str.contains('RL'))]
len(count)

# best fit of data
(mu, sigma) = norm.fit(data['SalePrice'])

# the histogram of the data
n, bins, patches = plt.hist(data['SalePrice'], 60, normed=True, facecolor='green', alpha=0.75)

# add a 'best fit' line
y = mlab.normpdf( bins, mu, sigma)
l = plt.plot(bins, y, 'r--', linewidth=2)

#plot
plt.xlabel('Sales prices')
plt.ylabel('Probability')
plt.title(r'$\mathrm{Histogram\ of\ IQ:}\ \mu=%.3f,\ \sigma=%.3f$' %(mu, sigma))
plt.grid(True)

plt.show()

prices = data['SalePrice']
features = data.drop('SalePrice', axis = 1)

# i: index
for i, col in enumerate(features.columns):
    plt.figure(figsize=(20, 35))
    # 3 plots here hence 1, 3
    plt.subplot(5, 1, i+1)
    x...


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
House Sales in King County, USA
Dataset features are selfexplanatory.
Dataset is taken from Kaggle website
Can you write Python code for it?

# read csv file
nn = pd.read_csv('kc_house_data.csv')

# top 5 data records
nn.head(10)

# check are there any null values in any of the columns
nn.isnull().any()

len(nn)

# add one record with NaN values
nn = nn.append({'id':'12345', 'price':'12345.23'}, ignore_index=True)

len(nn)

# check number of NaN values in some column
len(nn[nn.bedrooms.isnull()])

# show list of the records where column bedrooms contain NaN values
nn[nn.bedrooms.isnull()]

# drop NaN values
nn = nn.dropna()

# check number of NaN records after droping NaNs
len(nn[nn.bedrooms.isnull()])

len(nn)

nn.describe()


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
Loops
Can you write Python code for it?

for i in [1, 2, 3, 4, 5]:
    print i

for x in range(5):
    if x == 3:
        continue  # go immediately to the next iteration
    if x == 5:
        break     # quit the loop entirely
    print x

x = 0
while x < 5:
    print x, "is less than 5"
    x += 1

a = True
x = 0
while a:
    print x, "is less than 10"
    x += 1
    if x >= 10:
        a = False


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
Miscellaneous
Can you write Python code for it?

def colorFunction(x):
    if x == 0:
        return 'black'
    elif x == 1:
        return 'brown'
    elif x == 2:
        return 'red'
    elif x == 3:
        return 'blue'
    elif x == 4:
        return 'green'
    elif x == 5:
        return 'pink'
    elif x == 6:
        return 'orange'
    elif x ==7:
        return 'cyan'
    elif x ==8:
        return 'yellow'
    elif x == 9:
        return 'magenta'
    else:
        return 'pink'

nn['color'] = nn['bedrooms'].apply(colorFunction)

figure = plt.figure()
subplot = figure.add_subplot(111)
scatter = subplot.scatter(nn['long'], nn['lat'], s=10, c=nn['color'])
subplot.set_xlabel('Longitude')
subplot.set_ylabel('Latitude')
figure.set_figheight(10)
figure.set_figwidth(15)
plt.show()

features = nn.drop(['id','price','date','color'], axis = 1)

# Using pyplot
plt.figure(figsize=(20, 55))

# i: index
for i, col in enumerate(features.columns):
    # 3 plots here hence 1, 3
    plt.subplot(10, 3, i+1)
    x = nn[col]
    y = nn['price']
    plt.plot(x, y, 'o')
    # Create regression line
    plt.plot(np.unique(x), np.poly1d(np.polyfit(x, y, 1))(np.unique(x)))
    plt.title(col)
    plt.xlabel(col)
    plt.ylabel('prices')
plt.show()

# best fit of data
(mu, sigma) = norm.fit(nn['price'])

# the histogram of the data
n, bins, patches = plt.hist(nn['price'], 60, normed=True, facecolor='green', alpha=0.75)

# add a 'best fit' line
y = mlab.normpdf(bins, mu, sigma)
l = plt.plot(bins, y, 'r--', linewidth=2)

#plot
plt.xlabel('Sales prices')
plt.ylabel('Probability')
plt.title(r'$\mathrm{Histogram\ of\ IQ:}\ \mu=%.3f,\ \sigma=%.3f$' %(mu, sigma))
plt.grid(True)

plt.show()


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
Pandas basics
Can you write Python code for it?

# import panda library
import pandas as pd

# Show version of panda library
print pd.__version__

# it is all about describing the data
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

def randrange(n, vmin, vmax):
    '''
    Helper function to make an array of random numbers having shape (n, )
    with each number distributed Uniform(vmin, vmax).
    '''
    return (vmax - vmin)*np.random.rand(n) + vmin

fig = plt.figure(figsize=(14, 12))
ax = fig.add_subplot(111, projection='3d')

n = 100

# For each set of style and range settings, plot n random points in the box
# defined by x in [23, 32], y in [0, 100], z in [zlow, zhigh].
for c, m, zlow, zhigh in [('r', 'o', -50, -25), ('b', '^', -30, -5)]:
    xs = randrange(n, 23, 32)
    ys = randrange(n, 0, 100)
    zs = randrange(n, zlow, zhigh)
    ax.scatter(xs, ys, zs, c=c, marker=m)

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.show()


I am working on the file "Machine Learning Dojo - Part I.ipynb".
The first task is:
References:

Fast Style Transfer
Tensorflow Tutorials

Python basics
Variables
Can you write Python code for it?

# String
string = 'Machine learning '
string2 = ' dojo '
string3 = ' part I'
print string + string2 + string3
print 'String variable type is: {}'.format(type(string))

# Integers
number = 10
number2 = 20
number3 = 30
print number + number2 + number3
print 'number variable type is: {}'.format(type(number))

# Booleans
boolean = True
boolean2 = True
boolean3 = False
print boolean and boolean2 or boolean3
print 'bolean variable type is: {}'.format(type(boolean))

# Floating point numbers
floating = 3.14
floating2 = 2.79
floating3 = 10.01
print floating + floating2 + floating3
print 'floating variable type is: {}'.format(type(floating))


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_five/.ipynb_checkpoints/Deep Features for Image Retrieval-checkpoint.ipynb".
The first task is:
Just for fun, let's create a lambda to find and show nearest neighbor images
Can you write Python code for it?

show_neighbors = lambda i: get_images_from_ids(knn_model.query(image_train[i:i+1]))['image'].show()

show_neighbors(8)

show_neighbors(26)

image_train['label'].sketch_summary()

graphlab.canvas.set_target('ipynb')
cat = image_test[0:1]
cat['image'].show()

knn_model.query(cat)
knn_model.query(cat)['distance'].mean()

cat_neighbors = get_images_from_ids(knn_model.query(cat))
cat_neighbors['image'].show()

graphlab.canvas.set_target('ipynb')

knn1_model = graphlab.nearest_neighbors.create(image_train,features=['deep_features'],
                                             label='id')
knn1_model.query(cat)['distance'].mean()

dog_neighbors = get_images_from_ids(knn_model.query(cat))
dog_neighbors['image'].show()

train_dog = image_train[image_train['label'=='dog']]
train_cat = image_train[image_train['label'=='cat']]
train_automobile = image_train[image_train['label'=='automobile']]
train_bird = image_train[image_train['label'=='bird']]
knn_dog_model = graphlab.nearest_neighbors.create(train_dog,features=['deep_features'],                                             label='id')

knn_cat_model = graphlab.nearest_neighbors.create(train_cat,features=['deep_features'],                                             label='id')

knn_automobile_model = graphlab.nearest_neighbors.create(train_automobile,features=['deep_features'],                                             label='id')

cat = image_test[0:1]
cat['image'].show()

knn_dog_model.query(cat)
knn_cat_model.query(cat)
knn_automobile_model.query(cat)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_five/.ipynb_checkpoints/Deep Features for Image Retrieval-checkpoint.ipynb".
The first task is:
Load the CIFAR-10 dataset
We will use a popular benchmark dataset in computer vision called CIFAR-10.
(We've reduced the data to just 4 categories = {'cat','bird','automobile','dog'}.)
This dataset is already split into a training set and test set. In this simple retrieval example, there is no notion of "testing", so we will only use the training data.
Can you write Python code for it?

image_train = graphlab.SFrame('image_train_data/')
image_test = graphlab.SFrame('image_test_data/')


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_five/.ipynb_checkpoints/Deep Features for Image Retrieval-checkpoint.ipynb".
The first task is:
Train a nearest-neighbors model for retrieving images using deep features
We will now build a simple image retrieval system that finds the nearest neighbors for any image.
Can you write Python code for it?

knn_model = graphlab.nearest_neighbors.create(image_train,features=['deep_features'],
                                             label='id')


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_five/.ipynb_checkpoints/Deep Features for Image Retrieval-checkpoint.ipynb".
The first task is:
Use image retrieval model with deep features to find similar images
Let's find similar images to this cat picture.
Can you write Python code for it?

graphlab.canvas.set_target('ipynb')
cat = image_train[18:19]
cat['image'].show()

knn_model.query(cat)
knn_model.query(cat)['distance'].mean()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_five/.ipynb_checkpoints/Deep Features for Image Retrieval-checkpoint.ipynb".
The first task is:
Very cool results showing similar cats.
Finding similar images to a car
Can you write Python code for it?

car = image_train[8:9]
car['image'].show()

get_images_from_ids(knn_model.query(car))['image'].show()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_five/.ipynb_checkpoints/Deep Features for Image Retrieval-checkpoint.ipynb".
The first task is:
We are going to create a simple function to view the nearest neighbors to save typing:
Can you write Python code for it?

def get_images_from_ids(query_result):
    return image_train.filter_by(query_result['reference_label'],'id')

cat_neighbors = get_images_from_ids(knn_model.query(cat))

cat_neighbors['image'].show()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
<img src="https://ssl.cdn-redfin.com/photo/1/bigphoto/302/734302_0.jpg">
Can you write Python code for it?

print sqft_model.predict(house2)

print my_features_model.predict(house2)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Bill_gates%27_house.jpg/2560px-Bill_gates%27_house.jpg">
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
Above:  blue dots are original data, green line is the prediction from the simple regression.
Below: we can view the learned regression coefficients.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
Comparing the results of the simple model with adding more features
Can you write Python code for it?

print sqft_model.evaluate(test_data)
print my_features_model.evaluate(test_data)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
Create a simple regression model of sqft_living to price
Split data into training and testing.
We use seed=0 so that everyone running this notebook gets the same results.  In practice, you may set a random seed (or let GraphLab Create pick a random seed for you).
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
Evaluate the simple model
Can you write Python code for it?

print test_data['price'].mean()

print sqft_model.evaluate(test_data)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
In this case, the model with more features provides a better prediction.  This behavior is expected here, because this house is more differentiated by features that go beyond its square feet of living space, especially the fact that it's a waterfront house.
Last house, super fancy
Our last house is a very large one owned by a famous Seattleite.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
In this case, the model with more features provides a worse prediction than the simpler model with only 1 feature.  However, on average, the model with more features is better.
Prediction for a second, fancier house
We will now examine the predictions for a fancier house.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
Pull the bar at the bottom to view more of the data.
98039 is the most expensive zip code.
Build a regression model with more features
Can you write Python code for it?

my_features_model = graphlab.linear_regression.create(train_data,target='price',features=my_features,validation_set=None)

print my_features


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
RMSE of about \$255,170!
Let's show what our predictions look like
Matplotlib is a Python plotting library that is also useful for plotting.  You can install it with:
'pip install matplotlib'
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(test_data['sqft_living'],test_data['price'],'.',
        test_data['sqft_living'],sqft_model.predict(test_data),'-')


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_one/Predicting house prices.ipynb".
The first task is:
The model predicts a price of over $13M for this house! But we expect the house to cost much more.  (There are very few samples in the dataset of houses that are this fancy, so we don't expect the model to capture a perfect prediction here.)
Can you write Python code for it?

sales[sales['zipcode'] == '98039']['price'].mean()

sales[(sales['sqft_living']>=2000) & (sales['sqft_living']<4000)].num_rows()

sales.num_rows()

advance_features = ['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'floors', 'zipcode', 'view', 'sqft_above', 'sqft_basement']

print advance_features

advance_features_model = graphlab.linear_regression.create(train_data,target='price',features=advance_features,validation_set=None)

print my_features_model.evaluate(test_data);
print advance_features_model.evaluate(test_data);


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Applying the nearest-neighbors model for retrieval
Who is closest to Obama?
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
As we can see, president Obama's article is closest to the one about his vice-president Biden, and those of other politicians.
Other examples of document retrieval
Can you write Python code for it?

swift = people[people['name'] == 'Taylor Swift']

knn_model.query(swift)

jolie = people[people['name'] == 'Angelina Jolie']

knn_model.query(jolie)

arnold = people[people['name'] == 'Arnold Schwarzenegger']

knn_model.query(arnold)

elton = people[people['name'] == 'Elton John']
elton['word_count'] = graphlab.text_analytics.count_words(elton['text'])

print elton['word_count']

elton_word_count_table = elton[['word_count']].stack('word_count', new_column_name = ['word','count'])
elton_word_count_table.sort('count',ascending=False)

elton[['tfidf']].stack('tfidf',new_column_name=['word','tfidf']).sort('tfidf',ascending=False)

victoria = people[people['name'] == 'Victoria Beckham']

graphlab.distances.cosine(elton['tfidf'][0],victoria['tfidf'][0])

paul = people[people['name'] == 'Paul McCartney']

graphlab.distances.cosine(elton['tfidf'][0],paul['tfidf'][0])

knn_model.query(elton, k=None).print_rows(num_rows=30)

kwc_model = graphlab.nearest_neighbors.create(people,features=['word_count'],label='name')

kwc_model.query(elton)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Build a nearest neighbor model for document retrieval
We now create a nearest-neighbors model and apply it to document retrieval.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Examine the TF-IDF for the Obama article
Can you write Python code for it?

obama = people[people['name'] == 'Barack Obama']

obama[['tfidf']].stack('tfidf',new_column_name=['word','tfidf']).sort('tfidf',ascending=False)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Explore the dataset and checkout the text it contains
Exploring the entry for president Obama
Can you write Python code for it?

obama = people[people['name'] == 'Barack Obama']

obama

obama['text']


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Exploring the entry for actor George Clooney
Can you write Python code for it?

clooney = people[people['name'] == 'George Clooney']
clooney['text']


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Most common words include uninformative words like "the", "in", "and",...
Compute TF-IDF for the corpus
To give more weight to informative words, we weigh them by their TF-IDF scores.
Can you write Python code for it?

people['word_count'] = graphlab.text_analytics.count_words(people['text'])
people.head()

tfidf = graphlab.text_analytics.tf_idf(people['word_count'])
tfidf

people['tfidf'] = tfidf['docs']


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Sort the word counts for the Obama article
Turning dictonary of word counts into a table
Can you write Python code for it?

obama_word_count_table = obama[['word_count']].stack('word_count', new_column_name = ['word','count'])


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Sorting the word counts to show most common words at the top
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/.ipynb_checkpoints/Document retrieval-checkpoint.ipynb".
The first task is:
Words with highest TF-IDF are much more informative.
Manually compute distances between a few people
Let's manually compare the distances between the articles for a few famous people.
Can you write Python code for it?

clinton = people[people['name'] == 'Bill Clinton']

beckham = people[people['name'] == 'David Beckham']


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Build a nearest neighbor model for document retrieval
We now create a nearest-neighbors model and apply it to document retrieval.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Data contains:  link to wikipedia article, name of person, text of article.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Examine the TF-IDF for the Obama article
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Explore the dataset and checkout the text it contains
Exploring the entry for president Obama
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Exploring the entry for actor George Clooney
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Get the word counts for Obama article
Can you write Python code for it?

obama['word_count'] = graphlab.text_analytics.count_words(obama['text'])

print obama['word_count']


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Is Obama closer to Clinton than to Beckham?
We will use cosine distance, which is given by
(1-cosine_similarity)
and find that the article about president Obama is closer to the one about former president Clinton than that of footballer David Beckham.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Load some text data - from wikipedia, pages on people
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_three/Document retrieval.ipynb".
The first task is:
Words with highest TF-IDF are much more informative.
Manually compute distances between a few people
Let's manually compare the distances between the articles for a few famous people.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Applying the learned model to understand sentiment for Giraffe
Can you write Python code for it?

giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type='probability')

giraffe_reviews.head()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Build a sentiment classifier
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Build the word count vector for each review
Can you write Python code for it?

products['word_count'] = graphlab.text_analytics.count_words(products['review'])

products.head()

graphlab.canvas.set_target('ipynb')

products['name'].show()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Define what's a positive and a negative sentiment
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.  Reviews with a rating of 4 or higher will be considered positive, while the ones with rating of 2 or lower will have a negative sentiment.
Can you write Python code for it?

#ignore all 3* reviews
products = products[products['rating'] != 3]

#positive sentiment = 4* or 5* reviews
products['sentiment'] = products['rating'] >=4

products.head()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Evaluate the sentiment model
Can you write Python code for it?

sentiment_model.evaluate(test_data, metric='roc_curve')

sentiment_model.show(view='Evaluation')


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Let's explore this data together
Data includes the product name, the review text and the rating of the review.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Let's train the sentiment classifier
Can you write Python code for it?

train_data,test_data = products.random_split(.8, seed=0)

sentiment_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=['word_count'],
                                                     validation_set=test_data)


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Most positive reviews for the giraffe
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/.ipynb_checkpoints/Analyzing product sentiment-checkpoint.ipynb".
The first task is:
Read some product review data
Loading reviews for a set of baby products.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Applying the learned model to understand sentiment for Giraffe
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Build a sentiment classifier
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Build the word count vector for each review
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Define what's a positive and a negative sentiment
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.  Reviews with a rating of 4 or higher will be considered positive, while the ones with rating of 2 or lower will have a negative sentiment.
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Evaluate the sentiment model
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Most positive reviews for the giraffe
Can you write Python code for it?


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Show most negative reviews for giraffe
Can you write Python code for it?

giraffe_reviews[-1]['review']

giraffe_reviews[-2]['review']

selected_words = ['awesome', 'great', 'fantastic', 'amazing', 'love', 'horrible', 'bad', 'terrible', 'awful', 'wow', 'hate']

products['word_count'] = graphlab.text_analytics.count_words(products['review'])
selected_words = ['awesome', 'great', 'fantastic', 'amazing', 'love', 'horrible', 'bad', 'terrible', 'awful', 'wow', 'hate']

def awesome_count(cell):
    if 'hate' in cell:
        return cell['hate']
    else:
        return 0
products['hate'] = products['word_count'].apply(awesome_count)
products.head()

train_data,test_data = products.random_split(.8, seed=0)
selected_words = ['awesome', 'great', 'fantastic', 'amazing', 'love', 'horrible', 'bad', 'terrible', 'awful', 'wow', 'hate']
selected_words_model = graphlab.logistic_classifier.create(train_data,target='sentiment',features=selected_words,validation_set=test_data, )
selected_words_model['coefficients'].sort('value', ascending = True)

selected_words_model.evaluate(test_data)

sentiment_model.evaluate(test_data)

giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type='probability')
giraffe_reviews.head()


I am working on the file "Machine Learning Foundations: A Case Study Approach/Assignment_two/Analyzing product sentiment.ipynb".
The first task is:
Sort the reviews based on the predicted sentiment and explore
Can you write Python code for it?

giraffe_reviews = giraffe_reviews.sort('predicted_sentiment', ascending=False)

giraffe_reviews.head()


I am working on the file "Machine Learning MNIST with TF.ipynb".
The first task is:
Classify a few examples
We can make predictions on individual images using the predict method
Can you write Python code for it?

# here's one it gets right
idx = [0]
predictions = classifier.predict(x=np.array(test_data[idx]))
for i, p in enumerate(predictions):
    print("Predicted %d, Label: %d" % (p, test_labels[idx[i]]))
    display(idx[i])

# here's one it gets wrong
idx = [8]
predictions = classifier.predict(x=np.array(test_data[idx]))
for i, p in enumerate(predictions):
    print("Predicted %d, Label: %d" % (p, test_labels[idx[i]]))
    display(idx[i])


I am working on the file "Machine Learning MNIST with TF.ipynb".
The first task is:
Evaluate accuracy
Can you write Python code for it?

classifier.evaluate(test_data, test_labels)
print classifier.evaluate(test_data, test_labels)["accuracy"]


I am working on the file "Machine Learning MNIST with TF.ipynb".
The first task is:
Import the dataset
Can you write Python code for it?

mnist = learn.datasets.load_dataset('mnist')
data = mnist.train.images
labels = np.asarray(mnist.train.labels, dtype=np.int32)
test_data = mnist.test.images
test_labels = np.asarray(mnist.test.labels, dtype=np.int32)


I am working on the file "Machine Learning MNIST with TF.ipynb".
The first task is:
Now let's take a look at how many features we have.
Can you write Python code for it?


I am working on the file "Machine Learning Nanodegree/Building a Student Intervention System/student_intervention.ipynb".
The first task is:
3. Preparing the Data
In this section, we will prepare the data for modeling, training and testing.
Identify feature and target columns
It is often the case that the data you obtain contains non-numeric features. This can be a problem, as most machine learning algorithms expect numeric data to perform computations with.
Let's first separate our data into feature and target columns, and see if any features are non-numeric.<br/>
Note: For this dataset, the last column ('passed') is the target or label we are trying to predict.
Can you write Python code for it?

# Extract feature (X) and target (y) columns
feature_cols = list(student_data.columns[:-1])  # all columns but last are features
target_col = student_data.columns[-1]  # last column is the target/label
print "Feature column(s):-\n{}".format(feature_cols)
print "Target column: {}".format(target_col)

X_all = student_data[feature_cols]  # feature values for all students
y_all = student_data[target_col]  # corresponding targets/labels
print "\nFeature values:-"
print X_all.head()  # print the first 5 rows


I am working on the file "Machine Learning Nanodegree/Building a Student Intervention System/student_intervention.ipynb".
The first task is:
Preprocess feature columns
As you can see, there are several non-numeric columns that need to be converted! Many of them are simply yes/no, e.g. internet. These can be reasonably converted into 1/0 (binary) values.
Other columns, like Mjob and Fjob, have more than two values, and are known as categorical variables. The recommended way to handle such a column is to create as many columns as possible values (e.g. Fjob_teacher, Fjob_other, Fjob_services, etc.), and assign a 1 to one of them and 0 to all others.
These generated columns are sometimes called dummy variables, and we will use the pandas.get_dummies() function to perform this transformation.
Can you write Python code for it?

# Preprocess feature columns
def preprocess_features(X):
    outX = pd.DataFrame(index=X.index)  # output dataframe, initially empty

    # Check each column
    for col, col_data in X.iteritems():
        # If data type is non-numeric, try to replace all yes/no values with 1/0
        if col_data.dtype == object:
            col_data = col_data.replace(['yes', 'no'], [1, 0])
        # Note: This should change the data type for yes/no columns to int

        # If still non-numeric, convert to one or more dummy variables
        if col_data.dtype == object:
            col_data = pd.get_dummies(col_data, prefix=col)  # e.g. 'school' => 'school_GP', 'school_MS'

        outX = outX.join(col_data)  # collect column(s) in output dataframe

    return outX

X_all = preprocess_features(X_all)
print "Processed feature columns ({}):-\n{}".format(len(X_all.columns), list(X_all.columns))


I am working on the file "Machine Learning Nanodegree/Building a Student Intervention System/student_intervention.ipynb".
The first task is:
Split data into training and test sets
So far, we have converted all categorical features into numeric values. In this next step, we split the data (both features and corresponding labels) into training and test sets.
Can you write Python code for it?

# First, decide how many training vs test samples you want
num_all = student_data.shape[0]  # same as len(student_data)
num_train = 300  # about 75% of the data
num_test = num_all - num_train

# TODO: Then, select features (X) and corresponding labels (y) for the training and test sets
# Note: Shuffle the data or randomly select samples to avoid any bias due to ordering in the dataset
from sklearn import cross_validation

X_train,X_test,y_train,y_test=cross_validation.train_test_split(X_all,y_all,train_size=num_train,random_state=100)
print "Training set: {} samples".format(X_train.shape[0])
print "Test set: {} samples".format(X_test.shape[0])
# Note: If you need a validation set, extract it from within training data


I am working on the file "Machine Learning Pipeline Example.ipynb".
The first task is:
But manually chaining this steps together makes it difficult to experiment. Instead we can put it together in a pipeline:
Can you write Python code for it?

pipeline_and_model = Pipeline().setStages([assembler, indexer, dt])
pipeline_model = pipeline_and_model.fit(df)

dt_model.transform(prepared).select("prediction", "category-index").take(20)

pipeline_model.transform(df).select("prediction", "category-index").take(20)


I am working on the file "Machine Learning Pipeline Example.ipynb".
The first task is:
Now the pipeline only works on doubles, so we need to take our category and turn it into a double. The StringIndexer will do this for us:
Can you write Python code for it?

indexer = StringIndexer(inputCol="category").setOutputCol("category-index")

pipeline = Pipeline().setStages([assembler, indexer])


I am working on the file "Machine Learning Pipeline Example.ipynb".
The first task is:
Now we start by downloading loading some data which is in csv format so its a good thing we got that csv package included already for us.
Note: the data is a modified version of https://archive.ics.uci.edu/ml/datasets/Adult
Can you write Python code for it?

# If your running this on a cluster you will need to copy the data file into HDFS or whatever cluster file system
# you are using
df = sqlContext.read.format("csv").option("header", "true").load("resources/adult.data")

df.cache()

df.head()


I am working on the file "Machine Learning Pipeline Example.ipynb".
The first task is:
We then transform our data into the prepaired format for our machine learning model to work on:
Can you write Python code for it?

prepared = model.transform(df)

prepared.head()

dt = DecisionTreeClassifier(labelCol = "category-index", featuresCol="feautres")


I am working on the file "Machine Learning Retraining Image Classifier with TF.ipynb".
The first task is:
Getting the dataset and setting things up
First let's download the dataset.
Can you write Python code for it?

cmd = '''mkdir tf_files && \
cd tf_files && \
curl -O http://download.tensorflow.org/example_images/flower_photos.tgz && \
tar xzf flower_photos.tgz && \
rm flower_photos.tgz && \
cd ..'''

os.system(cmd)


I am working on the file "Machine Learning Retraining Image Classifier with TF.ipynb".
The first task is:
Now retrain on the dataset
This is done by calling the tensorflow script provided in the examples folder.
Can you write Python code for it?

cmd = '''cd ../tensorflow && \
python tensorflow/examples/image_retraining/retrain.py \
	--bottleneck_dir=../ml-notebooks/tf_files/bottlenecks \
	--how_many_training_steps 500 \
	--model_dir=../ml-notebooks/tf_files/inception \
	--output_graph=../ml-notebooks/tf_files/retrained_graph.pb \
	--output_labels=../ml-notebooks/tf_files/retrained_labels.txt \
	--image_dir ../ml-notebooks/tf_files/flower_photos'''
os.system(cmd)


I am working on the file "Machine Learning Retraining Image Classifier with TF.ipynb".
The first task is:
Using the model
Now let's use the trained model to evaluate it on an image of a rose.
First let's see if our rose actually looks like a rose.
Can you write Python code for it?

image_path = "rose.jpg"

from IPython.display import Image
Image(filename=image_path)


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
A couple of different hypotheses from this model:
Can you write Python code for it?

def h0(x) : return h(x, [0, 0.5, -0.5])
def h1(x) : return h(x, [1, 1, 2.0])
def h2(x) : return h(x, [-1, 2, 1])

for hx in [h0, h1, h2] : plot_problem(X, y, hx, surfaces=False)


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
A linear classifier divides the input space into two half-spaces
$h(\mathbf{x}) \geq 0$ - instances labeled as $y=1$
$h(\mathbf{x})=0$ - the boundary
$h(\mathbf{x}) \leq 0$ - instances labeled as $y=0$
Can you write Python code for it?


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Algo 3: Decision tree
Can you write Python code for it?

from sklearn import tree
h = tree.DecisionTreeClassifier()
h.fit(X, y)
plot_problem(X, y, h.predict)

from sklearn.externals.six import StringIO
import pyparsing
import pydot
from IPython.display import Image
dot_data = StringIO()
tree.export_graphviz(h, out_file=dot_data)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
img = Image(graph.create_png())

img.width=300; img

X, y = make_classification(n_samples=100, n_features=2, n_classes=2, n_redundant=0, n_clusters_per_class=2, random_state=54)
plot_problem(X, y)

h = tree.DecisionTreeClassifier()
h.fit(X, y)
plot_problem(X, y, h.predict)

misclassification_error(h.predict, X, y)

X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3)

h.fit(X_train, y_train)
plot_problem(X_train, y_train, h.predict)

misclassification_error(h.predict, X_train, y_train)

misclassification_error(h.predict, X_test, y_test)

dot_data = StringIO()
tree.export_graphviz(h, out_file=dot_data)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
img = Image(graph.create_png())
img.width=500; img

h = tree.DecisionTreeClassifier(max_depth=3)
h.fit(X_train, y_train);

misclassification_error(h.predict, X_train, y_train)

misclassification_error(h.predict, X_test, y_test)


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Algo 4: Naive Bayes classifier

The above three models are discriminative

Naive Bayes (NB) is a generative model. This means that it not only computes the boundary between the classes, but also the joint probability distribution of instances and their labels, $P(\mathbf{x},y)$

Uses the Bayes' rule:
$$
P(y=j|\mathbf{x}) =
\frac{P(\mathbf{x},y=j)}{P(\mathbf{x})} =
\frac{p(\mathbf{x}|y=j) P(y=j)}{p(\mathbf{x})} =
\frac{p(\mathbf{x}|y=j)P(y=j)}{\sum_{k=1}^K p(\mathbf{x}|y=k)P(y=k)}
$$

(1) The model:
$$
h(\mathbf{x}) = \mathrm{argmax}_{j}\ p(\mathbf{x}|y=j) P(y=j)
$$

Or, if we want probabilities for each class:

$$h_j(\mathbf{x})=P(y=j|\mathbf{x})$$

We introduce the "naive assumption" that, given a class, all features are mutually independent. The model simplifies to:

$$
h(\mathbf{x}) = \mathrm{argmax}j\ P(y=j)\prod{k=1}^n P(x_k|y=j)
$$

(2) Error function:

NB seeks to maximize the likelihood of the training data under the model (parameters):

$$
\mathcal{L}(\boldsymbol{\theta} | \mathcal{D}) \equiv
 p(\mathcal{D} | \boldsymbol{\theta}) =
 p\big((\mathbf{x}^{(1)},y^{(1)}),\dots,(\mathbf{x}^{(N),y^{(N)}}) | \boldsymbol{\theta}\big) =
\prod_{i=1}^N p(\mathbf{x}^{(i)}, y^{(i)} | \boldsymbol{\theta})\
$$

In other words: it chooses the parameters $\boldsymbol\theta$ that make our data most probable

(3) Optimization procedure:

Parameter optimization in this case is amounts to parameter estimation from the dataset

We need to estimate $P(\mathbf{x}|y)$ (likelihood) and $P(y)$ (prior) for each class $y$

We can use a maximum likelihood estimator (MLE), a maximum aposteriori estimator (MAP) or a bayesian estimator

The latter two account for data sparsity and avoid overfitting

MLE is the simplest: simply compute probabilities as the relative frequencies
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB
h = GaussianNB()
h.fit(X_train, y_train)
plot_problem(X_train, y_train, h.predict)


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Feature mapping

Logistic regression is essentially a linear model. What if our problem is not linearly separable?

In logistic regression (and many other algorithms), we can map our instances into a higher dimensional space, where they hopefully will become linearly separable

We do this using a feature mapping function

$$\phi:\mathbb{R}^n\to\mathbb{R}^m$$

For example, mapping from a 2-dimensional input space to a 3-dimensional feature space (excluding the dummy feature $x_0$):

$$\phi(\mathbf{x}) = (1,x_1,x_2,x_1 x_2)$$

[Example: 2-d XOR problem]

A linear boundary in high-dimensional space gives as a non-linear boundary in the original, low-dimensional space

We get non-linearity without having to change the model!
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(4)
X2 = poly.fit_transform(X)

h = LogisticRegression()
h.fit(X2, y)
plot_problem(X, y, lambda x: h.predict(poly.transform(x)))


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Hyperparameters

We know that we have to choose a model $\mathcal{H}$, otherwise learning is futile

Often times we choose a model from within a family of models:

$$
{\mathcal{H}_1, \mathcal{H}_2, \dots, \mathcal{H}_k}
$$

E.g., in logistic regression, we can choose whether we want to use a feature mapping function, and of what kind

What mapping to use defines the "amount of non-linearity", and it can be considered a hyperparameter of the model

The "amount of non-linearity" can be considered a hyperparameter of the model

Let's call this parameter $C$. The larger $C$, the more non-linearity we get

$C$ is a hyperparameter of the model, whereas $w_i$ are just "ordinary" parameters

Example: Logistic regression
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures

for m in [1, 2, 3, 4, 5]:
    poly = PolynomialFeatures(m)
    X2 = poly.fit_transform(X)
    h = LogisticRegression()
    h.fit(X2, y)
    plot_problem(X, y, lambda x: h.predict(poly.transform(x)))
    error = misclassification_error(lambda x : h.predict(poly.transform(x)), X, y)
    plt.title('$m = %d, E(h|\mathcal{D})=%.2f$' % (m, error))
    plt.show()


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Hypothesis

Hypothesis: $h : \mathcal{X} \to \mathcal{Y}$

Assigns class labels to each instance from the instance space

Binary classification: $h : \mathcal{X} \to {0, 1}$ or $h : \mathcal{X} \to {-1, +1}$

More generally, a hypothesis is a function  $h(\mathbf{x} | \boldsymbol\theta)$, defined up to a vector of parameters $\boldsymbol\theta$

For example, a linear classifier:

$$\boldsymbol\theta = (w_0, w_1, w_2)$$
$$h(x_1,x_2|w_0,w_1,w_2) = \mathbf{1}{w_0 + w_1 x_1 + w_2 x_2 \geq 0}$$
Can you write Python code for it?


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Kernel trick

Similarly to logistic regression, SVM maps instances to high-dimensional spaces to achieve non-linearity

However, unlike in logistic regression, the mapping need not really take place: SVM can compute the boundary without actually doing the mapping, using a kernel function

Different kernel functions give different non-linearities

Two extremes: linear kernel (equivalent to not using a mapping at all) and a radial basis function (RBF) kernel

RBF kernel effectively maps to an infinite-dimension space, in which each problem becomes linearly separable
Can you write Python code for it?

X, y = make_classification(n_samples=20, n_features=2, n_classes=2, n_redundant=0, n_clusters_per_class=1, random_state=42)
plot_problem(X, y)

h = SVC(kernel='rbf', C=1)
h.fit(X, y)
plot_problem(X, y, h.predict)

h = SVC(kernel='rbf', C=10^5)
h.fit(X, y)
plot_problem(X, y, h.predict)


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
Regularization

Instead of trying out different feature mappings, a nice trick is simply to map to a rather high dimension, regardless of whether we really need it, but then give preference to simpler models

That is, we allow the model to become complex if the data calls for it, but we put some pressure on it to stay as simple as possible

We do this by tweaking the error function so that it penalizes models that become shamelessly non-linear

You can think of this as a spandex suit: it does stretch as required, but it gets uncomfortable the more it stretches

This is called regularization

The error function trades off between empirical error and model complexity

The trade-off is regulated by a regularization factor $C$

If we care more about simplicity, we set $C$ to a small value, otherwise we set it to a large value and get a complex model

Note that $C$ is again a hyperparameter. We are responsible for defining its value

This simplifies things a bit, because we don't have to care about feature mapping, but still we have to choose $C$, which means we still have to do model selection

Cross-validation + model selection

Cross-validation becomes a bit more complex when we also wish to also optimize the hyperparameters of the model (which is what we want to do in most cases)

We are not allowed to optimize the hyperparameters on the test set. If we do this, we ruin our experiment, as we  cannot claim that test instances were not seen before by the model

We need one additional set: validation set

We partition our dataset into three disjoint sets:
    $$
        \mathcal{D} = \mathcal{D}{\mathrm{train}} \cup \mathcal{D}{\mathrm{val}} \cup \mathcal{D}{\mathrm{test}}
    $$
    $$
        \mathcal{D}{\mathrm{train}} \cap \mathcal{D}{\mathrm{val}} =
        \mathcal{D}{\mathrm{train}} \cap \mathcal{D}{\mathrm{test}} =
        \mathcal{D}{\mathrm{val}} \cap \mathcal{D}_{\mathrm{test}} = \emptyset
    $$

Folded cross-validation

The problem with train-test split is that it is a single and...
Can you write Python code for it?

from sklearn.svm import SVC

X, y = make_classification(n_samples=20, n_features=2, n_classes=2, n_redundant=0, n_clusters_per_class=1, random_state=41)
plot_problem(X, y)

h = SVC(kernel='linear')
h.fit(X, y)
plot_problem(X, y, h.predict)

h = LogisticRegression()
h.fit(X, y)
plot_problem(X, y, h.predict)


I am working on the file "Machine Learning Tutorial.ipynb".
The first task is:
To get binary decisions, we simply use a threshold of 0.5

(2) Error function (cross-entropy error):

$$
E(\mathbf{w}|\mathcal{D}) = \frac{1}{N} \sum_{i=1}^N\Big( - y^{(i)} \ln h(\mathbf{x}^{(i)}|\mathbf{w})- (1-y^{(i)})\ln \big(1-h(\mathbf{x}^{(i)}|\mathbf{w})\big)\Big)
$$

Seems convoluted at first, but the idea is actually very simple:
if $y=1$, we want the probability $h(x)$ to be close to 1
if $y=0$, we want the probability $h(x)$ to be close to 0

the more we depart from this, the more we penalize the model

(3) Optimization:

Stochastic gradient descent (SGD)

An iterative fine tuning of the weights $\mathbf{w}$ in search of the point of minimal error

Very efficient (fast) and guaranteed to find the best weights (best hypothesis within the model)

In sklearn, all of the above is packed into a single class:
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

h = LogisticRegression()
h.fit(X, y)
plot_problem(X, y, h.predict)

h.coef_

h.intercept_

misclassification_error(h.predict, X, y)


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Applying the learned model to understand sentiment for Giraffe
Can you write Python code for it?

giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type='probability')

giraffe_reviews.head()


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Build a sentiment classifier
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Build the word count vector for each review
Can you write Python code for it?

products['word_count'] = graphlab.text_analytics.count_words(products['review'])

products.head()

graphlab.canvas.set_target('ipynb')

products['name'].show()


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Evaluate the sentiment model
Can you write Python code for it?

sentiment_model.evaluate(test_data, metric='roc_curve')

sentiment_model.show(view='Evaluation')


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Exploring the data
Data includes the product name, the review text and the rating of the review.
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Most positive reviews for the giraffe
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Show most negative reviews for giraffe
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Analyzing Product Sentiment using GraphLab Create.ipynb".
The first task is:
Sort the reviews based on the predicted sentiment and explore
Can you write Python code for it?

giraffe_reviews = giraffe_reviews.sort('predicted_sentiment', ascending=False)

giraffe_reviews.head()


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Build a nearest neighbor model for document retrieval
We now create a nearest-neighbors model and apply it to document retrieval.
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Data contains:  link to wikipedia article, name of person, text of article.
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Explore the dataset and checkout the text it contains
Exploring the entry for president Obama
Can you write Python code for it?

obama = people[people['name'] == 'Barack Obama']

obama

obama['text']


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Exploring the entry for actor George Clooney
Can you write Python code for it?

clooney = people[people['name'] == 'George Clooney']
clooney['text']


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Get the word counts for Obama article
Can you write Python code for it?

obama['word_count'] = graphlab.text_analytics.count_words(obama['text'])

print obama['word_count']


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Load some text data - from wikipedia, pages on people
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Most common words include uninformative words like "the", "in", "and",...
Compute TF-IDF for the corpus
To give more weight to informative words, we weigh them by their TF-IDF scores.
Can you write Python code for it?

people['word_count'] = graphlab.text_analytics.count_words(people['text'])
people.head()

tfidf = graphlab.text_analytics.tf_idf(people['word_count'])

# Earlier versions of GraphLab Create returned an SFrame rather than a single SArray
# This notebook was created using Graphlab Create version 1.7.1
if graphlab.version <= '1.6.1':
    tfidf = tfidf['docs']

tfidf

people['tfidf'] = tfidf


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Sort the word counts for the Obama article
Turning dictonary of word counts into a table
Can you write Python code for it?

obama_word_count_table = obama[['word_count']].stack('word_count', new_column_name = ['word','count'])


I am working on the file "Machine Learning using GraphLab/Document Retrieval using GraphLab Create.ipynb".
The first task is:
Sorting the word counts to show most common words at the top
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Predicting House Prices using GraphLab Create.ipynb".
The first task is:
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Bill_gates%27_house.jpg/2560px-Bill_gates%27_house.jpg">
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Predicting House Prices using GraphLab Create.ipynb".
The first task is:
Build the regression model using only sqft_living as a feature
Can you write Python code for it?

sqft_model = graphlab.linear_regression.create(train_data, target='price', features=['sqft_living'],validation_set=None)


I am working on the file "Machine Learning using GraphLab/Predicting House Prices using GraphLab Create.ipynb".
The first task is:
Create a simple regression model of sqft_living to price
Split data into training and testing.
We use seed=0 so that everyone running this notebook gets the same results.  In practice, you may set a random seed (or let GraphLab Create pick a random seed for you).
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Predicting House Prices using GraphLab Create.ipynb".
The first task is:
Exploring the data for housing sales
The house price is correlated with the number of square feet of living space.
Can you write Python code for it?

graphlab.canvas.set_target('ipynb')
sales.show(view="Scatter Plot", x="sqft_living", y="price")


I am working on the file "Machine Learning using GraphLab/Predicting House Prices using GraphLab Create.ipynb".
The first task is:
In this case, the model with more features provides a better prediction.  This behavior is expected here, because this house is more differentiated by features that go beyond its square feet of living space, especially the fact that it's a waterfront house.
Last house, super fancy
Our last house is a very large one owned by a famous Seattleite.
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Predicting House Prices using GraphLab Create.ipynb".
The first task is:
RMSE of about \$255,170!
Let's show what our predictions look like
Matplotlib is a Python plotting library that is also useful for plotting.  You can install it with:
'pip install matplotlib'
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(test_data['sqft_living'],test_data['price'],'.',
        test_data['sqft_living'],sqft_model.predict(test_data),'-')


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
As we can see it returns 2968 frequent itemsets of varying lengths. You'll notice
that the number of itemsets grows as the length increases before it shrinks. It grows
because of the increasing number of possible rules. After a while, the large number
of combinations no longer has the support necessary to be considered frequent.
This results in the number shrinking. This shrinking is the benefit of the Apriori
algorithm. If we search all possible itemsets (not just the supersets of frequent ones),
we would be searching thousands of times more itemsets to see if they are frequent.
Extracting association rules
After the Apriori algorithm has completed, we have a list of frequent itemsets.
These aren't exactly association rules, but they are similar to it. A frequent itemset
is a set of items with a minimum support, while an association rule has a premise
and a conclusion.
We can make an association rule from a frequent itemset by taking one of the movies
in the itemset and denoting it as the conclusion. The other movies in the itemset will
be the premise. This will form rules of the following form: if a reviewer recommends all
of the movies in the premise, they will also recommend the conclusion.
For each itemset, we can generate a number of association rules by setting each
movie to be the conclusion and the remaining movies as the premise.
In code, we first generate a list of all of the rules from each of the frequent itemsets,
by iterating over each of the discovered frequent itemsets of each length. We then iterate over every movie in this itemset, using it as our conclusion.
The remaining movies in the itemset are the premise. We save the premise and
conclusion as our candidate rule. This returns a very large number of candidate rules. We can see some by printing
out the first few rules in the list.
Can you write Python code for it?

# Now we create the association rules. First, they are candidates until the confidence has been tested
candidate_rules = []
for itemset_length, itemset_counts in frequent_itemsets.items():
    for itemset in itemset_counts.keys():
        for conclusion in itemset:
            premise = itemset - set((conclusion,))
            candidate_rules.append((premise, conclusion))
print("There are {} candidate rules".format(len(candidate_rules)))

print(candidate_rules[:5])


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
Now we can print the top five rules by sorting this confidence dictionary and
printing the results:
Can you write Python code for it?

from operator import itemgetter
sorted_confidence = sorted(rule_confidence.items(), key=itemgetter(1), reverse=True)

for index in range(5):
    print("Rule #{0}".format(index + 1))
    (premise, conclusion) = sorted_confidence[index][0]
    print("Rule: If a person recommends {0} they will also recommend {1}".format(premise, conclusion))
    print(" - Confidence: {0:.3f}".format(rule_confidence[(premise, conclusion)]))
    print("")


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
Sparse data formats:
This dataset is in a sparse format. Each row can be thought of as a cell in a large
feature matrix of the type used in previous chapters, where rows are users and
columns are individual movies. The first column would be each user's review
of the first movie, the second column would be each user's review of the second
movie, and so on.
There are 1,000 users and 1,700 movies in this dataset, which means that the full
matrix would be quite large. We may run into issues storing the whole matrix in
memory and computing on it would be troublesome. However, this matrix has the
property that most cells are empty, that is, there is no review for most movies for
most users. There is no review of movie #675 for user #213 though, and not for most
other combinations of user and movie.
Can you write Python code for it?

# As you can see, there are no review for most movies, such as #213
all_ratings[all_ratings["UserID"] == 675].sort("MovieID")


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
The Apriori algorithm  revisited
The Apriori algorithm is part of our affinity analysis and deals specifically with
finding frequent itemsets within the data. The basic procedure of Apriori builds
up new candidate itemsets from previously discovered frequent itemsets. These
candidates are tested to see if they are frequent, and then the algorithm iterates as
explained here:

Create initial frequent itemsets by placing each item in its own itemset. Only items with at least the minimum support are used in this step.
New candidate itemsets are created from the most recently discovered frequent itemsets by finding supersets of the existing frequent itemsets.
All candidate itemsets are tested to see if they are frequent. If a candidate is not frequent then it is discarded. If there are no new frequent itemsets from this step, go to the last step.
Store the newly discovered frequent itemsets and go to the second step.
Return all of the discovered frequent itemsets.

Implementation
On the first iteration of Apriori, the newly discovered itemsets will have a length
of 2, as they will be supersets of the initial itemsets created in the first step. On the
second iteration (after applying the fourth step), the newly discovered itemsets will
have a length of 3. This allows us to quickly identify the newly discovered itemsets,
as needed in second step.
We can store our discovered frequent itemsets in a dictionary, where the key is the
length of the itemsets. This allows us to quickly access the itemsets of a given length,
and therefore the most recently discovered frequent itemsets, with the help of the
following code:
Can you write Python code for it?

frequent_itemsets = {}  # itemsets are sorted by length


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
The format given here represents the full matrix, but in a more compact way.
The first row indicates that user #196 reviewed movie #242, giving it a ranking
of 3 (out of five) on the December 4, 1997.
Any combination of user and movie that isn't in this database is assumed to not exist.
This saves significant space, as opposed to storing a bunch of zeroes in memory. This
type of format is called a sparse matrix format. As a rule of thumb, if you expect
about 60 percent or more of your dataset to be empty or zero, a sparse format will
take less space to store.
When computing on sparse matrices, the focus isn't usually on the data we don't
have—comparing all of the zeroes. We usually focus on the data we have and
compare those.
The Apriori implementation
The goal of this chapter is to produce rules of the following form: if a person
recommends these movies, they will also recommend this movie. We will also discuss
extensions where a person recommends a set of movies is likely to recommend
another particular movie.
To do this, we first need to determine if a person recommends a movie. We can
do this by creating a new feature Favorable, which is True if the person gave a
favorable review to a movie:
Can you write Python code for it?

# Not all reviews are favourable! Our goal is "other recommended books", so we only want favourable reviews
all_ratings["Favorable"] = all_ratings["Rating"] > 3
all_ratings[10:15]

all_ratings[all_ratings["UserID"] == 1][:5]


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
These rules were returned as the resulting output.
In these rules, the first part (the frozenset) is the list of movies in the premise,
while the number after it is the conclusion. In the first case, if a reviewer
recommends movie 50, they are also likely to recommend movie 64.
Next, we compute the confidence of each of these rules. The process starts by creating dictionaries to store how many times we see the
premise leading to the conclusion (a correct example of the rule) and how many
times it doesn't (an incorrect example). We iterate over all of the users, their favorable reviews, and over each candidate
association rule. We then test to see if the premise is applicable to this user. In other words, did the
user favorably review all of the movies in the premise? If the premise applies, we see if the conclusion movie was also rated favorably.
If so, the rule is correct in this instance. If not, it is incorrect. We then compute the confidence for each rule by dividing the correct count by the
total number of times the rule was seen.
Can you write Python code for it?

# Now, we compute the confidence of each of these rules. This is very similar to what we did in chapter 1
correct_counts = defaultdict(int)
incorrect_counts = defaultdict(int)
for user, reviews in favorable_reviews_by_users.items():
    for candidate_rule in candidate_rules:
        premise, conclusion = candidate_rule
        if premise.issubset(reviews):
            if conclusion in reviews:
                correct_counts[candidate_rule] += 1
            else:
                incorrect_counts[candidate_rule] += 1
rule_confidence = {candidate_rule:
                   correct_counts[candidate_rule] / float(correct_counts[candidate_rule] + incorrect_counts[candidate_rule])
                    for candidate_rule in candidate_rules}

# Choose only rules above a minimum confidence level
min_confidence = 0.9

# Filter out the rules with poor confidence
rule_confidence = {rule: confidence for rule, confidence in rule_confidence.items() if confidence > min_confidence}
print(len(rule_confidence))


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
This code may take a few minutes to run.
Can you write Python code for it?

print("Found a total of {0} frequent itemsets".format(sum(len(itemsets) for itemsets in frequent_itemsets.values())))


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
We also need to define the minimum support needed for an itemset to be considered frequent. This value is chosen based on the dataset but feel free to try different
values. I recommend only changing it by 10 percent at a time though, as the time the
algorithm takes to run will be significantly different! Let's apply minimum support:
Can you write Python code for it?


I am working on the file "Machine Learning using GraphLab/Recommender Systems using Affinity Analysis.ipynb".
The first task is:
We will sample our dataset to form a training dataset. This also helps reduce
the size of the dataset that will be searched, making the Apriori algorithm run faster.
We obtain all reviews from the first 200 users:
Can you write Python code for it?

# Sample the dataset. You can try increasing the size of the sample, but the run time will be considerably longer
ratings = all_ratings[all_ratings['UserID'].isin(range(200))]  # & ratings["UserID"].isin(range(100))]


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 2/Predicting House Prices.ipynb".
The first task is:
Build a regression model with more features
Can you write Python code for it?

train_with_dummies = pd.get_dummies(train_data[my_features])
test_with_dummies = pd.get_dummies(test_data[my_features])
train_with_dummies.head()

my_features_model = linear_model.LinearRegression()
my_features_model.fit(X = train_with_dummies, y = train_Y)

print(evaluate(sqft_model, test_X, test_Y))
print(evaluate(my_features_model, test_with_dummies, test_Y))


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 2/Predicting House Prices.ipynb".
The first task is:
Create simple regression model of size_sqft to price
Can you write Python code for it?

# Use the train_test_split function from scikit-learn (imported at the top)
# Here we have to give the size of the test (instead of the size of
# the train data in the lecture). random_state is the seed
train_data, test_data = train_test_split(sales, test_size=0.2, random_state=42)

print(len(train_data), len(test_data))


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 2/Predicting House Prices.ipynb".
The first task is:
Evaluate the simple model
Can you write Python code for it?

print(test_data['price'].mean())

# Define a helper function to assess model performance
def evaluate(model, test_x, test_y):
    errors = np.absolute(model.predict(test_x) - test_y)
    return {'rmse': sqrt(np.mean(errors ** 2)), 'max_error' : errors.max()}

test_X = np.reshape(test_data['sqft_living'], (-1, 1))
test_Y = test_data['price']
evaluate(sqft_model, test_X, test_Y)


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 2/Predicting House Prices.ipynb".
The first task is:
Exploring the data for housing sales
Can you write Python code for it?

sales.plot(kind = 'scatter', x='sqft_living', y='price')


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 2/Predicting House Prices.ipynb".
The first task is:
Let's show what our predictions look like
Can you write Python code for it?

plt.plot(test_data['sqft_living'], test_data['price'], '.',
        test_data['sqft_living'], sqft_model.predict(test_X), '-')

print(sqft_model.coef_[0], sqft_model.intercept_)


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 2/Predicting House Prices.ipynb".
The first task is:
The import into pandas looks correct, but the date is not formatted correctly and the zipcode is interpreted as a number rather than a category. Let's fix this.
Can you write Python code for it?

sales['date'] = pd.to_datetime(sales['date'])
sales['zipcode'] = sales['zipcode'].astype('category')

sales.head(10)

len(sales)


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 3/Analyzing product sentiment.ipynb".
The first task is:
Applying the learned model to understand sentiment for Giraffe
Can you write Python code for it?

giraffe_reviews['predicted_sentiment'] = sentiment_model.predict_proba(cv.transform(giraffe_reviews['review']))[:,1]

giraffe_reviews.head()


I am working on the file "Machine Learning/1. Foundations A Case Study Approach/Week 3/Analyzing product sentiment.ipynb".
The first task is:
Build a sentiment classifier
Define what's a positive and negative review
Can you write Python code for it?

# Ignore all 3* review
products = products[products['rating'] != 3]

products['sentiment'] = products['rating'] >= 4

products.head()