I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
The MSE keeps going down adding features, there is a great gain after the 11th feature is added.
3 - Do the same as in part (2) instead this time using recursive feature selection.
Can you write Python code for it?

from sklearn.feature_selection import RFE

mse = []
# looping through the number of features desired and storing the results in mse
for k in range(1, boston_df.shape[1]+1):
    # using Recursive Feature Selection with linear regression as estimator
    sel = RFE(estimator=lr, n_features_to_select=k)
    # fitting the selector
    sel.fit(Xtrain, ytrain)
    # transforming train and test sets
    Xtrain_k = sel.transform(Xtrain)
    Xtest_k = sel.transform(Xtest)
    # fitting linear regression model and printing out the k best features
    lr.fit(Xtrain_k, ytrain)
    print('Top {} features {}'.format(k, pd.Series(sel.support_, index=Xtrain.columns).\
                                      sort_values(ascending=False).\
                                      head(k).index.values))
    mse.append(mean_squared_error(lr.predict(Xtest_k), ytest))

plt.figure(figsize=(16, 8))
plt.plot(range(1, len(mse)+1), mse)
plt.title('MSE for models with different number of features')
plt.xlabel('Number of Features')
plt.ylabel('MSE');


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
The accuracy is way better for a normalized or standardized dataset, with the least having a slightly better generalization: K-Nearest Neighbors is sensitive to feature scaling.
3 - Fit a Naive Bayes model to the data, using a normalized data set, a standardized data set, and the original.  Comment on your results.
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
The median:
Can you write Python code for it?

imp = Imputer(strategy='median')
pm2_regr_median = imp.fit_transform(pm2_regr)

lr.fit(pm2_regr_median[:, :-1], pm2_regr_median[:, -1])
lr.score(pm2_regr_median[:, :-1], pm2_regr_median[:, -1])


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
The results are a bit better than before, but the performances are still very bad.
3 - Create a new encoding for days in which it rained, snowed, neither, and both, and then rerun the regression.  Are the results any better?
Can you write Python code for it?

# hours_snow and hours_rain are cumulative across days, so I'm taking the max for each day to see if it snowed
days = pm2_enc.groupby(['year', 'month', 'day'])['hours_snow', 'hours_rain'].max()
# creating columns for the encodings
days['snow'] = pd.Series(days['hours_snow'] > 0, dtype='int')
days['rain'] = pd.Series(days['hours_rain'] > 0, dtype='int')
days['rain_snow'] = pd.Series((days['hours_rain'] > 0) & (days['hours_snow'] > 0), dtype='int')
days['no_rain_snow'] = pd.Series((days['hours_rain'] == 0) & (days['hours_snow'] == 0), dtype='int')
# resetting index and dropping hours_snow and hours_rain
days.reset_index(inplace=True)
days.drop(['hours_snow', 'hours_rain'], inplace=True, axis=1)

# joining the dataframe with the new columns to the original one
pm2_enc = pm2_enc.merge(days, left_on=['year', 'month', 'day'], right_on=['year', 'month', 'day'])

pm2_enc.info()

lr.fit(pm2_enc.iloc[:, :-1], pm2_enc.iloc[:, -1])
lr.score(pm2_enc.iloc[:, :-1], pm2_enc.iloc[:, -1])


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
The variable is nominal, so I'm going to use one-hot encoding:
Can you write Python code for it?

# for simplicity I'm using pandas function
pm2_enc = pd.get_dummies(pm2)

pm2_enc = pm2_enc.loc[:, regr_cols[:-1] + ['wind_dir_NE', 'wind_dir_NW', 'wind_dir_SE', 'wind_dir_cv'] + regr_cols[-1:]].dropna()

# from solutions using sklearn:
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

l_enc = LabelEncoder()
oh_enc = OneHotEncoder(sparse=False)

# change categorical data labels to integers
data_sub = pm2.copy()
data_sub.wind_dir = l_enc.fit_transform(data_sub.wind_dir)

# one-hot encode
dummies = pd.DataFrame(oh_enc.fit_transform(data_sub.wind_dir.values.reshape(-1, 1)), columns=l_enc.classes_)

# join with original df
data_sub = data_sub.drop('wind_dir', axis=1)
data_sub = data_sub.join(dummies)
data_sub.head()


I am working on the file "Data Preprocessing/Preprocessing_exercise.ipynb".
The first task is:
Wow, now the fit is perfect!
4 - Create a new encoding for the quartile that a day falls under by wind speed and rerun the regression.  Comment on your results.
Can you write Python code for it?

# using pandas cut and subtracting 0.1 to include the min values
pm2_enc['wind_speed_quartile'] = pd.cut(pm2_enc.wind_speed,
                                        bins=list(pm2_enc.wind_speed.quantile([0])-0.1) + list(pm2_enc.wind_speed.quantile([0.25, 0.5, 0.75, 1])),
                                        labels=[0.25, 0.5, 0.75, 1])

# from solutions: using np.percentile:
quartile = np.percentile(data_sub['wind_speed_quartile'], [25, 50, 75, 100])

cat = []
for row in range(len(data_sub)):
    wind_speed = data_sub['wind_speed_quartile'].iloc[row]
    if wind_speed <= quartile[0]:
        cat.append('1st')
    if wind_speed <= quartile[1]:
        cat.append('2nd')
    if wind_speed <= quartile[2]:
        cat.append('3rd')
    if wind_speed <= quartile[3]:
        cat.append('4th')

data_sub['wind_quart'] = cat
# and then create dummies...

# transforming the column in numeric
pm2_enc.wind_speed_quartile = pd.to_numeric(pm2_enc.wind_speed_quartile)

lr.fit(pm2_enc.iloc[:, :-1], pm2_enc.iloc[:, -1])
lr.score(pm2_enc.iloc[:, :-1], pm2_enc.iloc[:, -1])


I am working on the file "Data Science Example Rental Cars.ipynb".
The first task is:
I want to count the number of rentals per vehicle ID in reservations.csv, appending these values as a column in vehicles.csv, in order to compare vehicle properties to reservation numbers directly. I also expect that a key factor in customer decisions may be the price difference between the actual and recommended prices, so I create a new column for this parameter as well. Finally, I merge the two dataframes to facilitate easy histogram plotting and analysis using the reservations data as a basis.
Can you write Python code for it?

# Count frequency of rentals and add column to VEHICLE_DATA
veh_id = VEHICLE_DATA.as_matrix(columns=['vehicle_id'])
res_id = RESERVATION_DATA.as_matrix(columns=['vehicle_id'])

# Use numpy here to ensure zero counts as a value
n_reservations = np.zeros(len(veh_id))
for i,id in enumerate(veh_id):
    n_reservations[i] = len(np.where(id == res_id)[0])
VEHICLE_DATA['num_reservations'] = n_reservations.astype(int)

# Add column with difference between the recommended price and the actual price
VEHICLE_DATA['diff_price'] = VEHICLE_DATA['recommended_price'] - VEHICLE_DATA['actual_price']

# Add a column that 'bucketizes' the number of reservations (low, med, high) categories
VEHICLE_DATA['categorical_reservations'] = pd.cut(VEHICLE_DATA['num_reservations'], 3, labels=["low","medium","high"])

# Merge databases to get vehicle features in the RESERVATION_DATA dataframe
MERGED_DATA = pd.merge(VEHICLE_DATA,RESERVATION_DATA)

# Define columns to plot initially as an exploration step
PLOT_COLUMNS = ['technology', 'num_images', 'street_parked', 'description','actual_price',
                'recommended_price','diff_price','num_reservations']


I am working on the file "Data Science Example Rental Cars.ipynb".
The first task is:
In the above code, I explore the data set using a Random Forest algorithm. It is a relatively quick and exceptionally versatile way to examine labelled data and derive relationships. In this case, I am using it to "score" the various parameters by how much they contribute to the number of reservations.
I ask the Random Forest algorithm first to classify based upon the raw number of reservations. I then also ask it to examine labels which are "bucketized," or binned, into three categories: low, medium, and high numbers of reservations, effectively smoothing out the options and ensuring that extreme values do not drive perceived trends.
We can conclude that the difference between the actual price of the car and the recommend price of the car is exceptionally important, followed by the actual price of the car, and then followed by the number of characters in the car's description. The number of images, whether or not it can be street parked, and the technology package do not play a large role in getting a car reserved frequently.
Can you write Python code for it?

fig, axes = plt.subplots(nrows=1, ncols=2, sharey=True)
fig.set_figheight(6)
fig.set_figwidth(16)

MERGED_DATA[['actual_price']].plot.hist(bins=15,ax=axes[0])
axes[0].set_xlabel("Actual Price")
MERGED_DATA[['diff_price']].plot.hist(bins=15,ax=axes[1])
plt.xlabel("Price Difference")
plt.show()


I am working on the file "Data Science Example Rental Cars.ipynb".
The first task is:
In the above figure, I show the recommended price versus the actual price in a hex density plot, with the color intensity representing the number of reservations. The orange line represents a one-to-one correlation between the two parameters. One can see immediately that there is a high density of reservations corresponding to the one-to-one line, where the actual price very nearly matches the recommended price. I also feel confident, now, that the recommended price does not need to be a free parameter in the future machine learning analysis, instead substituting the price difference.
Buyers naturally want to reserve a car at a price they perceive as fair, and coming close to the recommended price is very important.
Can you write Python code for it?

# Define feature columns to explore with machine learning
FEATURE_COLUMNS = ['technology', 'num_images', 'street_parked', 'description','diff_price','actual_price']

# Random forest regressor for continuous num_reservations
TARGET_COLUMN = ['num_reservations']
rf = RandomForestRegressor()
rf.fit(VEHICLE_DATA[FEATURE_COLUMNS],VEHICLE_DATA[TARGET_COLUMN].values.ravel())

print "====================================================================================="
print "Features Sorted by Score for Regressor:\n"
print sorted(zip(map(lambda x: round(x,4), rf.feature_importances_),FEATURE_COLUMNS),reverse=True)

# Random forest classifier for bucketized num_reservations
TARGET_COLUMN = ['categorical_reservations']
rf = RandomForestClassifier()
rf.fit(VEHICLE_DATA[FEATURE_COLUMNS],VEHICLE_DATA[TARGET_COLUMN].values.ravel())

print "\nFeatures Sorted by Score for Classifier:\n"
print sorted(zip(map(lambda x: round(x,4), rf.feature_importances_),FEATURE_COLUMNS),reverse=True)
print "====================================================================================="


I am working on the file "Data Science Example Rental Cars.ipynb".
The first task is:
In the above plots, I show the normalized frequency of reservation for the three types of reservations. On the x-axis, a 0 represents the absence of the technology package, and a 1 represents a vehicle having the technology. It is visually obvious that a proportionately larger number of hourly reservations are made with vehicles having the technology package. We can statistically support this claim.
Can you write Python code for it?

import scipy.stats

KSstatistic, pvalue = scipy.stats.ks_2samp(MERGED_DATA.loc[MERGED_DATA['reservation_type']==3]['technology'],
                                           MERGED_DATA.loc[MERGED_DATA['reservation_type']==2]['technology'])

print "KS probability that Weekly and Daily reservations are drawn from the same underlying population:\n"
print "P(KS) = {}\n".format(pvalue)

KSstatistic, pvalue = scipy.stats.ks_2samp(MERGED_DATA.loc[MERGED_DATA['reservation_type']==1]['technology'],
                                           MERGED_DATA.loc[MERGED_DATA['reservation_type']==2]['technology'])

print "KS probability that Hourly and Daily reservations are drawn from the same underlying population:\n"
print "P(KS) = {}\n".format(pvalue)


I am working on the file "Data Science Example Rental Cars.ipynb".
The first task is:
The final important parameter is the length of the description. The above plot shows the frequency of reservations as a function of the description character length. I can quickly conclude that more reservations are made for cars with descriptions less than 50 characters. After this point, the length of the description does not play a major role in one's decision whether or not to reserve a vehicle.
What role does the technology package play?
I already know that the presence of the technology package does not significantly influence the number of reservations. Knowing this, I want to explore whether it has a impact on the type of reservations made.
Can you write Python code for it?

plt.rc('figure',figsize=(8,6))
plt.figure(1)
MERGED_DATA.loc[MERGED_DATA['reservation_type']==1]['technology'].plot.hist(alpha=0.5,title="Hourly",
                                                                            normed=True)
plt.xlabel("With or Without Technology")

plt.figure(2)
MERGED_DATA.loc[MERGED_DATA['reservation_type']==2]['technology'].plot.hist(alpha=0.5,title="Daily",
                                                                            normed=True)
plt.xlabel("With or Without Technology")

plt.figure(3)
MERGED_DATA.loc[MERGED_DATA['reservation_type']==3]['technology'].plot.hist(alpha=0.5,title="Weekly",
                                                                            normed=True)
plt.xlabel("With or Without Technology")
plt.show()


I am working on the file "Data Science Example Rental Cars.ipynb".
The first task is:
The plot above shows every measured parameter, along with the newly added price-difference and number-of-reservations parameters, plotted against one another. I used this to try to quickly determine, visually, what parameters might drive the number of reservations. This is probably at the limit, in terms of number of parameters, of what I would just throw onto a scatter plot to inspect. A few things pop out:
 * Actual price, recommended price, and the price difference are all correlated (unsurprisingly)
 * The recommended price distribution is relatively flat, and it may not play a major role in reservations
 * Discrete parameters are hard to interpret here, but don't seem to show trends
The next step is to look at the price trends more closely.
Can you write Python code for it?

plt.rcParams.update({'font.size': LABEL_SIZE})
plt.rc('figure',figsize=(10,8))
VEHICLE_DATA.plot.hexbin(x='actual_price',y='recommended_price',C='num_reservations',reduce_C_function=np.max,
                         gridsize=25,figsize=(10,8))
plt.ylabel("Recommended Price")
plt.xlabel("Actual Price")
plt.plot([34,90],[34,90],color='orange',linewidth=3)
plt.show()


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Cleaning And Munging
Can you write Python code for it?

def parse_row(input_row, parsers):
    """given a list of parsers (some of which may be None)
    apply the appropriate one to each element of the input_row"""
    return [parser(value) if parser is not None else value for value, parser in zip(input_row, parsers)]

def parse_rows_with(reader, parsers):
    """wrap a reader to apply the parsers to each of its rows"""
    for row in reader:
        yield parse_row(row, parsers)

def try_or_none(f):
    """wraps f to return None if f raises an exception
    assumes f takes only one input"""
    def f_or_none(x):
        try: return f(x)
        except: return None
    return f_or_none

def parse_row(input_row, parsers):
    return [try_or_none(parser)(value) if parser is not None else value for value, parser in zip(input_row, parsers)]

import dateutil.parser
import csv

data = []

with open("stocks.csv", "r") as f:
    reader = csv.reader(f)
    for line in parse_rows_with(reader, [dateutil.parser.parse, None, float]):
        data.append(line)

data


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Dimensionality Reduction
On occasion, the actually useful dimension of the data might not correspond to the dimensions that we possess in the data. We can use principal component analysis to extract one or more dimensions associated with as much of the variation as possible.
Can you write Python code for it?

pca_data = [[7, 4, 3],
            [4, 1, 8],
            [6, 3, 5],
            [8, 6, 1],
            [8, 5, 7],
            [7, 2, 9],
            [5, 3, 3],
            [9, 5, 8],
            [7, 4, 5],
            [8, 2, 2]]
pca_xs = [random_normal() for _ in range(100)]
pca_ys1 = [x + random_normal() / 2 for x in pca_xs]
pca_ys2 = [-x + random_normal() / 2 for x in pca_xs]
pca_ys3 = [0 if x < 0 else 6 for x in pca_xs]
pca_data = [[x, y1, y2, y3] for (x, y1, y2, y3) in zip(pca_xs, pca_ys1, pca_ys2, pca_ys3)]

plt.scatter(pca_xs, pca_ys1);
plt.scatter(pca_xs, pca_ys2);
plt.scatter(pca_xs, pca_ys3);

def de_mean_matrix(A):
    """Returns each value in columns of A minus the mean of
    that column resulting in mean of 0 for all columns"""
    nr, nc = shape(A)
    column_means, _ = scale(A)
    return make_matrix(nr, nc, lambda i, j: A[i][j] - column_means[j])

pca_data_de_mean = de_mean_matrix(pca_data)
pca_data_de_mean[:5]

plt.scatter([x[0] for x in pca_data_de_mean], [0 for _ in pca_data_de_mean]);
plt.scatter([x[1] for x in pca_data_de_mean], [0.1 for _ in pca_data_de_mean]);
plt.scatter([x[2] for x in pca_data_de_mean], [0.2 for _ in pca_data_de_mean]);
plt.scatter([x[3] for x in pca_data_de_mean], [0.3 for _ in pca_data_de_mean]);
frame = plt.gca()
frame.axes.yaxis.set_visible(False);
plt.ylim((-0.1, 1));

plt.scatter([x[0] for x in pca_data_de_mean], [x[1] for x in pca_data_de_mean]);
plt.scatter([x[0] for x in pca_data_de_mean], [x[2] for x in pca_data_de_mean]);
plt.scatter([x[0] for x in pca_data_de_mean], [x[3] for x in pca_data_de_mean]);


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Exploring Your Data
Before you start building models and looking for answers, you should explore the data.
Exploring One-Dimensional Data
One-dimensional data is just a collection of some type, e.g., numbers.
A good place to start is with summary statistics.
Can you write Python code for it?

def bucketize(point, bucket_size):
    """floor the point to the next lower multiple of bucket size"""
    return bucket_size * math.floor(point / bucket_size)

def make_histogram(points, bucket_size):
    return Counter(bucketize(point, bucket_size) for point in points)

def plot_histogram(points, bucket_size, title=''):
    histogram = make_histogram(points, bucket_size)
    plt.bar(list(histogram.keys()), histogram.values(), width=bucket_size, edgecolor='white')
    plt.title(title)
    plt.show()

import random

random.seed(0)

# uniform #'s will be between -100 and 100
uniform = [200 * random.random() - 100 for _ in range(10000)]

# normal #'s will be from the normal distribution with mean of 0 and standard deviation of 57
normal = [57 * inverse_normal_cdf(random.random()) for _ in range(10000)]

plot_histogram(uniform, 10, 'Uniform Histogram')

plot_histogram(normal, 10, 'Normal Histogram')


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
For higher-dimensional data sets, iterate to find as many components as you like:
Can you write Python code for it?

def principal_component_analysis(X, num_components):
    components = []
    for _ in range(num_components):
        component = first_principal_component(X)
        components.append(component)
        X = remove_projection(X, component)

    return components


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
If we use inches, B's closest neighbor is A:
Can you write Python code for it?


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Now we can compute the variance of the data in the direction, w:
Can you write Python code for it?

def directional_variance_i(x_i, w):
    """the variance of the row x_i in the direction determined by w"""
    return dot(x_i, direction(w)) ** 2

def directional_variance(X, w):
    """the variance of the data in the direction determined w"""
    return sum(directional_variance_i(x_i, w) for x_i in X)

def directional_variance_gradient_i(x_i, w):
    """the contribution of row x_i to the gradient of
    the direction-w variance"""
    projection_length = dot(x_i, direction(w))
    return [2 * projection_length * x_ij for x_ij in x_i]

def directional_variance_gradient(X, w):
    return vector_sum(directional_variance_gradient_i(x_i, w) for x_i in X)

pca_data_variance = [directional_variance(pca_data_direction, w) for w in pca_data_direction]
pca_data_variance[:5]


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Now when we calculate distances, you can see that we end up with almost equivalent calculations for inches as we do centimeters:
Can you write Python code for it?

a_to_b_rescaled = distance([hts_in_rescaled[0][0], wts_rescaled[0][0]], [hts_in_rescaled[1][0], wts_rescaled[1][0]])
a_to_c_rescaled = distance([hts_in_rescaled[0][0], wts_rescaled[0][0]], [hts_in_rescaled[2][0], wts_rescaled[2][0]])
b_to_c_rescaled = distance([hts_in_rescaled[1][0], wts_rescaled[1][0]], [hts_in_rescaled[2][0], wts_rescaled[2][0]])

a_to_b_rescaled, a_to_c_rescaled, b_to_c_rescaled

a_to_b_rescaled = distance([hts_cm_rescaled[0][0], wts_rescaled[0][0]], [hts_cm_rescaled[1][0], wts_rescaled[1][0]])
a_to_c_rescaled = distance([hts_cm_rescaled[0][0], wts_rescaled[0][0]], [hts_cm_rescaled[2][0], wts_rescaled[2][0]])
b_to_c_rescaled = distance([hts_cm_rescaled[1][0], wts_rescaled[1][0]], [hts_cm_rescaled[2][0], wts_rescaled[2][0]])

a_to_b_rescaled, a_to_c_rescaled, b_to_c_rescaled


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Rescale our weights so that they do not differ by units:
Can you write Python code for it?

hts_in = [[63], [67], [70]]
hts_cm = [[160], [170.2], [177.8]]
wts = [[150], [160], [171]]

hts_in_rescaled = rescale(hts_in)
hts_cm_rescaled = rescale(hts_cm)
wts_rescaled = rescale(wts)

hts_in_rescaled, hts_cm_rescaled, wts_rescaled


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
The first principal component is the direction that maximizes the directional variance above.
Can you write Python code for it?

def first_principal_component(X):
    guess = [1 for _ in X[0]]
    unscaled_maximizer = maximize_batch(
        partial(directional_variance, X), # is now a function of w
        partial(directional_variance_gradient, X), # is now a function of w
        guess)
    return direction(unscaled_maximizer)

first_principal_component(pca_data)


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Then transform the data into a lower-dimensional space:
Can you write Python code for it?

def transform_vector(v, components):
    return [dot(v, w) for w in components]

def transform(X, components):
    return [transform_vector(x_i, components) for x_i in X]

pca_components = principal_component_analysis(pca_data, 3)
pca_transformed = transform(pca_data, pca_components)
pca_transformed[:5]

plt.scatter([x[0] for x in pca_transformed], [x[1] for x in pca_transformed]);
plt.scatter([x[0] for x in pca_transformed], [x[2] for x in pca_transformed]);
plt.title('3-dimensions n=' + str(len(pca_transformed)));

pca_components = principal_component_analysis(pca_data, 2)
pca_transformed = transform(pca_data, pca_components)
pca_transformed[:5]

plt.scatter([x[0] for x in pca_transformed], [x[1] for x in pca_transformed]);
plt.title('2-dimensions n=' + str(len(pca_transformed)));

pca_components = principal_component_analysis(pca_data, 1)
pca_transformed = transform(pca_data, pca_components)
pca_transformed[:5]

plt.scatter([x[0] for x in pca_transformed], [0 for _ in pca_transformed]);
plt.title('1-dimension n=' + str(len(pca_transformed)));


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
Two Dimensions
Can you write Python code for it?

def random_normal():
    """retuns random #'s from standard normal distribution"""
    return inverse_normal_cdf(random.random())

xs = [random_normal() for _ in range(1000)]
ys1 = [x + random_normal() / 2 for x in xs]
ys2 = [-x + random_normal() / 2 for x in xs]
ys3 = [0 if x < 0 else 6 for x in xs]

plt.scatter(xs, ys1, marker='.', color='black', label='ys1');
plt.scatter(xs, ys2, marker='.', color='gray', label='ys2');
plt.xlabel('xs');
plt.ylabel('ys');
plt.legend(loc=5);
plt.title('Very Different Joint Distributions');

correlation(xs, ys1), correlation(xs, ys2)


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
We can use stochastic gradient descent here:
Can you write Python code for it?

def first_principal_component_sgd(X):
    guess = [1 for _ in X[0]]
    unscaled_maximizer = maximize_stochastic(
        lambda x, _, w: directional_variance_i(x, w),
        lambda x, _, w: directional_variance_gradient_i(x, w),
        X,
        [None for _ in X], # the fake "y"
        guess)
    return direction(unscaled_maximizer)

first_principal_component_sgd(pca_data)

def project(v, w):
    """returns the projection of v onto direction w"""
    projection_length = dot(v, w)
    return scalar_multiply(projection_length, w)

def remove_projection_from_vector(v, w):
    """projects v onto w and substracts the result from v"""
    return vector_subtract(v, project(v, w))

def remove_projection(X, w):
    return [remove_projection_from_vector(x_i, w) for x_i in X]


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
When results are impacted by units of measure like in the above example, scaling can help. Rescaling projects values into a dimension with mean 0 and standard deviation 1. This effectively converts data into a common unit of "standard deviations from the mean."
Can you write Python code for it?

def scale(data_matrix):
    """returns the means and standard deviations of each column"""
    num_rows, num_cols = shape(data_matrix)
    means = [mean(get_column(data_matrix,j)) for j in range(num_cols)]
    stdevs = [standard_deviation(get_column(data_matrix,j)) for j in range(num_cols)]

    return means, stdevs

def rescale(data_matrix):
    """rescales the input data so that each column
    has mean 0 and standard deviation 1
    leaves alone columns with no deviation"""
    means, stdevs = scale(data_matrix)

    def rescaled(i, j):
        if stdevs[j] > 0:
            return (data_matrix[i][j] - means[j]) / stdevs[j]
        else:
            return data_matrix[i][j]

    num_rows, num_cols = shape(data_matrix)
    return make_matrix(num_rows, num_cols, rescaled)


I am working on the file "Data Science From Scratch/10 - Working With Data.ipynb".
The first task is:
With the de-meaned matrix, we can figure out which direction captures the most variance in the data:
Can you write Python code for it?

def direction(w):
    mag = magnitude(w)
    return [w_i / mag for w_i in w]

pca_data_direction = [direction(w) for w in pca_data_de_mean]
pca_data_direction[:5]

plt.scatter([x[0] for x in pca_data_direction], [x[1] for x in pca_data_direction]);
plt.scatter([x[0] for x in pca_data_direction], [x[2] for x in pca_data_direction]);
plt.scatter([x[0] for x in pca_data_direction], [x[3] for x in pca_data_direction]);


I am working on the file "Data Science From Scratch/11 - Machine Learning.ipynb".
The first task is:
Correctness
Predictions fall into one of four categories:
| True Positive | Predicted true, and it was true |
| :----------- | :--------------- |
| False Positive (Type 1 Error) | Predicted true, but it was false |
| False Negative (Type 2 Error) | Predicted false, but it was true |
| True Negative | Predicted false, and it was false |
Suppose we predicted that a person has leukemia if they are named "Luke." We can predict with 98% accuracy--but this clearly shows that accuracy is itself solely a measure of a good model:
| | leukemia | no leukemia | total |
| --- | --- | --- | --- |
| "Luke" | 70 | 4,930 | 5,000 |
| not "Luke" | 13,930 | 981,070 | 995,000 |
| total | 14,000 | 986,000 | 1,000,000 |
Can you write Python code for it?

true_positives = 70
false_positives = 4930
true_negatives = 981070
false_negatives = 13930

def accuracy(tp, fp, fn, tn):
    correct = tp + tn
    total = tp + fp + fn + tn
    return correct / total

accuracy(true_positives, false_positives, false_negatives, true_negatives) # 0.98114


I am working on the file "Data Science From Scratch/11 - Machine Learning.ipynb".
The first task is:
Recall measures the proportion of positives predicted by the model:
Can you write Python code for it?

def recall(tp, fp, fn, tn):
    return tp / (tp + fn)

recall(true_positives, false_positives, false_negatives, true_negatives) # 0.005


I am working on the file "Data Science From Scratch/11 - Machine Learning.ipynb".
The first task is:
The precision and recall scores clearly point to the inadqequacy of our model. The F1 score combines precision and recall into a single new measure:
Can you write Python code for it?

def f1_score(tp, fp, fn, tn):
    p = precision(tp, fp, fn, tn)
    r = recall(tp, fp, fn, tn)
    return 2 * p * r / (p + r)

f1_score(true_positives, false_positives, false_negatives, true_negatives) # 0.007


I am working on the file "Data Science From Scratch/11 - Machine Learning.ipynb".
The first task is:
When splitting data, it's important to keep input data and target data in the same order:
Can you write Python code for it?

def train_test_split(x, y, test_pct):
    data = zip(x, y) # pair corresponding values
    train, test = split_data(data, 1 - test_pct) # split the data set of pairs
    x_train, y_train = zip(*train) # magical un-zip trick
    x_test, y_test = zip(*test)
    return x_train, x_test, y_train, y_test


I am working on the file "Data Science From Scratch/14 - Simple Linear Regression.ipynb".
The first task is:
Higher R-squared scores represent a better fitting model. 1 is the highest that an R-squared score can go.
Using Gradient Descent
We can also make predictions using the Gradient Descent algorithm:
Can you write Python code for it?

def squared_error(x_i, y_i, theta):
    alpha, beta = theta
    return error(alpha, beta, x_i, y_i)**2

def squared_error_gradient(x_i, y_i, theta):
    alpha, beta = theta
    return [-2 * error(alpha, beta, x_i, y_i), # alpha partial derivative
            -2 * error(alpha, beta, x_i, y_i) * x_i] # beta partial derivative

# choose random value to start
random.seed(0)
theta = [random.random(), random.random()]
alpha, beta = minimize_stochastic(squared_error, squared_error_gradient, num_friends_clean, daily_minutes_clean, theta, 0.0001)
alpha, beta


I am working on the file "Data Science From Scratch/4 - Linear Algebra.ipynb".
The first task is:
Matrices
A matrix is a two-dimensional list of numbers.
Can you write Python code for it?

# 2 x 3 matrix
A = [[1, 2, 3],
     [4, 5, 6]]
# 3 x 2 matrix
B = [[1, 2],
     [3, 4],
     [5, 6]]

def shape(A):
    num_rows = len(A)
    num_cols = len(A[0]) if A else 0
    return num_rows, num_cols

shape(A), shape(B)

def get_row(A, i):
    return A[i]

def get_column(A, j):
    return [A_i[j] for A_i in A]

get_row(A, 1), get_column(A, 1)

def make_matrix(num_rows, num_cols, entry_fn):
    """returns a num_rows x num_cols matrix"""
    return [[entry_fn(i, j) for j in range(num_cols)] for i in range(num_rows)]

make_matrix(2, 2, lambda i, j: i + j)

def diagonal(i, j):
    """given row i and col j, returns 1 for diagonal, 0 otherwise"""
    return 1 if i == j else 0

make_matrix(5, 5, diagonal)


I am working on the file "Data Science From Scratch/4 - Linear Algebra.ipynb".
The first task is:
We can now use the above to compute the distance between two vectors:
    $$\sqrt{(v_i - w_1)^2 + ... + (v_n - w_n)^2}$$
Can you write Python code for it?

def squared_distance(v, w):
    """(v_1 - w_1) ** 2 + ... + (v_n - w_n) ** 2"""
    return sum_of_squares(vector_subtract(v, w))

def distance(v, w):
    return math.sqrt(squared_distance(v, w))

distance(grades, [90, 75, 70, 60])


I am working on the file "Data Science From Scratch/5 - Statistics.ipynb".
The first task is:
Correlation
We saw that variance measures a single variable's deviation from the mean, now we will see covariance and how it measures how two variables in tandem deviate.
Can you write Python code for it?


I am working on the file "Data Science From Scratch/5 - Statistics.ipynb".
The first task is:
Covariance can be hard to interpret because:
- its units are the same as the inputs' units.
- it is hard to settle on what counts as a "large" covariance.
Correlation is an easier to understand measure. It has no units and is always between -1 and 1.
Can you write Python code for it?

def correlation(x, y):
    sd_x = standard_deviation(x)
    sd_y = standard_deviation(y)
    if sd_x > 0 and sd_y > 0:
        return covariance(x, y) / sd_x / sd_y
    else:
        return 0 # correlation is 0 when there is no variation

correlation(num_friends, daily_minutes)


I am working on the file "Data Science From Scratch/5 - Statistics.ipynb".
The first task is:
Outliers can greatly affect correlation. One stands out:
Can you write Python code for it?


I am working on the file "Data Science From Scratch/5 - Statistics.ipynb".
The first task is:
The variance is in squared units which can make it difficult to grasp. An easier to understand measure is the standard deviation.
Can you write Python code for it?

def standard_deviation(x):
    return math.sqrt(variance(x))

standard_deviation(num_friends)

def interquartile_range(x):
    return quantile(x, 0.75) - quantile(x, 0.25)

interquartile_range(num_friends)


I am working on the file "Data Science From Scratch/6 - Probability.ipynb".
The first task is:
Probabilities are a way of quantifying the possibility of the occurrence of a specific event or events given the set of all possible events.
Notationally, $P(E)$ means "the probability of event $E$."
Dependence and Independence
Events $E$ and $F$ are dependent if information about $E$ gives us information about the probability of $F$ occurring (or vice versa). If this is not the case, the variables are independent of each other.
For independent events, the probability of both occurring is the product of the probabilities of each occurring:
$$P(E, F) = P(E)P(F)$$
Conditional Probability
If events are not independent, we can express conditional probability ($E$ is conditional on $F$ or what is the probability that $E$ happens given that $F$ happens):
$$P(E\ |\ F) = P(E, F)\ /\ P(F)$$
which (if $E$ and $F$ are dependent) can be written as
$$P(E, F) = P(E\ |\ F)P(F)$$
When $E$ and $F$ are independent:
$$P(E\ |\ F) = P(E)$$
Bayes's Theorem
Conditional probabilities can be "reversed":
$$P(E \text{ | } F) = P(E, F) \text{ / } P(F) = P(F \text{ | } E)P(E) \text{ / } P(F)$$
If $E$ doesn't happen:
$$P(F) = P(F, E) + P(F, \neg E)$$
Leads to Bayes's Theorem:
$$P(E\ |\ F) = P(F\ |\ E)P(E)\ /\ [P(F\ |\ E)P(E) + P(F\ |\ \neg E)P(\ \neg E)]$$
Random Variables
A random variable is one whose possible values can be placed on an associated probability distribution. The distribution refines the probabilities that the variable will take on each of the possible values.
Continuous Distributions
Coin flips represent a discrete distribution, i.e., one that takes on mutually exclusive values with no "in-betweens." A continuous distribution is one that allows for a full range of values along a continuum such as height or weight.
Continuous distributions use a probability density function (pdf) to define probability of a value within a given range.
The pdf for the uniform distribution is:
Can you write Python code for it?

def uniform_pdf(x):
    return 1 if x >= 0 and x < 1 else 0

xs = np.arange(-1, 2, .001)
ys = [uniform_pdf(x) for x in xs]
plt.plot(xs, ys);

uniform_pdf(-0.01)


I am working on the file "Data Science From Scratch/6 - Probability.ipynb".
The first task is:
The Normal Distribution
The normal distribution is the definitive example of a random distribution (the classic bell curve shape). It is defined by two parameters: the mean $\mu$ and the standard deviation $\sigma$.
The function for the distribution is:
$$f(x\ |\ \mu, \sigma) = \frac{1}{\sqrt{2\pi}\sigma}\ exp\bigg(-\frac{(x - \mu)^2}{2\sigma^2}\bigg)$$
Can you write Python code for it?

def normal_pdf(x, mu=0, sigma=1):
    sqrt_two_pi = math.sqrt(2 * math.pi)
    return (math.exp(-(x - mu)**2 / 2 / sigma**2) / (sqrt_two_pi * sigma))

xs = [x / 10.0 for x in range(-50, 50)]
plt.plot(xs, [normal_pdf(x, sigma=1) for x in xs],'-',label='mu=0,sigma=1')
plt.plot(xs, [normal_pdf(x, sigma=2) for x in xs],'--',label='mu=0,sigma=2')
plt.plot(xs, [normal_pdf(x, sigma=0.5) for x in xs],':',label='mu=0,sigma=0.5')
plt.plot(xs, [normal_pdf(x, mu=-1) for x in xs],'-.',label='mu=-1,sigma=1')
plt.legend()
plt.title("Various Normal pdfs")
plt.show()


I am working on the file "Data Science From Scratch/6 - Probability.ipynb".
The first task is:
The mean of a Bernoulli(p) variable is $p$ and its standard deviation is $\sqrt{p(1 - p)}$.
Can you write Python code for it?

def plot_binomial(p, n, num_points):
    data = [binomial(n, p) for _ in range(num_points)]

    histogram = Counter(data)
    plt.bar([x - 0.04 for x in histogram.keys()],
            [v / num_points for v in histogram.values()],
           0.8,
           color='0.75')

    mu = p * n
    sigma = math.sqrt(n * p * (1 - p))

    xs = range(min(data), max(data) + 1)
    ys = [normal_cdf(i + 0.5, mu, sigma) - normal_cdf(i - 0.5, mu, sigma) for i in xs]
    plt.plot(xs, ys)
    plt.title('Binomial Distribution vs. Normal Approximation')

plot_binomial(0.75, 100, 10000)


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
0.5 falls within our interval so we do not conclude that the coin is unfair.
What if we observe 540 heads though?
Can you write Python code for it?


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
A decision must be made with respect to significance, i.e., how willing are we to accept "false positives" (type 1 errors) by rejecting $H_0$ even though it is true? This is often set to 5% or 1%. We will use 5%:
Can you write Python code for it?


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Assuming a normal distribution, we conclude that we are 95% confident that the interval below includes the true $p$:
Can you write Python code for it?


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Bayesian Inference
In Bayesian Inference we start with a prior distribution for the parameters and then use the actual observations to get the posterior distribution of the same parameters. So instead of judging the probability of hypotheses, we make probability judgments about the parameters.
We will use the Beta distribution to convert all probabilities to a value between 0 and 1.
Can you write Python code for it?

def B(alpha, beta):
    """a normalizing constant so that the total probability is 1"""
    return math.gamma(alpha) * math.gamma(beta) / math.gamma(alpha + beta)

def beta_pdf(x, alpha, beta):
    if x < 0 or x > 1: # no weight outside of [0, 1]
        return 0
    return x ** (alpha - 1) * (1 - x) ** (beta - 1) / B(alpha, beta)


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
If $H_0$ is true, $p$ should equal 0.5. The interval above denotes the values outside of which there is a 5% chance that this is false. Now we can determine the power of a test, i.e., how willing are we to accept "false positives" (type 2 errors) by failing to reject $H_0$ even though it is false.
Can you write Python code for it?

# 95% bounds based on assumption p is 0.5
lo, hi = normal_two_sided_bounds(0.95, mu_0, sigma_0)

# actual mu and sigma based on p = 0.55
mu_1, sigma_1 = normal_approximation_to_binomial(1000, 0.55)

# a type 2 error means we fail to reject the null hypothesis
# which will happen when X is still in our original interval
type_2_probability = normal_probability_between(lo, hi, mu_1, sigma_1)
power = 1 - type_2_probability # 0.887


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
In this scenario, 0.5 falls outside of our interval so the "fair coin" hypothesis is not confirmed.
P-hacking
P-hacking involves using various "hacks" to get a $p$ value to go below 0.05: creating a superfluous number of hypotheses, selectively removing outliers, etc.
Can you write Python code for it?

def run_experiment():
    """flip a fair coin 1000 times, True = heads, False = tails"""
    return [random.random() < 0.5 for _ in range(1000)]

def reject_fairness(experiment):
    """using the 5% significance levels"""
    num_heads = len([flip for flip in experiment if flip])
    return num_heads < 469 or num_heads > 531

random.seed(0)
experiments = [run_experiment() for _ in range(1000)]
num_rejections = len([experiment for experiment in experiments if reject_fairness(experiment)])
num_rejections # 46


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Not knowing $p$ we use our estimate:
Can you write Python code for it?

p_hat = 525 / 1000
mu = p_hat
sigma = math.sqrt(p_hat * (1 - p_hat) / 1000)
sigma


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Run a 5% significance test to find the cutoff below which 95% of the probability lies:
Can you write Python code for it?

hi = normal_upper_bound(0.95, mu_0, sigma_0)

# is 526 (< 531, since we need more probability in the upper tail)
type_2_probability = normal_probability_below(hi, mu_1, sigma_1)

power = 1 - type_2_probability # 0.936

def two_sided_p_value(x, mu=0, sigma=1):
    if x >= mu:
        # if x is greater than the mean, the tail is what's greater than x
        return 2 * normal_probability_above(x, mu, sigma)
    else:
        # if x is less than the mean, the tail is what's less than x
        return 2 * normal_probability_below(x, mu, sigma)

two_sided_p_value(529.5, mu_0, sigma_0) # 0.062

extreme_value_count = 0
for _ in range(100000):
    num_heads = sum(1 if random.random() < 0.5 else 0 # count # of heads
                    for _ in range(1000)) # in 1000 flips
    if num_heads >= 530 or num_heads <= 470: # and count how often
        extreme_value_count += 1 # the # is 'extreme'

print(extreme_value_count / 100000) # 0.062

two_sided_p_value(531.5, mu_0, sigma_0) # 0.0463


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Statistical Hypothesis Testing
Classical testing involves a null hypothesis $H_0$ that represents the default and an alternative hypothesis $H_1$ to test. Statistics helps us to determine whether $H_0$ should be considered false or not.
Example: Flipping A Coin
Null hypothesis = the coin is fair = $p = 0.5$
Alternative hypothesis = coins is not fair = $p \not = 0.5$
To test, $n$ samples will be collected. Each toss is a Bernoulli(n, p) trial.
Can you write Python code for it?

def normal_approximation_to_binomial(n, p):
    """return mu and sigma corresponding to Binomial(n, p)"""
    mu = p * n
    sigma = math.sqrt(p * (1 - p) * n)
    return mu, sigma

normal_probability_below = normal_cdf

def normal_probability_above(lo, mu=0, sigma=1):
    return 1 - normal_cdf(lo, mu, sigma)

def normal_probability_between(lo, hi, mu=0, sigma=1):
    return normal_cdf(hi, mu, sigma) - normal_cdf(lo, mu, sigma)

def normal_probabilty_outside(lo, hi, mu=0, sigma=1):
    return 1 - normal_probability_between(lo, hi, mu, sigma)

def normal_upper_bound(probability, mu=0, sigma=1):
    """returns the z for which P(Z <= z) = probability"""
    return inverse_normal_cdf(probability, mu, sigma)

def normal_lower_bound(probability, mu=0, sigma=1):
    """returns the z for which P(Z >= z) = probability"""
    return inverse_normal_cdf(1 - probability, mu, sigma)

def normal_two_sided_bounds(probability, mu=0, sigma=1):
    """returns the symmetric (about the mean) bounds
    that contain the specified probability"""
    tail_probability = (1 - probability) / 2

    # upper bound should have tail_probability above it
    upper_bound = normal_lower_bound(tail_probability, mu, sigma)

    # lower bound should have tail_probability below it
    lower_bound = normal_upper_bound(tail_probability, mu, sigma)

    return lower_bound, upper_bound


I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Valid inferences come from a priori hypotheses (hypotheses created before collecting any data) and data cleansing without reference to the hypotheses.
Example: Running An A/B Test
Can you write Python code for it?

def estimated_parameters(N, n):
    p = n / N
    sigma = math.sqrt(p * (1 - p) / N)
    return p, sigma

def a_b_test_statistic(N_A, n_A, N_B, n_B):
    p_A, sigma_A = estimated_parameters(N_A, n_A)
    p_B, sigma_B = estimated_parameters(N_B, n_B)
    return (p_B - p_A) / math.sqrt(sigma_A ** 2 + sigma_B ** 2)

z = a_b_test_statistic(1000, 200, 1000, 180)
z

two_sided_p_value(z)

z = a_b_test_statistic(1000, 200, 1000, 150)
two_sided_p_value(z)


I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
Another optimization involves maximizing a function by minimizing its negative:
Can you write Python code for it?

def negate(f):
    """return a function that for any input x returns -f(x)"""
    return lambda *args, **kwargs: -f(*args, **kwargs)

def negate_all(f):
    """the same when f returns a list of numbers"""
    return lambda *args, **kwargs: [-y for y in f(*args, **kwargs)]

def maximize_batch(target_fn, gradient_fn, theta_0, tolerance=0.000001):
    return minimize_batch(negate(target_fn),
                          negate_all(gradient_fn),
                          theta_0,
                          tolerance)


I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
Choose The Right Step Size
Although using the right step size is important, there is not an exact way to know which size to use. Some general rules of thumb are:
- use a fixed step size.
- gradually shrink the step size over iterations.
- finding the step size that leads to the minimum at each step.
We can illustrate this last point with an example:
Can you write Python code for it?


I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
Some step sizes provide invalid inputs to our function so we will need a safe version:
Can you write Python code for it?

def safe(f):
    """return a new function that's the same as f,
    except that it outputs infinity whenever f produces an error"""
    def safe_f(*args, **kwargs):
        try:
            return f(*args, **kwargs)
        except:
            return float('inf') # this means "infinity" in Python
    return safe_f


I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
Stochastic Gradient Descent
The approach above is computationally expensive: the gradient is computed for the whole dataset. A more efficient approach would be to compute the gradient for one point at a time. This is what stochastic gradient descent does. It will run over our data--each point individually--until the stopping point which we define.
Best results can be achieved by selecting our data at random:
Can you write Python code for it?

def in_random_order(data):
    """generator that returns the elements of data in random order"""
    indexes = [i for i, _ in enumerate(data)] # create a list of indexes
    random.shuffle(indexes) # shuffle them
    for i in indexes: # return the data in that order
        yield data[i]


I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
Stochastic gradient descent is a lot faster than our first version. Here is a negated version:
Can you write Python code for it?

def maximize_stochastic(target_fn, gradient_fn, x, y, theta_0, alpha_0=0.01):
    return minimize_stochastic(negate(target_fn), negate_all(gradient_fn), x, y, theta_0, alpha_0)


I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
The Idea Behind Gradient Descent
In calculus, the gradient is the vector of partial derivatives and it identifies the direction of inputs that increase the functions outputs most quickly.
We can use gradient descent to maximize a function by starting at a random point and moving in the direction of the gradient--then repeat.
Estimating The Gradient
The derivative of $f(x)$ measures how $f(x)$ changes when making very small changes to $x$:
Can you write Python code for it?

def difference_quotient(f, x, h):
    return (f(x + h) - f(x)) / h


I am working on the file "Data Science From Scratch/9 - Getting Data.ipynb".
The first task is:
Delimited Files
It's easy to work with delimited files.
Can you write Python code for it?

import csv

with open('stocks.csv', 'r') as f:
    reader = csv.reader(f, delimiter=',')
    for row in reader:
        date = row[0]
        symbol = row[1]
        closing_price = float(row[2])
        print(date, symbol, closing_price)

with open('stocks-headers.csv', 'r') as f:
    reader = csv.DictReader(f, delimiter=':')
    for row in reader:
        date = row['date']
        symbol = row['symbol']
        closing_price = float(row['closing_price'])
        print(date, symbol, closing_price)


I am working on the file "Data Science From Scratch/9 - Getting Data.ipynb".
The first task is:
HTML And The Parsing Thereof
Can you write Python code for it?

some_html = """

<title>A web page</title>

Joel Grus

Data Science

"""

from bs4 import BeautifulSoup
import requests

html = requests.get('http://www.example.com').text
html = some_html
soup = BeautifulSoup(html, 'html5lib')

first_paragraph = soup.find('p')
first_paragraph

soup.p.text, soup.p.text.split()

soup.p['id']

soup.p.get('id')

soup.find_all('p')

[p for p in soup('p') if p.get('id')]

soup('p', {'class' : 'important'})

soup('p', 'important')

[p for p in soup('p') if 'important' in p.get('class', [])]


I am working on the file "Data Science From Scratch/9 - Getting Data.ipynb".
The first task is:
Reading Files
The Basics Of Text Files
Use the open function to open a file:
Can you write Python code for it?

# 'r' means read-only
file_for_reading = open('reading_file.txt', 'r')

# 'w' is write—will destroy the file if it already exists!
file_for_writing = open('writing_file.txt', 'w')

# 'a' is append—for adding to the end of the file
file_for_appending = open('appending_file.txt', 'a')

# don't forget to close your files when you're done
file_for_writing.close()


I am working on the file "Data Science From Scratch/9 - Getting Data.ipynb".
The first task is:
Use a with block to ensure that files are closed:
Can you write Python code for it?

with open('SomeFile.txt', 'r') as f:
    for line in f:
        print(line.strip())

# After with block, file is closed


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Examining distribution of encounter durations (with loc_facility_cd)
Analyze the durations of the RRT event patients.
Can you write Python code for it?

query_count = """
SELECT count(*)
FROM (
    SELECT DISTINCT ce.encntr_id
    FROM clinical_event ce
    INNER JOIN encounter enc ON enc.encntr_id = ce.encntr_id
    WHERE ce.event_cd = '54411998'
    AND ce.result_status_cd NOT IN ('31', '36')
    AND ce.valid_until_dt_tm > 4e12
    AND ce.event_class_cd not in ('654645')
    AND enc.loc_facility_cd = '633867'
    AND enc.encntr_complete_dt_tm < 4e12
    AND enc.admit_type_cd!='0'
    AND enc.encntr_type_class_cd='391'
    ) AS A ;
"""
cur.execute(query_count)
cur.fetchall()

query_count = """
SELECT count(*)
FROM (
    SELECT DISTINCT encntr_id
    FROM encounter enc
    WHERE enc.loc_facility_cd = '633867'
    AND enc.encntr_complete_dt_tm < 4e12
    AND enc.admit_type_cd!='0'
    AND enc.encntr_type_class_cd='391'
    AND encntr_id NOT IN (
                        SELECT DISTINCT ce.encntr_id
                        FROM clinical_event ce
                        INNER JOIN encounter enc ON enc.encntr_id = ce.encntr_id
                        WHERE ce.event_cd = '54411998'
                        AND ce.result_status_cd NOT IN ('31', '36')
                        AND ce.valid_until_dt_tm > 4e12
                        AND ce.event_class_cd not in ('654645')
                        )
    ) AS A;
"""
cur.execute(query_count)
cur.fetchall()


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
For admit_type_cd!='0' & encntr_type_class_cd='391
Can you write Python code for it?

query_TotalEncs = """
SELECT count(1)
FROM (
    SELECT DISTINCT encntr_id
    FROM encounter
    WHERE encntr_complete_dt_tm < 4e12
    AND loc_facility_cd = '633867'
    AND admit_type_cd!='0'
    AND encntr_type_class_cd='391'
    ) t;
"""

cur.execute(query_TotalEncs)
cur.fetchall()


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
From scipy documentation: "If the KS statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same"
Null hypothesis: the distributions are the same.
Looks like samples from 4382273 are different... plot that & 633867
Can you write Python code for it?

plt.figure(figsize=(10,8))
df[df['loc_facility_cd']=='633867']['diff_hours'].plot.hist(alpha=0.4, bins=500,normed=True)
df[df['loc_facility_cd']=='4382273']['diff_hours'].plot.hist(alpha=0.4, bins=700,normed=True)
plt.xlabel('Hospital Stay Durations, hours')
plt.legend(['633867', '4382273'])
plt.xlim(0, 1000)


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Group by facility
We want to pull from similar patient populations
Can you write Python code for it?

df.head()

df.loc_desc.value_counts()

grouped = df.groupby('loc_desc')

grouped.describe()


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Let's look at durations for inpatients WITH RRTs from the Main Hospital where encounter_admit_type is not zero
Can you write Python code for it?

query = """
SELECT
      DISTINCT ce.encntr_id
    , COALESCE(tci.checkin_dt_tm, enc.arrive_dt_tm) AS checkin_dt_tm
    , enc.depart_dt_tm as depart_dt_tm
    , (enc.depart_dt_tm - COALESCE(tci.checkin_dt_tm, enc.arrive_dt_tm))/3600000 AS diff_hours
    , enc.reason_for_visit
    , enc.admit_src_cd
    , enc.admit_type_cd
FROM clinical_event ce
INNER JOIN encounter enc ON enc.encntr_id = ce.encntr_id
LEFT OUTER JOIN  (
                SELECT
                      ti.encntr_id AS encntr_id
                    , MIN(tc.checkin_dt_tm) AS checkin_dt_tm
                FROM tracking_item ti
                JOIN tracking_checkin tc ON ti.tracking_id = tc.tracking_id
                GROUP BY ti.encntr_id
                 ) tci
ON tci.encntr_id = enc.encntr_id
WHERE enc.loc_facility_cd = '633867'
AND enc.encntr_complete_dt_tm < 4e12
AND enc.admit_type_cd!='0'
AND enc.encntr_type_class_cd='391'
AND enc.encntr_id IN (
                    SELECT DISTINCT ce.encntr_id
                    FROM clinical_event ce
                    WHERE ce.event_cd = '54411998'
                    AND ce.result_status_cd NOT IN ('31', '36')
                    AND ce.valid_until_dt_tm > 4e12
                    AND ce.event_class_cd not in ('654645')
                    )
;"""

cur.execute(query)
df_rrt = as_pandas(cur)

df_rrt.head()

df_rrt.describe().T
# the mean stay is 292 hours (12.1 days).
# The median stay is 184 hours (7.67 days)
# The minimum stay is 8 hours. The longest stay is 3550 hours (~148 days)

plt.figure()
df_rrt.diff_hours.hist(bins = 300)
plt.xlim(0, 600)

# Records with short durations:
df_rrt[df_rrt.diff_hours < 12]


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Let's look at durations for inpatients WITHOUT RRTs from the Main Hospital where encounter_admit_type is not zero
Can you write Python code for it?

query = """
SELECT DISTINCT
      ce.encntr_id
    , COALESCE(tci.checkin_dt_tm
    , enc.arrive_dt_tm) AS checkin_dt_tm
    , enc.depart_dt_tm as depart_dt_tm
    , (enc.depart_dt_tm - COALESCE(tci.checkin_dt_tm, enc.arrive_dt_tm))/3600000 AS diff_hours
    , enc.reason_for_visit
    , enc.admit_src_cd
    , enc.admit_type_cd
FROM clinical_event ce
INNER JOIN encounter enc ON enc.encntr_id = ce.encntr_id
LEFT OUTER JOIN  (
                SELECT
                      ti.encntr_id AS encntr_id
                    , MIN(tc.checkin_dt_tm)  AS checkin_dt_tm
                FROM tracking_item ti
                JOIN tracking_checkin tc ON ti.tracking_id = tc.tracking_id
                GROUP BY ti.encntr_id
                ) tci
ON tci.encntr_id = enc.encntr_id
WHERE enc.loc_facility_cd = '633867'
AND enc.encntr_complete_dt_tm < 4e12
AND enc.admit_type_cd!='0'
AND enc.encntr_type_class_cd='391'
AND enc.encntr_id NOT IN (
                        SELECT DISTINCT ce.encntr_id
                        FROM clinical_event ce
                        WHERE ce.event_cd = '54411998'
                        AND ce.result_status_cd NOT IN ('31', '36')
                        AND ce.valid_until_dt_tm > 4e12
                        AND ce.event_class_cd not in ('654645')
                        )
;"""

cur.execute(query)
df_nonrrt = as_pandas(cur)

df_nonrrt.describe().T
# NonRRT: The mean stay is 122 hours (5 days) // RRT: The mean stay is 292 hours (12.1 days).
# NonRRT: The median stay is 77 hours (3.21 days)// RRT: The median stay is 184 hours (7.67 days)
# NonRRT: The minimum stay is 0.08 hours // RRT: The minimum stay is ~8 hours.

plt.figure()
df_nonrrt.diff_hours.hist(bins = 500)
plt.xlim(0, 600)


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Looks like these three locations (633867, 4382264, 4382273) have about the same distribution.
Appropriate test to verify this: 2-sample Kolmogorov-Smirnov, if you're willing to compare pairwise...other tests? Wikipedia has a good article with references: https://en.wikipedia.org/wiki/Kolmogorov–Smirnov_test. Null hypothesis: the samples come from the same distribution. The null hypothesis is rejected if the test statistic is greater than the critical value (see wiki article)
Can you write Python code for it?

from scipy.stats import ks_2samp

ks_2samp(df[df['loc_facility_cd']=='633867']['diff_hours'],df[df['loc_facility_cd']=='4382264']['diff_hours'])

# Critical test statistic at alpha = 0.05: = 1.36 * sqrt((n1+n2)/n1*n2) = 1.36*(sqrt((1775+582)/(1775*582)) = 0.065
# 0.074 > 0.065 -> null hypothesis rejected at level 0.05. --> histograms are different

ks_2samp(df[df['loc_facility_cd']=='4382264']['diff_hours'], df[df['loc_facility_cd']=='4382273']['diff_hours'])

# Critical test statistic at alpha = 0.05: = 1.36 * sqrt((n1+n2)/n1*n2) = 1.36*(sqrt((997+582)/(997*582)) = 0.071
# 0.05 !> 0.071 -> fail to reject null hypothesis at level 0.05. --> histograms are similar

ks_2samp(df[df['loc_facility_cd']=='633867']['diff_hours'],df[df['loc_facility_cd']=='4382273']['diff_hours'])

# Critical test statistic at alpha = 0.05: = 1.36 * sqrt((n1+n2)/n1*n2) = 1.36*(sqrt((1775+997)/(1775*997)) = 0.054
# 0.094 > 0.054 -> null hypothesis rejected at level 0.05. --> histograms are different; p-value indicates they're very different


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Most number of results from 633867, or The Main Hospital
Can you write Python code for it?

df.diff_hours.hist(by=df.loc_desc, bins=300)

# Use locations 4382264, 4382273, 633867

plt.figure(figsize=(12, 6))
df[df['loc_facility_cd']=='633867']['diff_hours'].plot.hist(alpha=0.4, bins=300,normed=True)
df[df['loc_facility_cd']=='4382264']['diff_hours'].plot.hist(alpha=0.4, bins=300,normed=True)
df[df['loc_facility_cd']=='4382273']['diff_hours'].plot.hist(alpha=0.4, bins=300,normed=True)
plt.xlabel('Hospital Stay Durations, days', fontsize=14)
plt.ylabel('Normalized Frequency', fontsize=14)
# plt.legend(['633867', '4382264', '4382273'])
plt.legend(["Main Hospital", "Sattelite Hospital 1", "Sattelite Hospital 2"])
plt.tick_params(labelsize=14)
plt.xlim(0, 1000)


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
Plot both together to see how encounter duration distributions are different
Can you write Python code for it?

plt.figure(figsize = (10,8))
df_rrt.diff_hours.plot.hist(alpha=0.4, bins=400,normed=True)
df_nonrrt.diff_hours.plot.hist(alpha=0.4, bins=800,normed=True)
plt.xlabel('Hospital Stay Durations, hours', fontsize=14)
plt.ylabel('Normalized Frequency', fontsize=14)
plt.legend(['RRT', 'Non RRT'])
plt.tick_params(labelsize=14)
plt.xlim(0, 1000)


I am working on the file "Data Science Notebooks/Notebooks/EDA/encounter_durations[EDA].ipynb".
The first task is:
The notebook Probe_encounter_types_classes explores admit type, class types & counts
Can you write Python code for it?

plt.figure()
df['diff_hours'].plot.hist(bins=500)
plt.xlabel("Hospital Stay Duration, days")
plt.title("Range of stays, patients with RRT")
plt.xlim(0, 2000)


I am working on the file "Data Science Notebooks/Notebooks/EDA/rrt_reasons[EDA].ipynb".
The first task is:
Count the occurrences -- what are the most frequent reasons?
Can you write Python code for it?

df_reasons = find_rrt_freq(reasons)

df_reasons


I am working on the file "Data Science Notebooks/Notebooks/EDA/rrt_reasons[EDA].ipynb".
The first task is:
Number of reasons given (note, not all rrt events have reasons given; we have 2048 rrt events)
Can you write Python code for it?


I am working on the file "Data Science Notebooks/Notebooks/EDA/rrt_reasons[EDA].ipynb".
The first task is:
Query impala for rrt reasons
only looking at valid events (ce.valid_until_dt_tm > 4e12) & complete encounters for inpatients (enc.encntr_type_class_cd='391') at Main Hospital (enc.loc_facility_cd='633867')
Can you write Python code for it?

query = '''
SELECT ce.event_tag
FROM encounter enc
INNER JOIN clinical_event ce
ON enc.encntr_id = ce.encntr_id
WHERE enc.loc_facility_cd='633867'
AND enc.encntr_complete_dt_tm < 4e12
AND ce.event_cd='54408578'
AND ce.result_status_cd NOT IN ('31', '36')
AND ce.valid_until_dt_tm > 4e12
AND ce.event_class_cd not in ('654645')
AND enc.admit_type_cd !='0'
AND enc.encntr_type_class_cd='391';
'''

cur.execute(query)
reasons = cur.fetchall()  # would read result into a list of tuples, e.g. [('Other: iv start',),(...)...]

# look at the result
reasons

# Make the tuple within a list a simple list
reasons = [reason[0] for reason in reasons]

reasons


I am working on the file "Data Science Notebooks/Notebooks/EDA/rrt_reasons[EDA].ipynb".
The first task is:
This means num_only_other/num_rrt_events => % of reasons have only a personnel-specified reason listed
Number of reasons including "Staff Concern" as a reason, by itself & with others
Can you write Python code for it?


I am working on the file "Data Science Notebooks/Notebooks/EDA/rrt_reasons[EDA].ipynb".
The first task is:
function definitions
Can you write Python code for it?

def find_rrt_freq(reasons):
    '''
    reasons is a python list of RRT reasons -- 1 line for each RRT event.
    Each entry is a different RRT event; there may be multiple reasons per event
    output: a pandas dataframe with the counts for each reason.
    '''

    rrt_reasons = defaultdict(int)

    for reason in reasons:
        otherreason = ''
        if reason.lower().startswith("other:"):
            # if the line starts with "other" --> the only reason is the otherreason(s)
            otherreason = reason.lower().split('other:')[1]
            rrts = []
        else:
            # if the line contains "other:" or not
            splitreason = reason.lower().split('other:')
            if len(splitreason) > 1:
                # if an "other" reason exists, process it differently
                otherreason = splitreason[1] # text of the line after 'other:'
            primaryreason = splitreason[0].strip().strip('"')
            rrts = primaryreason.split(',')

        for rrt in rrts:
            rrt = rrt.strip()
            # loop through list of rrt reasons for patient & add to count tracker
            if len(rrt) > 0:
                # included len check b/c splitting on "other" above caused trailing comma
                if rrt not in rrt_reasons.keys():
                    rrt_reasons[rrt] = 1
                else:
                    rrt_reasons[rrt] += 1

        if len(otherreason) > 0:
            # handle the "other" reason(s)
            otherreason = "other: " + otherreason.strip().strip('"')
            if otherreason not in rrt_reasons.keys():
                rrt_reasons[otherreason] = 1
            else:
                rrt_reasons[otherreason] += 1

    return pd.DataFrame(rrt_reasons, index=['count']).transpose().sort_values('count', ascending=False).reset_index()

def count_others(reasons):
    '''
    Count how many "Other" reasons there are, both occurring alone & with other reasons.
    "other_counts" is a dict which contains "only_other" &...


I am working on the file "Data Science UA - September 2017/Lecture 01 - Introduction/US_Baby_Names-2010.ipynb".
The first task is:
Insert prop column for each group
Can you write Python code for it?

def add_prop(group):
    # Integer division floors
    births = group.births.astype(float)

    group['prop'] = births / births.sum()
    return group
names2010 = names2010.groupby(['sex']).apply(add_prop)

names2010


I am working on the file "Data Science UA - September 2017/Lecture 01 - Introduction/US_Baby_Names-2010.ipynb".
The first task is:
Plot proportion of boys and girls names starting in each letter
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

fig, axes = plt.subplots(2, 1, figsize=(10, 8))
letter_prop['M'].plot(kind='bar', rot=0, ax=axes[0], title='Male')
letter_prop['F'].plot(kind='bar', rot=0, ax=axes[1], title='Female',
                      legend=False)


I am working on the file "Data Science UA - September 2017/Lecture 01 - Introduction/US_Baby_Names-2010.ipynb".
The first task is:
Total number of birth in year 2010 by sex
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 01 - Introduction/US_Baby_Names-2010.ipynb".
The first task is:
Verify that the prop clumn sums to 1 within all the groups
Can you write Python code for it?

import numpy as np

np.allclose(names2010.groupby(['sex']).prop.sum(), 1)


I am working on the file "Data Science UA - September 2017/Lecture 01 - Introduction/US_Baby_Names-2010.ipynb".
The first task is:
http://www.ssa.gov/oact/babynames/limits.html
Load file into a DataFrame
Can you write Python code for it?

import pandas as pd

names2010 = pd.read_csv('/resources/yob2010.txt', names=['name', 'sex', 'births'])
names2010


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Compute descriptive statistics for the data:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Compute mean for cardspent for female population only:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Confidence intervals tell us how close we think the mean is to the true value, with a certain level of confidence.
We compute mean mu, standard deviation sigma and the number of observations N in our sample of the debt-to-income ratio:
Can you write Python code for it?

mu, sigma = np.mean(df[['debtinc']]), np.std(df[['debtinc']])
print ("mean = %G, st. dev = %g" % (mu, sigma))

N = len(df[['debtinc']])
N


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Data Exploration
Select 4 data columns for visualizing:
Can you write Python code for it?

viz = df[['cardspent','debtinc','carditems','commutetime']]
viz.head()


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Drop NaN (Not-a-Number) observations:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
In the case of amount spent on primary credit card, we conclude that men tend to charge more on their primary card (p-value = 2e-6 < 0.05, statistically significant).
Can you write Python code for it?

female_debt = adf[adf['gender'] == 'Female']['debtinc']
male_debt   = adf[adf['gender'] == 'Male']['debtinc']
td, pd      = stats.ttest_ind(female_debt, male_debt)
print ("t-test: t = %g  p = %g" % (td, pd))


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
In the case of debt-to-income ratio, we conclude that there is no significant difference between men and women (p-value = 0.758 > 0.05, not statistically significant).
Plot Data
Plot statistical measures for amounts spent on primary credit card
Use boxplot to compare medians, 25% and 75% percentiles, 12.5% and 87.5% percentiles:
Can you write Python code for it?

adf.boxplot(column='cardspent', by='gender', grid=False, showfliers=False)
plt.show()


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Labeling Data
income > 30000  --> High-income --> 1
income < 30000  --> Low-income  --> 0
Can you write Python code for it?

# To label data into high-income and low-income
df['income_category'] = df['annual_income'].map(lambda x: 1 if x>30000 else 0)
df[['annual_income','income_category']].head()


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Plot age vs. income data to find some interesting relationships.
Can you write Python code for it?

plt.scatter(df.age, df.annual_income)
plt.xlabel("Age")
plt.ylabel("Income")
plt.show()


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Print the current version of Python:
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
The 95% confidence interval for the mean of N draws from a Normal distribution with mean mu and standard deviation sigma is
Can you write Python code for it?

conf_int = stats.norm.interval( 0.95, loc = mu, scale = sigma/np.sqrt(N) )
conf_int

print ("95%% confidence interval for the mean of debt to income ratio = [%g %g]") % (conf_int[0], conf_int[1])


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
To download the data, we will use !wget (on DataScientistWorkbench)
Can you write Python code for it?

if sys.platform.startswith('linux'):
    !wget -O /resources/customer_dbase_sel.csv http://analytics.romanko.ca/data/customer_dbase_sel.csv


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Understanding the Data
customer_dbase_sel.csv:
We have downloaded an extract from IBM SPSS sample dataset with customer data, customer_dbase_sel.csv, which contains customer-specific data such as age, income, credit card spendings, commute type and time, etc. Dataset source

custid e.g. 0648-AIPJSP-UVM (customer id)
gender e.g. Female or Male
age e.g. 26
debtinc e.g. 11.1 (debt to income ratio in %)
card e.g. Visa, Mastercard (type of primary credit card)
carditems e.g. 1, 2, 3 ... (# of primary credit card purchases in the last month)
cardspent e.g 228.27 (amount in \$ spent on the primary credit card last month)
commute e.g. Walk, Car, Bus (commute type)
commutetime e.g. 22 (time in minutes to commute to work)
income e.g. 16.00 (income in thousands \$ per year)
edcat e.g. College degree, Post-undergraduate degree (education level)

Reading the data in
Can you write Python code for it?

url = "http://analytics.romanko.ca/data/customer_dbase_sel.csv"
df = pd.read_csv(url)

## On DataScientistWorkbench you can read from /resources directory
#df = pd.read_csv("/resources/customer_dbase_sel.csv")

# display first 5 rows of the dataset
df.head()


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
Visualize data:
Can you write Python code for it?

viz.hist()
plt.show()

df[['cardspent']].hist()
plt.show()

df[['commutetime']].hist()
plt.show()


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Basic_statistics.ipynb".
The first task is:
We have seen above that the mean cardspent and debtinc in the male and female populations were different. To test if this is significant, we do a 2-sample t-test with scipy.stats.ttest_ind():
Can you write Python code for it?

female_card = adf[adf['gender'] == 'Female']['cardspent']
male_card = adf[adf['gender'] == 'Male']['cardspent']
tc, pc = stats.ttest_ind(female_card, male_card)
print ("t-test: t = %g  p = %g" % (tc, pc))


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
For continuous probability density functions, we can use pdf() to check the probability density at a given x value. For discrete distributions like the binomial, we can use stats.distribution.pmf(), the probability mass function to check the proportion of observations at given number of successes k.
Can you write Python code for it?

print(stats.binom.pmf(k=5,        # Probability of k = 5 successes
                      n=10,       # With 10 flips
                      p=0.5))     # And success probability 0.5

print(stats.binom.pmf(k=8,        # Probability of k = 8 successes
                      n=10,       # With 10 flips
                      p=0.8))     # And success probability 0.8


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
In the 10,000 trails generated, the longest it took to get a heads was 13 flips.
We can use cdf() to check the probability of needing 3 flips or more to get a success:
Can you write Python code for it?

first_three = stats.geom.cdf(k=3,   # Prob of success in first 5 flips
                           p=0.5)

1 - first_three

#Use pmf() to check the probability of seeing a specific number of flips before a successes
stats.geom.pmf(k=2,   # Prob of needing exactly 2 flips to get the first success
              p=0.5)


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
Notes:
The plot above is an approximation of the theoretical uniform distribution because it is based on a sample of observations: we generated 10,000 data points from a uniform distribution spanning the range 0 to 20.
In the density plot, we see that the density of sample data is almost level, i.e., all values have the same probability of occurring.
Note: the area under a probability density curve is always equal to 1.
More useful scipy functions:

stats.distribution.rvs() generates random numbers from the specified distribution. The arguments to rvs() vary depending on the type of distribution considered, e.g., the starting and ending points and the sample size for a uniform distribution.

 stats.distribution.cdf() returns the probability that an observation drawn from a distribution is below a specified value (a.k.a., the cumulative distribution function) calculated as the area under the distribution's density curve to the left of the specified value (on the x axis). For example, in the uniform distribution above, there is a 25% chance that an observation will be in the range 0 to 5 and a 75% chance it will fall in the range 5 to 20. We can confirm this with cdf():
Can you write Python code for it?

stats.uniform.cdf(x     = 5,       # quantile to check
                  loc   = 0,       # start value of the distribution
                  scale = 20)      # end value of the distribution


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
The Geometric and Exponential Distributions
The geometric distribution is discrete and models the number of trials it takes to achieve a success in repeated experiments with a given probability of success. The exponential distribution is continuous and models the amount of time before an event occurs given a certain occurrence rate.
The scipy nickname for the geometric distribution is "geom". Below we model the number of trials it takes to get a success (heads) when flipping a fair coin:
Can you write Python code for it?

random.seed(12)

flips_till_heads = stats.geom.rvs(size=10000,  # Generate geometric data
                                  p=0.5)       # With success prob 0.5

# Print table of counts
print( pd.crosstab(index="counts", columns= flips_till_heads))

# Plot histogram
pd.DataFrame(flips_till_heads).hist(range=(-0.5,max(flips_till_heads)+0.5)
                                    , bins=max(flips_till_heads)+1)


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
The Normal Distribution
The normal (or Gaussian distribution) is a continuous probability distribution with a bell-shaped curve and is characterized by its center point (mean) and spread (standard deviation). Most observations from a normal distribution lie close to the mean, i.e., about 68% of the data lies within 1 standard deviation of the mean, 95% lies within 2 standard deviations and 99.7% within 3 standard deviations.
Note that many common statistical tests assume distributions are normal. (In the scipy module the normal distribution is referred to as norm.)
Can you write Python code for it?

prob_under_minus1 = stats.norm.cdf(x     = -1,
                                   loc   = 0,
                                   scale = 1)

prob_over_1 = 1 - stats.norm.cdf(x= 1,
                                loc = 0,
                                scale= 1)

between_prob = 1 - (prob_under_minus1 + prob_over_1)

print(prob_under_minus1, prob_over_1, between_prob)


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
The Poisson Distribution
The Poisson distribution models the probability of seeing a certain number of successes within a time interval, where the time it takes for the next success is modeled by an exponential distribution. The Poisson distribution can be used to model the number of arrivals a hospital can expect in a hour's time, the number of emailsone can expect to receive in a day, etc.
The scipy name for the Poisson distribution is "poisson". Below is a plot for data from a Poisson distribution with an arrival rate of 1 per time unit:
Can you write Python code for it?

random.seed(12)

arrival_rate_1 = stats.poisson.rvs(size=10000,  # Generate Poisson data
                                   mu=1 )       # Average arrival time 1

# Print table of counts
print( pd.crosstab(index="counts", columns= arrival_rate_1))

# Plot histogram
pd.DataFrame(arrival_rate_1).hist(range=(-0.5,max(arrival_rate_1)+0.5)
                                    , bins=max(arrival_rate_1)+1)


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
The binomial distribution is discrete and can be summarized with a frequency table (and  a histogram). The histogram shows us that a binomial distribution with a 50% probability of success is roughly symmetric, but this is not the case in general.
Can you write Python code for it?

biased_coin_flips = stats.binom.rvs(n=10,      # Number of flips per trial
                                  p=0.8,       # Success probability
                                  size=10000)  # Number of trials

# Print table of counts
print( pd.crosstab(index="counts", columns= biased_coin_flips))

# Plot histogram
pd.DataFrame(biased_coin_flips).hist(range=(-0.5,10.5), bins=11)

#cdf() check the probability of achieving a number of successes within a certain range

print(stats.binom.cdf(k=5,        # Probability of k = 5 successes or less
                n=10,       # With 10 flips
                p=0.8))# And success probability 0.8

print(1 - stats.binom.cdf(k=8,        # Probability of k = 9 successes or more
                    n=10,       # With 10 flips
                    p=0.8))    # And success probability 0.8


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
The figures above show that approximately 5% of the data is further than 2 standard deviations from the mean.
The Binomial Distribution
The binomial distribution is a discrete probability distribution that models the outcomes of a given number of random trails of some experiment or event. The binomial distribution has two parameters: the probability of success in a trial and the number of trials. The binomial distribution represents the likelihood to achieve a given number of successes in n trials of an experiment. We could model flipping a fair coin 10 times with a binomial distribution where the number of trials is set to 10 and the probability of success is set to 0.5.
The scipy name for the binomial is binom.
Can you write Python code for it?

fair_coin_flips = stats.binom.rvs(n=10,        # Number of flips per trial
                                  p=0.5,       # Success probability
                                  size=10000)  # Number of trials

print( pd.crosstab(index="counts", columns= fair_coin_flips))

pd.DataFrame(fair_coin_flips).hist(range=(-0.5,10.5), bins=11)


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
The plot above shows the bell shape of the normal distribution as well as the area below, above and within one standard deviation of the mean. One can also check the quantiles of a normal distribution with stats.norm.ppf():
Can you write Python code for it?

print( stats.norm.ppf(q=0.025) ) # Find the quantile for the 2.5% cutoff

print( stats.norm.ppf(q=0.975) ) # Find the quantile for the 97.5% cutoff


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
We can use cdf() to check the probability of achieving more (or less) than a certain number of successes and pmf() to check the probability of obtaining a specific number of successes:
Can you write Python code for it?

print(stats.poisson.cdf(k=5,     # Check the probability of 5 arrivals or less
                  mu=10))  # With arrival rate 10

print(stats.poisson.pmf(k=10,     # Check the prob f exactly 10 arrivals
                  mu=10))   # With arrival rate 10


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
stats.distribution.pdf() returns the probability density (height of the distribution) at a given x value. Note that all values within the range of a uniform distribution have the same probability density and all values outside the range of the distribution have a probability density of 0.

Probability distribution functions in scipy also support median(), mean(), var() and std().
Can you write Python code for it?

for x in range(-2,24,4):
    print("Density at x value " + str(x))
    print( stats.uniform.pdf(x, loc = 0, scale = 20) )


I am working on the file "Data Science UA - September 2017/Lecture 03 - Basic Statistics/Overview_of_distributions.ipynb".
The first task is:
stats.distribution.ppf() is the inverse of cdf(): it returns the quantile (x axis cutoff value) associated with a given probability. For example, to get the cutoff value for which there is a 20% chance of drawing an observation below that value, we can use ppf():
Can you write Python code for it?

stats.uniform.ppf(  q   = 0.2,     # Probability cutoff
                  loc   = 0,       # start value of the distribution
                  scale = 20)      # end value of the distribution


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Finding a Root of a Function - Bisection and Newton Methods.ipynb".
The first task is:
Bisection method
Given a continuous function $f(x)$ and two values of $x_1$, $x_2$ such that $f(x_1)$ and $f(x_2)$ have opposite signs the Bisection method is a root-finding method that repeatedly bisects the interval $[x_1, x_2]$ and then selects a subinterval (in which a root must be) for further processing. (Since $f(x_1)$ and $f(x_2)$ have opposite signs, it follows that $f(x)$ is zero somewhere between $x_1$ and $x_2$.) The Bisection method iterate towards the zero of the function by cutting the root search interval in half at every iteration. The method calculates the middle point $x_m$ between $x_1$ and $x_2$ and compute $f(x_m)$ and then replaces either $x_1$ or $x_2$ by $x_m$ such the values of $f$ at the end points of the interval are of opposite signs. The process is repeated until the interval is small enough that its middle point can be considered a good approximation of the root of the function. In summary, the algorithm works as follows:

Compute $f(x_1)$ and $f(x_2)$
Compute $x_m = \frac{1}{2}(x_1 + x_2)$.
Compute $f(x_m)$.
If $f(x_m)f(x_2) < 0$, replace $x_1$ by $x_m$, otherwise, replace $x_2$ by $x_m$.
If $|x_1 - x_2|<\varepsilon$, where $\varepsilon$ is a user-specified tolerance, return $\frac{1}{2}(x_1 + x_2)$, otherwise return to step 2.

Example: let $f(x)$ be $\frac{1}{2}-\text{e}^{-x}$ and $x_1$  and $x_2$ be 0 and 4, respectively. Notice that $f(x)$ has a zero somewhere on the plotted interval.
Can you write Python code for it?

def exponential_function(x):
    return 0.5 - np.exp(-x)

x = np.linspace(0, 4, 100)
y = exponential_function(x)
plt.plot(x, y)
plt.axhline(0, color='r', ls='--')


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Finding a Root of a Function - Bisection and Newton Methods.ipynb".
The first task is:
Plotting the root
We plot the function $f(x)=x+2\cos(x)$ for $x$ going from -2 to 4, and on the same graph, we also plot a red dot at the location where $f(x)=0$.
Can you write Python code for it?

from scipy.optimize import fsolve

def g(x):
    return x + 2 * np.cos(x)

x = np.linspace(-2, 4, 100)
x0 = fsolve(g, 1)
plt.plot(x, g(x))
plt.plot(x0, g(x0), 'ro')
plt.axhline(y=0, color='r')


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Finding a Root of a Function - Bisection and Newton Methods.ipynb".
The first task is:
We test newtonsmethod by finding the root of $f(x)=\frac{1}{2}-\text{e}^{-x}$ using $x_0=1$ as the starting point of the search. How many iterations do we need if we start at $x=4$?
Can you write Python code for it?

def fp(x):
    return np.exp(-x)

xs = 1
func = exponential_function
funcp = fp
tol = 1e-6
nmax = 10

xzero, iterations = newtonsmethod(func, funcp, xs, tol, nmax)
print("First Example")
if xzero != None:
    print("Starting search from x = %.2f" % xs)
    print("root at x = %f, exponential_function(root) = %f" % (xzero, exponential_function(xzero)))
    print("tolerance reached in %d iterations" % iterations)
else:
    print("Starting search from x = %.2f" % xs)
    print('Max number of iterations reached before convergence')

print("")

xs = 4
nmax = 40

xzero, iterations = newtonsmethod(func, funcp, xs, nmax)

print("Second Example")
if xzero != None:
    print("Starting search from x = %.2f" % xs)
    print("root at x = %f, exponential_function(root) = %f" % (xzero, exponential_function(xzero)))
    print("tolerance reached in %d iterations" % iterations)
else:
    print("Starting search from x = %.2f" % xs)
    print('Max number of iterations reached before convergence')


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Finding a Root of a Function - Bisection and Newton Methods.ipynb".
The first task is:
We use the bisection method to find the root of the $exponential_function$ defined above
Can you write Python code for it?

x1 = 0
x2 = 4
function = exponential_function
xzero = bisection(function, x1, x2, tol=1e-3, nmax=20, silent=True)
print ("The root of exponential_function between %.2f and %.2f is %f" % (x1, x2, xzero))
print ("The value of the function at the 'root' is %f" % exponential_function(xzero))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Finding a Root of a Function - Bisection and Newton Methods.ipynb".
The first task is:
and of $cos$  between 0 and 3.
Can you write Python code for it?

x1 = 0
x2 = 3
function = np.cos
root = bisection(function, 0, 3, tol=1e-6, nmax=30)
print ("The root of cos between %.2f and %.2f is %f" % (x1, x2, root))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Alternatively we can use an LU decomposition.  Notice the decomposition is only performed once.
Can you write Python code for it?

lufactors = la.lu_factor(A)
print("LU solution for b1: %s" % np.array_str(la.lu_solve(lufactors, b1)))
print("LU solution for b2: %s" % np.array_str(la.lu_solve(lufactors, b2)))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
By default we are given the permutation matrix $P$ and the lower and upper triangular matrices that, when recombined, produce $\mat A$.  This function is good when we want to see the decomposition in a form easy for us to read.  If we want to use the decomposition for solving systems of linear equations the information could be stored in a more efficient form for the computer's use.
Note that our choice of using generic arrays for storing matrices comes at a cost when we want to multiply many of them together.  Here we have had to use nested calls to np.dot.  This can be slightly streamlined, but remains tedious.  It is just a price we must pay for our choice.  In practice it is usually a rather small price.
Getting back to solving a system of equations, we can use scipy.linalg.lu_factor to decompose in a form more useful for the computer.
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Factoring the system
The work required to solve the system of equations mostly involves manipulating the matrix, $A$, and then performing the same manipulations on the right hand side of the equations, $b$.  We could instead have many right hand sides (a two dimensional array with multiple columns, one for each set of values for which we want to find a solution).  This is handled by solve.  Alternatively, we can decompose $A$ into pieces that encode the required manipulations using the LU decomposition.  The decomposition only needs to be performed once, it can then be applied whenever needed.  Finally, for numerical stability we should also pivot the matrix with permutation matrix $P$.  LU decomposition with pivoting is provided by scipy.linalg.lu.
Can you write Python code for it?

(P, L, U) = la.lu(A)

if (sys.version_info > (3, 0)):
    # Python 3 code in this block
    print("Permutation matrix:\n", P)
    print("Lower triangular matrix:\n", L)
    print("Upper triangluar matrix:\n", U)
    print("PLU == A?", np.allclose(np.dot(P,np.dot(L,U)), A))
else:
    # Python 2 code in this block
    print("Permutation matrix:\n%s" % np.array_str(P))
    print("Lower triangular matrix:\n%s" % np.array_str(L))
    print("Upper triangluar matrix:\n%s" % np.array_str(U))
    print("PLU == A? %r" % np.allclose(np.dot(P,np.dot(L,U)), A))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Here $L$ and $U$ have been merged into a single matrix (notice in the form above that $L$ had ones along the diagonal so they do not need to be stored) and the permutations are represented by a one dimensional array instead a matrix.  This is much more memory efficient, but is also much harder for us to read.  Even so, this can be used in scipy.linalg.lu_solve.  In fact, the tuple returned here is exactly what needs to be provided as the first argument to that function.
Can you write Python code for it?

lufactors = la.lu_factor(A)
xlu = la.lu_solve(lufactors, b)
print("LU solution: %s" % np.array_str(xlu))
print("Consistent with previous solution? %r" % np.allclose(xlu, x))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Notice that there are a few functions related to the LU decomposition.  We will learn why this is the case below.
Sample linear system
Consider a sample linear system of equations
$$ \begin{align}
10 x_1 - 7 x_2 + x_3 &= 8, \
-3 x_1 + 2 x_2 + 6 x_3 &= 4, \
5 x_1 - x_2 + 5 x_3 &= 6 .
\end{align} $$
We can write this in the form
$$\def\mat#1{\mathsf{#1}}
A x = b, $$
where
$$ A = \left( \begin{array}{rrr}
10 & -7 & 1 \
-3 & 2 & 6 \
5 & -1 & 5
\end{array} \right)
\quad\mathrm{and}\quad
b = \left( \begin{array}{c} 8 \ 4 \ 6 \end{array} \right). $$
We can create these as arrays using np.array.  Notice that we force the result to be a floating point array, not an integer array, by making any one of the entries a floating point number.  NumPy tries to use the "simplest" data type when it creates an array.  There are a few ways to force the type it chooses, this is one way.
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Notice what happened: the first component of $b$ multiplied the first column of $A$, the second component of $b$ multiplied the second column of $A$, and similarly for the third.  This is what we mean by component-wise multiplication.
If we want to perform matrix multiplication we need to use a function, np.dot.  Consider
Can you write Python code for it?

if (sys.version_info > (3, 0)):
    # Python 3 code in this block
    print('np.dot(A, x) =', np.dot(A,x))
else:
    # Python 2 code in this block
    print("np.dot(A, x) = %s" % np.array_str(np.dot(A,x)))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Solving the System
To solve this system we can, not surprisingly, use solve.
Can you write Python code for it?


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
Verifying the Solution
To verify the solution we can directly evaluate $A x$ and compare it to $b$.  Of course this only verifies that the solve has worked correctly, it does not verify that we have entered $A$ and $b$ correctly.  There are two steps, one is to actually perform the multiplication and the second is to compare the $A x$ to $b$.
Matrix Multiplication
To multiply matrices we do not use '*' .  The usual multiplication is component-wise multiplication.  This is not the same as matrix multiplication.
To see this, consider using the usual multiplication:
Can you write Python code for it?

if (sys.version_info > (3, 0)):
    # Python 3 code in this block
    print('A =', A)
    print('x =', x)
    print('A*x =', A*x)
else:
    # Python 2 code in this block
    print("A = %s" % np.array_str(A))
    print("x = %s" % np.array_str(x))
    print("A*x = %s" % np.array_str(A*x))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
We can solve this directly using solve.
Can you write Python code for it?

print("Solution for b1: %s" % np.array_str(la.solve(A, b1)))
print("Solution for b2: %s" % np.array_str(la.solve(A, b2)))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
We get an incompatible dimensions error because b should have the same number of rows as A, not the same number of columns.  This is easy to fix, just take the transpose of b.  There is shorthand for doing this as shown below.
Can you write Python code for it?

print("Original b:\n%s" % np.array_str(b))
print("Transpose of b:\n%s" % np.array_str(b.T))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Linear_Equations.ipynb".
The first task is:
With this we find:
Can you write Python code for it?

print("scipy.linalg.solve:\n%s" % np.array_str(la.solve(A, b.T)))
print("scipy.linalg.lu_solve:\n%s" % np.array_str(la.lu_solve(lufactors, b.T)))


I am working on the file "Data Science UA - September 2017/Lecture 04 - Overview of Linear Algebra and Matrix Computations/Matrix_Computations.ipynb".
The first task is:
Alternatively, we can cast them into matrix objects and use the "*" operator:
Can you write Python code for it?