I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment01.ipynb".
The first task is:
```
3. Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, square feet, and # of bathrooms) but we can also consider transformations of existing variables e.g. the log of the square feet or even "interaction" variables such as the product of bedrooms and bathrooms. Add 4 new variables in both your train_data and test_data.
‘bedrooms_squared’ = ‘bedrooms’‘bedrooms’
‘bed_bath_rooms’ = ‘bedrooms’‘bathrooms’
‘log_sqft_living’ = log(‘sqft_living’)
‘lat_plus_long’ = ‘lat’ + ‘long’
Before we continue let’s explain these new variables:
Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this variable will mostly affect houses with many bedrooms.
Bedrooms times bathrooms is what's called an "interaction" variable. It is large when both of them are large.
Taking the log of square feet has the effect of bringing large values closer together and spreading out small values.
Adding latitude to longitude is non-sensical but we will do it anyway (you'll see why)
For those students not using SFrames you should first download and import the training and testing data sets provided and then add the four new variables each to both data sets (training and testing)
```
Can you write Python code for it?

train_data['bedrooms_squared'] = train_data['bedrooms']*train_data['bedrooms']
train_data['bed_bath_rooms'] = train_data['bedrooms']*train_data['bathrooms']
train_data['log_sqft_living'] = np.log(train_data['sqft_living'])
train_data['lat_plus_long'] = train_data['lat']+train_data['long']

test_data['bedrooms_squared'] = test_data['bedrooms']*test_data['bedrooms']
test_data['bed_bath_rooms'] = test_data['bedrooms']*test_data['bathrooms']
test_data['log_sqft_living'] = np.log(test_data['sqft_living'])
test_data['lat_plus_long'] = test_data['lat']+test_data['long']

train_data[:3]


I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment01.ipynb".
The first task is:
three estimated models compute the RSS on the Testing data
Can you write Python code for it?

calcRSS(model1, test_data[model1_features], test_data['price'])

calcRSS(model2, test_data[model2_features], test_data['price'])

calcRSS(model3, test_data[model3_features], test_data['price'])


I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment01.ipynb".
The first task is:
what are the mean (arithmetic average) values of your 4 new variables on TEST data? (round to 2 digits)
Can you write Python code for it?

print(np.average(test_data['bedrooms_squared']))
print(np.average(test_data['bed_bath_rooms']  ))
print(np.average(test_data['log_sqft_living'] ))
print(np.average(test_data['lat_plus_long']))

model1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat','long']
model2_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat','long', 'bed_bath_rooms']
model3_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat','long', 'bed_bath_rooms','bedrooms_squared', 'log_sqft_living','lat_plus_long']

model1 = gl.linear_regression.create(train_data, target='price', features=model1_features, validation_set = None)
model2 = gl.linear_regression.create(train_data, target='price', features=model2_features, validation_set = None)
model3 = gl.linear_regression.create(train_data, target='price', features=model3_features, validation_set = None)


I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment02.ipynb".
The first task is:
What is the sign (positive or negative) for the coefficient/weight for ‘bathrooms’ in Model 1?
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment02.ipynb".
The first task is:
What is the sign (positive or negative) for the coefficient/weight for ‘bathrooms’ in Model 2?
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment02.ipynb".
The first task is:
```
3. Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, square feet, and # of bathrooms) but we can also consider transformations of existing variables e.g. the log of the square feet or even "interaction" variables such as the product of bedrooms and bathrooms. Add 4 new variables in both your train_data and test_data.
‘bedrooms_squared’ = ‘bedrooms’‘bedrooms’
‘bed_bath_rooms’ = ‘bedrooms’‘bathrooms’
‘log_sqft_living’ = log(‘sqft_living’)
‘lat_plus_long’ = ‘lat’ + ‘long’
Before we continue let’s explain these new variables:
Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this variable will mostly affect houses with many bedrooms.
Bedrooms times bathrooms is what's called an "interaction" variable. It is large when both of them are large.
Taking the log of square feet has the effect of bringing large values closer together and spreading out small values.
Adding latitude to longitude is non-sensical but we will do it anyway (you'll see why)
For those students not using SFrames you should first download and import the training and testing data sets provided and then add the four new variables each to both data sets (training and testing)
```
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week02_Polynomial_Regression_Assignment02.ipynb".
The first task is:
three estimated models compute the RSS on the Testing data
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Above fit looks pretty wild, here's a clear example of how overfitting is associated with very large magnitute estimated coefficient.
Ridge Regression
Can you write Python code for it?

def polynomial_ridge_regression(data, deg, l2_penalty=0):

    model = gl.linear_regression.create(polynomial_features(data,deg),  target='Y',
                                        validation_set=None, l2_penalty=l2_penalty, l1_penalty=0,
                                        verbose=False)
    return model


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Computing y = sin(4x)
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Fitting a deg 2 polynomial
Can you write Python code for it?

model = polynomial_regression(data, deg=2)

print_coefficients(model)

plot_poly_predictions(data, model)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Fitting a deg 4
Can you write Python code for it?

model = polynomial_regression(data, deg=4)
print_coefficients(model)
plot_poly_predictions(data, model)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Fitting deg 16
Can you write Python code for it?

model = polynomial_regression(data, deg=16)
print_coefficients(model)
plot_poly_predictions(data, model)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Perform a ridge fit of a degree polynimial using a "good" penalty strength
We use the leave one out (LOO) cross validation to choose the penalty value
Can you write Python code for it?

def loo(data, deg, penalty_values):
    polynomial_features(data, deg)

    num_folds = len(data)
    folds = gl.cross_validation.KFold(data, num_folds)

    #for each value of penalty, fit a model for each fold and compute avg MSE
    l2_penalty_mse = []
    min_mse=None
    best_l2_penalty = None

    for l2_penalty in penalty_values:
        next_mse=0.0
        for train, validation in folds:
            #train model
            model = gl.linear_regression.create(train, target='Y',
                                               l2_penalty=l2_penalty,
                                               validation_set=None, verbose=False)
            y_pred = model.predict(validation)
            next_mse = next_mse+ ((y_pred -validation['Y'])**2).sum()

        #save sqrd error in list of MSE
        next_mse = next_mse/num_folds  #taking avg of mse
        l2_penalty_mse.append(next_mse)
        #finding out the min and best mse

        if min_mse is None or next_mse < min_mse:
            min_mse = next_mse
            best_l2_penalty = l2_penalty

    return l2_penalty_mse, best_l2_penalty


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
Run LOO cross validation for num values of lambda on a log scale
Can you write Python code for it?

l2_penalty_values = np.logspace(-4, 10, num=10)
l2_penalty_mse, best_l2_penalty = loo(data, 16, l2_penalty_values)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression.ipynb".
The first task is:
creating 30 observation between 0,1
Can you write Python code for it?

random.seed(98103)
n=30
x = gl.SArray([random.random() for i in range(30)]).sort()


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
After designating one of the k segments as the validation set, we train a model using the rest of the data. To choose the remainder, we slice (0:start) and (end+1:n) of the data and paste them together. SFrame has append() method that pastes together two disjoint sets of rows originating from a common dataset. For instance, the following cell pastes together the first and last two rows of the train_valid_shuffled dataframe.
Can you write Python code for it?

n = len(train_valid_shuffled)
first_two = train_valid_shuffled[0:2]
last_two = train_valid_shuffled[n-2:n]
print first_two.append(last_two)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
As in Week 3, we will use the sqft_living variable. For plotting purposes (connecting the dots), you'll need to sort by the values of sqft_living. For houses with identical square footage, we break the tie by their prices.
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
Let's use matplotlib to visualize what a polynomial regression looks like on the house data.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

sales = gl.SFrame('data/kc_house_data.gl/')


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
Next, fit a 15th degree polynomial on set_1, set_2, set_3, and set_4, using 'sqft_living' to predict prices. Print the weights and make a plot of the resulting model.
Hint: When calling graphlab.linear_regression.create(), use the same L2 penalty as before (i.e. l2_small_penalty).  Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set = None in this call.
Can you write Python code for it?

poly15_set_1 = polynomial_sframe(set_1['sqft_living'], 15)
poly15_set_1_names = poly15_set_1.column_names()
poly15_set_1['price'] = set_1['price']

#print(poly15_set_1.head(2))

model15_set_1 = gl.linear_regression.create(poly15_set_1, target = 'price', l2_penalty=l2_small_penalty ,
                                            features = poly15_set_1_names, validation_set = None,
                                            verbose=False)

plt.plot(poly15_set_1['power_15'], poly15_set_1['price'], '.',
         poly15_set_1['power_15'], model15_set_1.predict(poly15_set_1), '-', linewidth=2)
plt.grid(True)

coeff=model15_set_1.get('coefficients')
print(coeff[coeff['name']=='power_1'])

poly15_set_2 = polynomial_sframe(set_2['sqft_living'], 15)
poly15_set_2_names = poly15_set_2.column_names()
poly15_set_2['price'] = set_2['price']

model15_set_2 = gl.linear_regression.create(poly15_set_2, target = 'price', l2_penalty=l2_small_penalty,
                                            features=poly15_set_2_names,
                                            validation_set = None, verbose=False)

plt.plot(poly15_set_2['power_15'], poly15_set_2['price'], '.',
        poly15_set_2['power_15'], model15_set_2.predict(poly15_set_2), '-', linewidth=2)
plt.grid(True)

coeff=model15_set_2.get('coefficients')
print(coeff[coeff['name']=='power_1'])

poly15_set_3 = polynomial_sframe(set_3['sqft_living'], 15)
poly15_set_3_names = poly15_set_3.column_names()
poly15_set_3['price'] = set_3['price']

model15_set_3 = gl.linear_regression.create(poly15_set_3, target = 'price', l2_penalty=l2_small_penalty,
                                            features = poly15_set_3_names,
                                            validation_set = None, verbose=False)

plt.plot(poly15_set_3['power_15'], poly15_set_3['price'], '.',
        poly15_set_3['power_15'], model15_set_3.predict(poly15_set_3), '-', linewidth=2)
plt.grid(True)

coeff=model15_set_3.get('coefficients')
print(coeff[coeff['name']=='power_1'])

poly15_set_4 =...


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
Note: When we have so many features and so few data points, the solution can become highly numerically unstable, which can sometimes lead to strange unpredictable results.  Thus, rather than using no regularization, we will introduce a tiny amount of regularization (l2_penalty=1e-5) to make the solution numerically stable.  (In lecture, we discussed the fact that regularization can also help with numerical stability, and here we are seeing a practical example.)
With the L2 penalty specified above, fit the model and print out the learned weights.
Hint: make sure to add 'price' column to the new SFrame before calling graphlab.linear_regression.create(). Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set=None in this call.
Can you write Python code for it?

model = gl.linear_regression.create(poly15_data, target='price', features=poly15_columns,
                                   l2_penalty=1e-5,
                                   validation_set=None, verbose=False)
model.coefficients


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
Now let us extract individual segments with array slicing. Consider the scenario where we group the houses in the train_valid_shuffled dataframe into k=10 segments of roughly equal size, with starting and ending indices computed as above.
Extract the fourth segment (segment 3) and assign it to a variable called validation4.
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
Once we have a function to compute the average validation error for a model, we can write a loop to find the model that minimizes the average validation error. Write a loop that does the following:
* We will again be aiming to fit a 15th-order polynomial model using the sqft_living input
* For l2_penalty in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, you can use this Numpy function: np.logspace(1, 7, num=13).)
    * Run 10-fold cross-validation with l2_penalty
* Report which L2 penalty produced the lowest average validation error.
Note: since the degree of the polynomial is now fixed to 15, to make things faster, you should generate polynomial features in advance and re-use them throughout the loop. Make sure to use train_valid_shuffled when generating polynomial features!
Can you write Python code for it?

k = 10
data = polynomial_sframe(train_valid_shuffled['sqft_living'],15)
features_name = data.column_names()
data['price'] = train_valid_shuffled['price']
min_err=None
best_l2_penalty=None
l2_penalty_list=np.logspace(1, 7, num=13)
l2_error=[]
for l2_penalty in l2_penalty_list:
    error = k_fold_cross_validation(k, l2_penalty, data, 'price', features_list=features_name)
    l2_error.append(error)
    if min_err is None or min_err > error:
        min_err = error
        best_l2_penalty = l2_penalty
print min_err, best_l2_penalty


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
QUIZ QUESTION:  What's the learned value for the coefficient of feature power_1?
Observe overfitting
Recall from Week 3 that the polynomial fit of degree 15 changed wildly whenever the data changed. In particular, when we split the sales data into four subsets and fit the model of degree 15, the result came out to be very different for each subset. The model had a high variance. We will see in a moment that ridge regression reduces such variance. But first, we must reproduce the experiment we did in Week 3.
First, split the data into split the sales data into four subsets of roughly equal size and call them set_1, set_2, set_3, and set_4. Use .random_split function and make sure you set seed=0.
Can you write Python code for it?

(semi_split1, semi_split2) = sales.random_split(.5,seed=0)
(set_1, set_2) = semi_split1.random_split(0.5, seed=0)
(set_3, set_4) = semi_split2.random_split(0.5, seed=0)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
QUIZ QUESTION: Using the best L2 penalty found above, train a model using all training data. What is the RSS on the TEST data of the model you learn with this L2 penalty?
Can you write Python code for it?

test_data = polynomial_sframe(test['sqft_living'],15)
test_data['price'] = test['price']
RSS = np.sum(np.square(test['price'] - model.predict(test_data)))
print RSS


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
The four curves should differ from one another a lot, as should the coefficients you learned.
QUIZ QUESTION:  For the models learned in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature power_1?  (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)
Ridge regression comes to rescue
Generally, whenever we see weights change so much in response to change in data, we believe the variance of our estimate to be large. Ridge regression aims to address this issue by penalizing "large" weights. (Weights of model15 looked quite small, but they are not that small because 'sqft_living' input is in the order of thousands.)
With the argument l2_penalty=1e5, fit a 15th-order polynomial model on set_1, set_2, set_3, and set_4. Other than the change in the l2_penalty parameter, the code should be the same as the experiment above. Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set = None in this call.
Can you write Python code for it?

l2_penalty=1e5
poly15_set_1 = polynomial_sframe(set_1['sqft_living'], 15)
poly15_set_1_names = poly15_set_1.column_names()
poly15_set_1['price'] = set_1['price']

#print(poly15_set_1.head(2))

model15_set_1 = gl.linear_regression.create(poly15_set_1, target = 'price', l2_penalty=l2_penalty ,
                                            features = poly15_set_1_names, validation_set = None,
                                            verbose=False)

plt.plot(poly15_set_1['power_15'], poly15_set_1['price'], '.',
         poly15_set_1['power_15'], model15_set_1.predict(poly15_set_1), '-', linewidth=2)
plt.grid(True)

coeff=model15_set_1.get('coefficients')
print(coeff[coeff['name']=='power_1'])

poly15_set_2 = polynomial_sframe(set_2['sqft_living'], 15)
poly15_set_2_names = poly15_set_2.column_names()
poly15_set_2['price'] = set_2['price']

model15_set_2 = gl.linear_regression.create(poly15_set_2, target = 'price', l2_penalty=l2_penalty,
                                            features=poly15_set_2_names,
                                            validation_set = None, verbose=False)

plt.plot(poly15_set_2['power_15'], poly15_set_2['price'], '.',
        poly15_set_2['power_15'], model15_set_2.predict(poly15_set_2), '-', linewidth=2)
plt.grid(True)

coeff=model15_set_2.get('coefficients')
print(coeff[coeff['name']=='power_1'])

poly15_set_3 = polynomial_sframe(set_3['sqft_living'], 15)
poly15_set_3_names = poly15_set_3.column_names()
poly15_set_3['price'] = set_3['price']

model15_set_3 = gl.linear_regression.create(poly15_set_3, target = 'price', l2_penalty=l2_penalty,
                                            features = poly15_set_3_names,
                                            validation_set = None, verbose=False)

plt.plot(poly15_set_3['power_15'], poly15_set_3['price'], '.',
        poly15_set_3['power_15'], model15_set_3.predict(poly15_set_3), '-', linewidth=2)
plt.grid(True)

coeff=model15_set_3.get('coefficients')
print(coeff[coeff['name']=='power_1'])

poly15_set_4 =...


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the data with fourth segment excluded. When rounded to nearest whole number, the average should be $539,450.
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment01.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the fourth segment. When rounded to nearest whole number, the average should be $536,234.
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
Compute the RSS on the TEST data for the following three sets of weights:
1. The initial weights (all zeros)
2. The weights learned with no regularization
3. The weights learned with high regularization
Which weights perform best?
QUIZ QUESTIONS
1. What is the value of the coefficient for sqft_living that you learned with no regularization, rounded to 1 decimal place?  What about the one with high regularization?
2. Comparing the lines you fit with the with no regularization versus high regularization, which one is steeper?
3. What are the RSS on the test data for each of the set of weights above (initial, no regularization, high regularization)?
Running a multiple regression with L2 penalty
Let us now consider a model with 2 features: ['sqft_living', 'sqft_living15'].
First, create Numpy versions of your training and test data with these two features.
Can you write Python code for it?

model_features = ['sqft_living', 'sqft_living15'] # sqft_living15 is the average squarefeet for the nearest 15 neighbors.
my_output = 'price'
(feature_matrix, output) = get_numpy_data(train_data, model_features, my_output)
(test_feature_matrix, test_output) = get_numpy_data(test_data, model_features, my_output)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
First, let's consider no regularization.  Set the l2_penalty to 0.0 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
simple_weights_0_penalty
we'll use them later.
Next, let's consider high regularization.  Set the l2_penalty to 1e11 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
simple_weights_high_penalty
we'll use them later.
This code will plot the two learned models.  (The blue line is for the model with no regularization and the red line is for the one with high regularization.)
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(simple_feature_matrix,output,'k.',
         simple_feature_matrix,predict_output(simple_feature_matrix, simple_weights_0_penalty),'b-',
        simple_feature_matrix,predict_output(simple_feature_matrix, simple_weights_high_penalty),'r-')


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
Import useful functions from previous notebook
As in Week 2, we convert the SFrame into a 2D Numpy array. Copy and paste get_numpy_data() from the second notebook of Week 2.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # add a constant column to an SFrame
    # prepend variable 'constant' to the features list
    features = ['constant'] + features
    # select the columns of data_SFrame given by the ‘features’ list into the SFrame ‘features_sframe’
    features_sframe = data_sframe[features]
    # this will convert the features_sframe into a numpy matrix with GraphLab Create >= 1.7!!
    features_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the target to the variable ‘output_sarray’
    output_sarray['target']=output

    # this will convert the SArray into a numpy array:
    output_array = output_sarray.to_numpy() # GraphLab Create>= 1.7!!
    return(features_matrix, output_array)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
In this part, we will only use 'sqft_living' to predict 'price'. Use the get_numpy_data function to get a Numpy versions of your data with only this feature, for both the train_data and the test_data.
Can you write Python code for it?

(simple_feature_matrix, output) = get_numpy_data(train_data, simple_features, my_output)
(simple_test_feature_matrix, test_output) = get_numpy_data(test_data, simple_features, my_output)


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
Let us split the dataset into training set and test set. Make sure to use seed=0:
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
Let's set the parameters for our optimization:
Can you write Python code for it?

initial_weights = np.array([0., 0.])
step_size = 1e-12
max_iterations=1000


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
To test your feature derivartive run the following:
Can you write Python code for it?

(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price')
my_weights = np.array([1., 10.])
test_predictions = predict_output(example_features, my_weights)
errors = test_predictions - example_output # prediction errors

# next two lines should print the same values
print feature_derivative_ridge(errors, example_features[:,1], my_weights[1], 1, False)
print np.sum(errors*example_features[:,1])*2+20.
print ''

# next two lines should print the same values
print feature_derivative_ridge(errors, example_features[:,0], my_weights[0], 1, True)
print np.sum(errors)*2.


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
Visualizing effect of L2 penalty
The L2 penalty gets its name because it causes weights to have small L2 norms than otherwise. Let's see how large weights get penalized. Let us consider a simple model with 1 feature:
Can you write Python code for it?

simple_features = ['sqft_living']
my_output = 'price'


I am working on the file "ML_UoW/Course01_Regression/Week04_Ridge_Regression_Assignment02.ipynb".
The first task is:
We need to re-inialize the weights, since we have one extra parameter. Let us also set the step size and maximum number of iterations.
Can you write Python code for it?

initial_weights = np.array([0.0,0.0,0.0])
step_size = 1e-12
max_iterations = 1000


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(1b) Element-wise multiplication: vectors
In this exercise, you will calculate the element-wise multiplication of two vectors by hand and enter the result in the code cell below.  You'll later see that element-wise multiplication is the default method when two NumPy arrays are multiplied together.  Note we won't be performing element-wise multiplication in future labs, but we are introducing it here to distinguish it from other vector operators, and to because it is a common operations in NumPy, as we will discuss in Part (2b).
The element-wise calculation is as follows: $$ \mathbf{x} \odot \mathbf{y} =  \begin{bmatrix} x_1 y_1 \\  x_2 y_2 \\ \vdots \\ x_n y_n \end{bmatrix} $$
Calculate the value of $ \mathbf{z} $: $$ \mathbf{z} = \begin{bmatrix} 1 \\  2 \\ 3 \end{bmatrix} \odot \begin{bmatrix} 4 \\  5 \\ 6 \end{bmatrix} $$
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Manually calculate your answer and represent the vector as a list of integers values.
z = [4, 10, 18]

# TEST Element-wise multiplication: vectors (1b)
Test.assertEqualsHashed(z, '4b5fe28ee2d274d7e0378bf993e28400f66205c2',
                        'incorrect value for vector z')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(1c) Dot product
In this exercise, you will calculate the dot product of two vectors by hand and enter the result in the code cell below.  Note that the dot product is equivalent to performing element-wise multiplication and then summing the result.
Below, you'll find the calculation for the dot product of two vectors, where each vector has length $ n $: $$ \mathbf{w} \cdot \mathbf{x} = \sum_{i=1}^n w_i x_i $$
Note that you may also see $ \mathbf{w} \cdot \mathbf{x} $ represented as $ \mathbf{w}^\top \mathbf{x} $
Calculate the value for $ c_1 $ based on the dot product of the following two vectors:
$$ c_1 = \begin{bmatrix} 1 \\ -3 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 5 \end{bmatrix}$$
Calculate the value for $ c_2 $ based on the dot product of the following two vectors:
$$ c_2 = \begin{bmatrix} 3 \\ 4 \\ 5 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}$$
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Manually calculate your answer and set the variables to their appropriate integer values.
c1 = -11
c2 = 26

# TEST Dot product (1c)
Test.assertEqualsHashed(c1, '8d7a9046b6a6e21d66409ad0849d6ab8aa51007c', 'incorrect value for c1')
Test.assertEqualsHashed(c2, '887309d048beef83ad3eabf2a79a64a389ab1c9f', 'incorrect value for c2')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(2b) Element-wise multiplication and dot product
NumPy arrays support both element-wise multiplication and dot product.  Element-wise multiplication occurs automatically when you use the * operator to multiply two ndarray objects of the same length.
To perform the dot product you can use either np.dot() or np.ndarray.dot().  For example, if you had NumPy arrays x and y, you could compute their dot product four ways: np.dot(x, y), np.dot(y, x), x.dot(y), or y.dot(x).
For this exercise, multiply the arrays u and v element-wise and compute their dot product.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Create a ndarray based on a range and step size.
u = np.arange(0, 5, .5)
v = np.arange(5, 10, .5)

elementWise = u * v
dotProduct = u.dot(v)
print 'u: {0}'.format(u)
print 'v: {0}'.format(v)
print '\nelementWise\n{0}'.format(elementWise)
print '\ndotProduct\n{0}'.format(dotProduct)

# TEST Element-wise multiplication and dot product (2b)
Test.assertTrue(np.all(elementWise == [ 0., 2.75, 6., 9.75, 14., 18.75, 24., 29.75, 36., 42.75]),
                'incorrect value for elementWise')
Test.assertEquals(dotProduct, 183.75, 'incorrect value for dotProduct')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(2c) Matrix math
With NumPy it is very easy to perform matrix math.  You can use np.matrix() to generate a NumPy matrix.  Just pass a two-dimensional ndarray or a list of lists to the function.  You can perform matrix math on NumPy matrices using *.
You can transpose a matrix by calling numpy.matrix.transpose() or by using .T on the matrix object (e.g. myMatrix.T).  Transposing a matrix produces a matrix where the new rows are the columns from the old matrix. For example: $$  \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}^\mathbf{\top} = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix} $$
Inverting a matrix can be done using numpy.linalg.inv().  Note that only square matrices can be inverted, and square matrices are not guaranteed to have an inverse.  If the inverse exists, then multiplying a matrix by its inverse will produce the identity matrix.  $ \scriptsize ( \mathbf{A}^{-1} \mathbf{A} = \mathbf{I_n} ) $  The identity matrix $ \scriptsize \mathbf{I_n} $ has ones along its diagonal and zero elsewhere. $$ \mathbf{I_n} = \begin{bmatrix} 1 & 0 & 0 & \dots & 0 \\ 0 & 1 & 0 & \dots & 0 \\ 0 & 0 & 1 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & 1 \end{bmatrix} $$
For this exercise, multiply $ \mathbf{A} $ times its transpose $ ( \mathbf{A}^\top ) $ and then calculate the inverse of the result $ (  [ \mathbf{A} \mathbf{A}^\top ]^{-1}  ) $.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
from numpy.linalg import inv

A = np.matrix([[1,2,3,4],[5,6,7,8]])
print 'A:\n{0}'.format(A)
# Print A transpose
print '\nA transpose:\n{0}'.format(A.T)

# Multiply A by A transpose
AAt = A * A.T
print '\nAAt:\n{0}'.format(AAt)

# Invert AAt with np.linalg.inv()
AAtInv = np.linalg.inv(AAt)
print '\nAAtInv:\n{0}'.format(AAtInv)

# Show inverse times matrix equals identity
# We round due to numerical precision
print '\nAAtInv * AAt:\n{0}'.format((AAtInv * AAt).round(4))

# TEST Matrix math (2c)
Test.assertTrue(np.all(AAt == np.matrix([[30, 70], [70, 174]])), 'incorrect value for AAt')
Test.assertTrue(np.allclose(AAtInv, np.matrix([[0.54375, -0.21875], [-0.21875, 0.09375]])),
                'incorrect value for AAtInv')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(3b) Combining ndarray objects
NumPy provides many functions for creating new arrays from existing arrays.  We'll explore two functions: np.hstack(), which allows you to combine arrays column-wise, and np.vstack(), which allows you to combine arrays row-wise.  Note that both np.hstack() and np.vstack() take in a tuple of arrays as their first argument.  To horizontally combine three arrays a, b, and c, you would run np.hstack((a, b, c)).
If we had two arrays: a = [1, 2, 3, 4] and b = [5, 6, 7, 8], we could use np.vstack((a, b)) to produce the two-dimensional array: $$  \begin{bmatrix} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{bmatrix} $$
For this exercise, you'll combine the zeros and ones arrays both horizontally (column-wise) and vertically (row-wise).
Note that the result of stacking two arrays is an ndarray.  If you need the result to be a matrix, you can call np.matrix() on the result, which will return a NumPy matrix.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
zeros = np.zeros(8)
ones = np.ones(8)
print 'zeros:\n{0}'.format(zeros)
print '\nones:\n{0}'.format(ones)

zerosThenOnes = np.hstack((zeros, ones))   # A 1 by 16 array
zerosAboveOnes = np.vstack((zeros, ones))   # A 2 by 8 array

print '\nzerosThenOnes:\n{0}'.format(zerosThenOnes)
print '\nzerosAboveOnes:\n{0}'.format(zerosAboveOnes)

# TEST Combining ndarray objects (3b)
Test.assertTrue(np.all(zerosThenOnes == [0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1]),
                'incorrect value for zerosThenOnes')
Test.assertTrue(np.all(zerosAboveOnes == [[0,0,0,0,0,0,0,0],[1,1,1,1,1,1,1,1]]),
                'incorrect value for zerosAboveOnes')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(3c) PySpark's DenseVector
PySpark provides a DenseVector class within the module pyspark.mllib.linalg.  DenseVector is used to store arrays of values for use in PySpark.  DenseVector actually stores values in a NumPy array and delegates calculations to that object.  You can create a new DenseVector using DenseVector() and passing in an NumPy array or a Python list.
DenseVector implements several functions.  The only function needed for this course is DenseVector.dot(), which operates just like np.ndarray.dot().
Note that DenseVector stores all values as np.float64, so even if you pass in an NumPy array of integers, the resulting DenseVector will contain floating-point numbers. Also, DenseVector objects exist locally and are not inherently distributed.  DenseVector objects can be used in the distributed setting by either passing functions that contain them to resilient distributed dataset (RDD) transformations or by distributing them directly as RDDs.  You'll learn more about RDDs in the spark tutorial.
For this exercise, create a DenseVector consisting of the values [3.0, 4.0, 5.0] and compute the dot product of this vector with numpyVector.
Can you write Python code for it?

from pyspark.mllib.linalg import DenseVector

# TODO: Replace <FILL IN> with appropriate code
numpyVector = np.array([-3, -4, 5])
print '\nnumpyVector:\n{0}'.format(numpyVector)

# Create a DenseVector consisting of the values [3.0, 4.0, 5.0]
myDenseVector = DenseVector([3.0, 4.0, 5.0])
# Calculate the dot product between the two vectors.
denseDotProduct = myDenseVector.dot(numpyVector)

print 'myDenseVector:\n{0}'.format(myDenseVector)
print '\ndenseDotProduct:\n{0}'.format(denseDotProduct)

# TEST PySpark's DenseVector (3c)
Test.assertTrue(isinstance(myDenseVector, DenseVector), 'myDenseVector is not a DenseVector')
Test.assertTrue(np.allclose(myDenseVector, np.array([3., 4., 5.])),
                'incorrect value for myDenseVector')
Test.assertTrue(np.allclose(denseDotProduct, 0.0), 'incorrect value for denseDotProduct')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(4c) Lambda expression arguments
Lambda expressions can be used to generate functions that take in zero or more parameters.  The syntax for lambda allows for multiple ways to define the same function.  For example, we might want to create a function that takes in a single parameter, where the parameter is a tuple consisting of two values, and the function adds the two values together.  The syntax could be either: lambda x: x[0] + x[1] or lambda (x0, x1): x0 + x1.  If we called either function on the tuple (3, 4) it would return 7.  Note that the second lambda relies on the tuple (3, 4) being unpacked automatically, which means that x0 is assigned the value 3 and x1 is assigned the value 4.
As an other example, consider the following parameter lambda expressions: lambda x, y: (x[0] + y[0], x[1] + y[1]) and lambda (x0, x1), (y0, y1): (x0 + y0, x1 + y1).  The result of applying either of these functions to tuples  (1, 2) and (3, 4) would be the tuple (4, 6).
For this exercise: you'll create one-parameter functions swap1 and swap2 that swap the order of a tuple; a one-parameter function swapOrder that takes in a tuple with three values and changes the order to: second element, third element, first element; and finally, a three-parameter function sumThree that takes in three tuples, each with two values, and returns a tuple containing two values: the sum of the first element of each tuple and the sum of second element of each tuple.
Can you write Python code for it?

# Examples.  Note that the spacing has been modified to distinguish parameters from tuples.

# One-parameter function
a1 = lambda x: x[0] + x[1]
a2 = lambda (x0, x1): x0 + x1
print 'a1( (3,4) ) = {0}'.format( a1( (3,4) ) )
print 'a2( (3,4) ) = {0}'.format( a2( (3,4) ) )

# Two-parameter function
b1 = lambda x, y: (x[0] + y[0], x[1] + y[1])
b2 = lambda (x0, x1), (y0, y1): (x0 + y0, x1 + y1)
print '\nb1( (1,2), (3,4) ) = {0}'.format( b1( (1,2), (3,4) ) )
print 'b2( (1,2), (3,4) ) = {0}'.format( b2( (1,2), (3,4) ) )

# TODO: Replace <FILL IN> with appropriate code
# Use both syntaxes to create a function that takes in a tuple of two values and swaps their order
# E.g. (1, 2) => (2, 1)
swap1 = lambda x: (x[1], x[0])
swap2 = lambda (x0, x1): (x1, x0)
print 'swap1((1, 2)) = {0}'.format(swap1((1, 2)))
print 'swap2((1, 2)) = {0}'.format(swap2((1, 2)))

# Using either syntax, create a function that takes in a tuple with three values and returns a tuple
# of (2nd value, 3rd value, 1st value).  E.g. (1, 2, 3) => (2, 3, 1)
swapOrder = lambda x: (x[1], x[2], x[0])
print 'swapOrder((1, 2, 3)) = {0}'.format(swapOrder((1, 2, 3)))

# Using either syntax, create a function that takes in three tuples each with two values.  The
# function should return a tuple with the values in the first position summed and the values in the
# second position summed. E.g. (1, 2), (3, 4), (5, 6) => (1 + 3 + 5, 2 + 4 + 6) => (9, 12)
sumThree = lambda x, y, z: (x[0] + y[0]+ z[0], x[1] + y[1] + z[1])
print 'sumThree((1, 2), (3, 4), (5, 6)) = {0}'.format(sumThree((1, 2), (3, 4), (5, 6)))

# TEST Lambda expression arguments (4c)
Test.assertEquals(swap1((1, 2)), (2, 1), 'incorrect definition for swap1')
Test.assertEquals(swap2((1, 2)), (2, 1), 'incorrect definition for swap2')
Test.assertEquals(swapOrder((1, 2, 3)), (2, 3, 1), 'incorrect definition fo swapOrder')
Test.assertEquals(sumThree((1, 2), (3, 4), (5, 6)), (9, 12), 'incorrect definition for sumThree')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(4e) Functional programming
The lambda examples we have shown so far have been somewhat contrived.  This is because they were created to demonstrate the differences and similarities between lambda and def.  An excellent use case for lambda expressions is functional programming.  In functional programming, you will often pass functions to other functions as parameters, and lambda can be used to reduce the amount of code necessary and to make the code more readable.
Some commonly used functions in functional programming are map, filter, and reduce.  Map transforms a series of elements by applying a function individually to each element in the series.  It then returns the series of transformed elements.  Filter also applies a function individually to each element in a series; however, with filter, this function evaluates to True or False and only elements that evaluate to True are retained.  Finally, reduce operates on pairs of elements in a series.  It applies a function that takes in two values and returns a single value.  Using this function, reduce is able to, iteratively, "reduce" a series to a single value.
For this exercise, you'll create three simple lambda functions, one each for use in map, filter, and reduce.  The map lambda will multiply its input by 5, the filter lambda will evaluate to True for even numbers, and the reduce lambda will add two numbers.  Note that we have created a class called FunctionalWrapper so that the syntax for this exercise matches the syntax you'll see in PySpark.
Note that map requires a one parameter function that returns a new value, filter requires a one parameter function that returns True or False, and reduce requires a two parameter function that combines the two parameters and returns a new value.
Can you write Python code for it?

# Create a class to give our examples the same syntax as PySpark
class FunctionalWrapper(object):
    def __init__(self, data):
        self.data = data
    def map(self, function):
        """Call `map` on the items in `data` using the provided `function`"""
        return FunctionalWrapper(map(function, self.data))
    def reduce(self, function):
        """Call `reduce` on the items in `data` using the provided `function`"""
        return reduce(function, self.data)
    def filter(self, function):
        """Call `filter` on the items in `data` using the provided `function`"""
        return FunctionalWrapper(filter(function, self.data))
    def __eq__(self, other):
        return (isinstance(other, self.__class__)
            and self.__dict__ == other.__dict__)
    def __getattr__(self, name):  return getattr(self.data, name)
    def __getitem__(self, k):  return self.data.__getitem__(k)
    def __repr__(self):  return 'FunctionalWrapper({0})'.format(repr(self.data))
    def __str__(self):  return 'FunctionalWrapper({0})'.format(str(self.data))

# Map example

# Create some data
mapData = FunctionalWrapper(range(5))

# Define a function to be applied to each element
f = lambda x: x + 3

# Imperative programming: loop through and create a new object by applying f
mapResult = FunctionalWrapper([])  # Initialize the result
for element in mapData:
    mapResult.append(f(element))  # Apply f and save the new value
print 'Result from for loop: {0}'.format(mapResult)

# Functional programming: use map rather than a for loop
print 'Result from map call: {0}'.format(mapData.map(f))

# Note that the results are the same but that the map function abstracts away the implementation
# and requires less code

# TODO: Replace <FILL IN> with appropriate code
dataset = FunctionalWrapper(range(10))

# Multiply each element by 5
mapResult = dataset.map(lambda x: 5 *x)
# Keep the even elements
# Note that "x % 2" evaluates to the remainder of x divided by 2
filterResult = dataset.filter(lambda x: x % 2 == 0)
# Sum the...


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
(4f) Composability
Since our methods for map and filter in the FunctionalWrapper class return FunctionalWrapper objects, we can compose (or chain) together our function calls.  For example, dataset.map(f1).filter(f2).reduce(f3), where f1, f2, and f3 are functions or lambda expressions, first applies a map operation to dataset, then filters the result from map, and finally reduces the result from the first two operations.
Note that when we compose (chain) an operation, the output of one operation becomes the input for the next operation, and operations are applied from left to right.  It's likely you've seen chaining used with Python strings.  For example, 'Split this'.lower().split(' ') first returns a new string object 'split this' and then split(' ') is called on that string to produce ['split', 'this'].
For this exercise, reuse your lambda expressions from (4e) but apply them to dataset in the sequence: map, filter, reduce.  Note that since we are composing the operations our result will be different than in (4e).  Also, we can write our operations on separate lines to improve readability.
Can you write Python code for it?

# Example of a mult-line expression statement
# Note that placing parentheses around the expression allow it to exist on multiple lines without
# causing a syntax error.
(dataset
 .map(lambda x: x + 2)
 .reduce(lambda x, y: x * y))

# TODO: Replace <FILL IN> with appropriate code
# Multiply the elements in dataset by five, keep just the even values, and sum those values
finalSum = (dataset
            .map(lambda x: x * 5)
            .filter(lambda x: x % 2 == 0)
            .reduce(lambda x, y: x + y))
print finalSum

# TEST Composability (4f)
Test.assertEquals(finalSum, 100, 'incorrect value for finalSum')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
Part 1: Math review
 (1a) Scalar multiplication: vectors
In this exercise, you will calculate the product of a scalar and a vector by hand and enter the result in the code cell below.  Scalar multiplication is straightforward.  The resulting vector equals the product of the scalar, which is a single value, and each item in the original vector.  In the example below, $ a $ is the scalar (constant) and $ \mathbf{v} $ is the vector.  $$ a \mathbf{v} = \begin{bmatrix} a v_1 \\ a v_2 \\ \vdots \\ a v_n \end{bmatrix} $$
Calculate the value of $ \mathbf{x} $: $$ \mathbf{x} = 3 \begin{bmatrix} 1 \\ -2 \\ 0 \end{bmatrix} $$
Calculate the value of $ \mathbf{y} $: $$ \mathbf{y} = 2 \begin{bmatrix} 2 \\ 4 \\ 8 \end{bmatrix} $$
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Manually calculate your answer and represent the vector as a list of integers values.
# For example, [2, 4, 8].
x = [3, -6 , 0]
y = [4, 8 ,16]

# TEST Scalar multiplication: vectors (1a)
# Import test library
from test_helper import Test
Test.assertEqualsHashed(x, 'e460f5b87531a2b60e0f55c31b2e49914f779981',
                        'incorrect value for vector x')
Test.assertEqualsHashed(y, 'e2d37ff11427dbac7f833a5a7039c0de5a740b1e',
                        'incorrect value for vector y')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
Part 2: NumPy
 (2a) Scalar multiplication
NumPy is a Python library for working with arrays.  NumPy provides abstractions that make it easy to treat these underlying arrays as vectors and matrices.  The library is optimized to be fast and memory efficient, and we'll be using it throughout the course.  The building block for NumPy is the ndarray, which is a multidimensional array of fixed-size that contains elements of one type (e.g. array of floats).
For this exercise, you'll create a ndarray consisting of the elements [1, 2, 3] and multiply this array by 5.  Use np.array() to create the array.  Note that you can pass a Python list into np.array().  To perform scalar multiplication with an ndarray just use *.
Note that if you create an array from a Python list of integers you will obtain a one-dimensional array, which is equivalent to a vector for our purposes.
Can you write Python code for it?

# It is convention to import NumPy with the alias np
import numpy as np

# TODO: Replace <FILL IN> with appropriate code
# Create a numpy array with the values 1, 2, 3
simpleArray = np.array([1, 2, 3])
# Perform the scalar product of 5 and the numpy array
timesFive = 5 * simpleArray
print simpleArray
print timesFive

# TEST Scalar multiplication (2a)
Test.assertTrue(np.all(timesFive == [5, 10, 15]), 'incorrect value for timesFive')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
Part 3: Additional NumPy and Spark linear algebra
 (3a) Slices
You can select a subset of a one-dimensional NumPy ndarray's elements by using slices.  These slices operate the same way as slices for Python lists.  For example, [0, 1, 2, 3][:2] returns the first two elements [0, 1].  NumPy, additionally, has more sophisticated slicing that allows slicing across multiple dimensions; however, you'll only need to use basic slices in future labs for this course.
Note that if no index is placed to the left of a :, it is equivalent to starting at 0, and hence [0, 1, 2, 3][:2] and [0, 1, 2, 3][0:2] yield the same result.  Similarly, if no index is placed to the right of a :, it is equivalent to slicing to the end of the object.  Also, you can use negative indices to index relative to the end of the object, so [-2:] would return the last two elements of the object.
For this exercise, return the last 3 elements of the array features.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
features = np.array([1, 2, 3, 4])
print 'features:\n{0}'.format(features)

# The last three elements of features
lastThree = features[-3:]

print '\nlastThree:\n{0}'.format(lastThree)

# TEST Slices (3a)
Test.assertTrue(np.all(lastThree == [2, 3, 4]), 'incorrect value for lastThree')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
Part 4: Python lambda expressions
 (4a) Lambda is an anonymous function
We can use a lambda expression to create a function.  To do this, you type lambda followed by the names of the function's parameters separated by commas, followed by a :, and then the expression statement that the function will evaluate.  For example, lambda x, y: x + y is an anonymous function that computes the sum of its two inputs.
Lambda expressions return a function when evaluated.  The function is not bound to any variable, which is why lambdas are associated with anonymous functions.  However, it is possible to assign the function to a variable.  Lambda expressions are particularly useful when you need to pass a simple function into another function.  In that case, the lambda expression generates a function that is bound to the parameter being passed into the function.
Below, we'll see an example of how we can bind the function returned by a lambda expression to a variable named addSLambda.  From this example, we can see that lambda provides a shortcut for creating a simple function.  Note that the behavior of the function created using def and the function created using lambda is equivalent.  Both functions have the same type and return the same results.  The only differences are the names and the way they were created.
For this exercise, first run the two cells below to compare a function created using def with a corresponding anonymous function.  Next, write your own lambda expression that creates a function that multiplies its input (a single parameter) by 10.
Here are some additional references that explain lambdas: Lambda Functions, Lambda Tutorial, and Python Functions.
Can you write Python code for it?

# Example function
def addS(x):
    return x + 's'
print type(addS)
print addS
print addS('cat')

# As a lambda
addSLambda = lambda x: x + 's'
print type(addSLambda)
print addSLambda
print addSLambda('cat')

# TODO: Replace <FILL IN> with appropriate code
# Recall that: "lambda x, y: x + y" creates a function that adds together two numbers
multiplyByTen = lambda x: x * 10
print multiplyByTen(5)

# Note that the function still shows its name as <lambda>
print '\n', multiplyByTen

# TEST Python lambda expressions (4a)
Test.assertEquals(multiplyByTen(10), 100, 'incorrect definition for multiplyByTen')


I am working on the file "ML_lab1_review_student.ipynb".
The first task is:
Part 5: CTR data download
Lab four will explore website click-through data provided by Criteo.  To obtain the data, you must first accept Criteo's data sharing agreement.  Below is the agreement from Criteo.  After you accept the agreement, you can obtain the download URL by right-clicking on the "Download Sample" button and clicking "Copy link address" or "Copy Link Location", depending on your browser.  Paste the URL into the # TODO cell below.  The file is 8.4 MB compressed.  The script below will download the file to the virtual machine (VM) and then extract the data.
If running the cell below does not render a webpage, open the Criteo agreement in a separate browser tab. After you accept the agreement, you can obtain the download URL by right-clicking on the "Download Sample" button and clicking "Copy link address" or "Copy Link Location", depending on your browser.  Paste the URL into the # TODO cell below.
Note that the download could take a few minutes, depending upon your connection speed.
Can you write Python code for it?

# Run this code to view Criteo's agreement
# Note that some ad blocker software will prevent this IFrame from loading.
# If this happens, open the webpage in a separate tab and follow the instructions from above.
from IPython.lib.display import IFrame

IFrame("http://labs.criteo.com/downloads/2014-kaggle-display-advertising-challenge-dataset/",
       600, 350)

# TODO: Replace <FILL IN> with appropriate code
# Just replace <FILL IN> with the url for dac_sample.tar.gz
import glob
import os.path
import tarfile
import urllib
import urlparse

# Paste url, url should end with: dac_sample.tar.gz
url = 'http://labs.criteo.com/wp-content/uploads/2015/04/dac_sample.tar.gz'

url = url.strip()
baseDir = os.path.join('data')
inputPath = os.path.join('cs190', 'dac_sample.txt')
fileName = os.path.join(baseDir, inputPath)
inputDir = os.path.split(fileName)[0]

def extractTar(check = False):
    # Find the zipped archive and extract the dataset
    tars = glob.glob('dac_sample*.tar.gz*')
    if check and len(tars) == 0:
      return False

    if len(tars) > 0:
        try:
            tarFile = tarfile.open(tars[0])
        except tarfile.ReadError:
            if not check:
                print 'Unable to open tar.gz file.  Check your URL.'
            return False

        tarFile.extract('dac_sample.txt', path=inputDir)
        print 'Successfully extracted: dac_sample.txt'
        return True
    else:
        print 'You need to retry the download with the correct url.'
        print ('Alternatively, you can upload the dac_sample.tar.gz file to your Jupyter root ' +
              'directory')
        return False

if os.path.isfile(fileName):
    print 'File is already available. Nothing to do.'
elif extractTar(check = True):
    print 'tar.gz file was already available.'
elif not url.endswith('dac_sample.tar.gz'):
    print 'Check your download url.  Are you downloading the Sample dataset?'
else:
    # Download the file and store it in the same directory as this notebook
    try:
        urllib.urlretrieve(url,...


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(1c) Apply makePlural to the base RDD
Now pass each item in the base RDD into a map() transformation that applies the makePlural() function to each element. And then call the collect() action to see the transformed RDD.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
pluralRDD = wordsRDD.map(makePlural)
print pluralRDD.collect()

# TEST Apply makePlural to the base RDD(1c)
Test.assertEquals(pluralRDD.collect(), ['cats', 'elephants', 'rats', 'rats', 'cats'],
                  'incorrect values for pluralRDD')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(1d) Pass a lambda function to map
Let's create the same RDD using a lambda function.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
pluralLambdaRDD = wordsRDD.map(lambda a: a + "s")
print pluralLambdaRDD.collect()

# TEST Pass a lambda function to map (1d)
Test.assertEquals(pluralLambdaRDD.collect(), ['cats', 'elephants', 'rats', 'rats', 'cats'],
                  'incorrect values for pluralLambdaRDD (1d)')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(1e) Length of each word
Now use map() and a lambda function to return the number of characters in each word.  We'll collect this result directly into a variable.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
pluralLengths = (pluralRDD
                 .map(lambda a: len(a))
                 .collect())
print pluralLengths

# TEST Length of each word (1e)
Test.assertEquals(pluralLengths, [4, 9, 4, 4, 4],
                  'incorrect values for pluralLengths')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(1f) Pair RDDs
The next step in writing our word counting program is to create a new type of RDD, called a pair RDD. A pair RDD is an RDD where each element is a pair tuple (k, v) where k is the key and v is the value. In this example, we will create a pair consisting of ('<word>', 1) for each word element in the RDD.
We can create the pair RDD using the map() transformation with a lambda() function to create a new RDD.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
wordPairs = wordsRDD.map(lambda a: (a,1))
print wordPairs.collect()

# TEST Pair RDDs (1f)
Test.assertEquals(wordPairs.collect(),
                  [('cat', 1), ('elephant', 1), ('rat', 1), ('rat', 1), ('cat', 1)],
                  'incorrect value for wordPairs')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(2b) Use groupByKey() to obtain the counts
Using the groupByKey() transformation creates an RDD containing 3 elements, each of which is a pair of a word and a Python iterator.
Now sum the iterator using a map() transformation.  The result should be a pair RDD consisting of (word, count) pairs.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
wordCountsGrouped = wordsGrouped.map(lambda (a,b): (a, sum(b)))
print wordCountsGrouped.collect()

# TEST Use groupByKey() to obtain the counts (2b)
Test.assertEquals(sorted(wordCountsGrouped.collect()),
                  [('cat', 2), ('elephant', 1), ('rat', 2)],
                  'incorrect value for wordCountsGrouped')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(2c) Counting using reduceByKey
A better approach is to start from the pair RDD and then use the reduceByKey() transformation to create a new pair RDD. The reduceByKey() transformation gathers together pairs that have the same key and applies the function provided to two values at a time, iteratively reducing all of the values to a single value. reduceByKey() operates by applying the function first within each partition on a per-key basis and then across the partitions, allowing it to scale efficiently to large datasets.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Note that reduceByKey takes in a function that accepts two values and returns a single value
wordCounts = wordPairs.reduceByKey(lambda a,b: a+b)
print wordCounts.collect()

# TEST Counting using reduceByKey (2c)
Test.assertEquals(sorted(wordCounts.collect()), [('cat', 2), ('elephant', 1), ('rat', 2)],
                  'incorrect value for wordCounts')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(3b) Mean using reduce
Find the mean number of words per unique word in wordCounts.
Use a reduce() action to sum the counts in wordCounts and then divide by the number of unique words.  First map() the pair RDD wordCounts, which consists of (key, value) pairs, to an RDD of values.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
from operator import add
totalCount = (wordCounts
              .map(lambda (a,b): b)
              .reduce(lambda a,b: a+b))
average = totalCount / float(wordCounts.distinct().count())
print totalCount
print round(average, 2)

# TEST Mean using reduce (3b)
Test.assertEquals(round(average, 2), 1.67, 'incorrect value of average')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(4b) Capitalization and punctuation
Real world files are more complicated than the data we have been using in this lab. Some of the issues we have to address are:

Words should be counted independent of their capitialization (e.g., Spark and spark should be counted as the same word).

All punctuation should be removed.

Any leading or trailing spaces on a line should be removed.

Define the function removePunctuation that converts all text to lower case, removes any punctuation, and removes leading and trailing spaces.  Use the Python re module to remove any text that is not a letter, number, or space. Reading help(re.sub) might be useful.
If you are unfamiliar with regular expressions, you may want to review this tutorial from Google.  Also, this website is  a great resource for debugging your regular expression.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
import re
def removePunctuation(text):
    """Removes punctuation, changes to lower case, and strips leading and trailing spaces.

    Note:
        Only spaces, letters, and numbers should be retained.  Other characters should should be
        eliminated (e.g. it's becomes its).  Leading and trailing spaces should be removed after
        punctuation is removed.

    Args:
        text (str): A string.

    Returns:
        str: The cleaned up string.
    """
    return re.sub("[^a-zA-Z0-9 ]", "", text.strip(" ").lower())

print removePunctuation('Hi, you!')
print removePunctuation(' No under_score!')
print removePunctuation(' *      Remove punctuation then spaces  * ')

# TEST Capitalization and punctuation (4b)
Test.assertEquals(removePunctuation(" The Elephant's 4 cats. "),
                  'the elephants 4 cats',
                  'incorrect definition for removePunctuation function')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(4d) Words from lines
Before we can use the wordcount() function, we have to address two issues with the format of the RDD:

The first issue is that  that we need to split each line by its spaces.  Performed in (4d).

The second issue is we need to filter out empty lines.  Performed in (4e).

Apply a transformation that will split each element of the RDD by its spaces. For each element of the RDD, you should apply Python's string split() function. You might think that a map() transformation is the way to do this, but think about what the result of the split() function will be. Note that you should not use the default implemenation of split(), but should instead pass in a separator value.  For example, to split line by commas you would use line.split(',').
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
shakespeareWordsRDD = shakespeareRDD.flatMap(lambda a: a.split(" "))
shakespeareWordCount = shakespeareWordsRDD.count()
print shakespeareWordsRDD.top(5)
print shakespeareWordCount

# TEST Words from lines (4d)
# This test allows for leading spaces to be removed either before or after
# punctuation is removed.
Test.assertTrue(shakespeareWordCount == 927631 or shakespeareWordCount == 928908,
                'incorrect value for shakespeareWordCount')
Test.assertEquals(shakespeareWordsRDD.top(5),
                  [u'zwaggerd', u'zounds', u'zounds', u'zounds', u'zounds'],
                  'incorrect value for shakespeareWordsRDD')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
(4f) Count the words
We now have an RDD that is only words.  Next, let's apply the wordCount() function to produce a list of word counts. We can view the top 15 words by using the takeOrdered() action; however, since the elements of the RDD are pairs, we need a custom sort function that sorts using the value part of the pair.
You'll notice that many of the words are common English words. These are called stopwords. In a later lab, we will see how to eliminate them from the results.
Use the wordCount() function and takeOrdered() to obtain the fifteen most common words and their counts.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
top15WordsAndCounts = wordCount(shakeWordsRDD).takeOrdered(15, lambda(a,b): -b)
print '\n'.join(map(lambda (w, c): '{0}: {1}'.format(w, c), top15WordsAndCounts))

# TEST Count the words (4f)
Test.assertEquals(top15WordsAndCounts,
                  [(u'the', 27361), (u'and', 26028), (u'i', 20681), (u'to', 19150), (u'of', 17463),
                   (u'a', 14593), (u'you', 13615), (u'my', 12481), (u'in', 10956), (u'that', 10890),
                   (u'is', 9134), (u'not', 8497), (u'with', 7771), (u'me', 7769), (u'it', 7678)],
                  'incorrect value for top15WordsAndCounts')


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
Part 1: Creating a base RDD and pair RDDs
In this part of the lab, we will explore creating a base RDD with parallelize and using pair RDDs to count words.
 (1a) Create a base RDD
We'll start by generating a base RDD by using a Python list and the sc.parallelize method.  Then we'll print out the type of the base RDD.
Can you write Python code for it?

wordsList = ['cat', 'elephant', 'rat', 'rat', 'cat']
wordsRDD = sc.parallelize(wordsList, 4)
# Print out the type of wordsRDD
print type(wordsRDD)


I am working on the file "ML_lab2_word_count_student.ipynb".
The first task is:
Part 4: Apply word count to a file
In this section we will finish developing our word count application.  We'll have to build the wordCount function, deal with real world problems like capitalization and punctuation, load in our data source, and compute the word count on the new data.
 (4a) wordCount function
First, define a function for word counting.  You should reuse the techniques that have been covered in earlier parts of this lab.  This function should take in an RDD that is a list of words like wordsRDD and return a pair RDD that has all of the words and their associated counts.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def wordCount(wordListRDD):
    """Creates a pair RDD with word counts from an RDD of words.

    Args:
        wordListRDD (RDD of str): An RDD consisting of words.

    Returns:
        RDD of (str, int): An RDD consisting of (word, count) tuples.
    """
    return (wordListRDD
           .map(lambda a : (a,1))
           .reduceByKey(lambda a,b: a+b))

print wordCount(wordsRDD).collect()

# TEST wordCount function (4a)
Test.assertEquals(sorted(wordCount(wordsRDD).collect()),
                  [('cat', 2), ('elephant', 1), ('rat', 2)],
                  'incorrect definition for wordCount function')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(1b) Using LabeledPoint
In MLlib, labeled training instances are stored using the LabeledPoint object.  Write the parsePoint function that takes as input a raw data point, parses it using Python's unicode.split method, and returns a LabeledPoint.  Use this function to parse samplePoints (from the previous question).  Then print out the features and label for the first training point, using the LabeledPoint.features and LabeledPoint.label attributes. Finally, calculate the number features for this dataset.
Note that split() can be called directly on a unicode or str object.  For example, u'split,me'.split(',') returns [u'split', u'me'].
Can you write Python code for it?

from pyspark.mllib.regression import LabeledPoint
import numpy as np

# Here is a sample raw data point:
# '2001.0,0.884,0.610,0.600,0.474,0.247,0.357,0.344,0.33,0.600,0.425,0.60,0.419'
# In this raw data point, 2001.0 is the label, and the remaining values are features

# TODO: Replace <FILL IN> with appropriate code
def parsePoint(line):
    """Converts a comma separated unicode string into a `LabeledPoint`.

    Args:
        line (unicode): Comma separated unicode string where the first element is the label and the
            remaining elements are features.

    Returns:
        LabeledPoint: The line is converted into a `LabeledPoint`, which consists of a label and
            features.
    """
    splitline = np.fromstring(line, dtype=float, sep=',')
    return LabeledPoint(splitline[0], splitline[1:])

parsedSamplePoints = rawData.map(parsePoint)
firstPointFeatures = parsedSamplePoints.first().features
firstPointLabel = parsedSamplePoints.first().label
print firstPointFeatures, firstPointLabel

d = len(firstPointFeatures)
print d

# TEST Using LabeledPoint (1b)
Test.assertTrue(isinstance(firstPointLabel, float), 'label must be a float')
expectedX0 = [0.8841,0.6105,0.6005,0.4747,0.2472,0.3573,0.3441,0.3396,0.6009,0.4257,0.6049,0.4192]
Test.assertTrue(np.allclose(expectedX0, firstPointFeatures, 1e-4, 1e-4),
                'incorrect features for firstPointFeatures')
Test.assertTrue(np.allclose(2001.0, firstPointLabel), 'incorrect label for firstPointLabel')
Test.assertTrue(d == 12, 'incorrect number of features')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(1c) Find the range
Now let's examine the labels to find the range of song years.  To do this, first parse each element of the rawData RDD, and then find the smallest and largest labels.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
parsedDataInit = rawData.map(parsePoint)
onlyLabels = parsedDataInit.map(lambda x: x.label)
minYear = onlyLabels.min()
maxYear = onlyLabels.max()
print maxYear, minYear

# TEST Find the range (1c)
Test.assertEquals(len(parsedDataInit.take(1)[0].features), 12,
                  'unexpected number of features in sample point')
sumFeatTwo = parsedDataInit.map(lambda lp: lp.features[2]).sum()
Test.assertTrue(np.allclose(sumFeatTwo, 3158.96224351), 'parsedDataInit has unexpected values')
yearRange = maxYear - minYear
Test.assertTrue(yearRange == 89, 'incorrect range for minYear to maxYear')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(1d) Shift labels
As we just saw, the labels are years in the 1900s and 2000s.  In learning problems, it is often natural to shift labels such that they start from zero.  Starting with parsedDataInit, create a new RDD consisting of LabeledPoint objects in which the labels are shifted such that smallest label equals zero.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
parsedData = parsedDataInit.map(lambda x: LabeledPoint(x.label - minYear, x.features))

# Should be a LabeledPoint
print type(parsedData.take(1)[0])
# View the first point
print '\n{0}'.format(parsedData.take(1))

# TEST Shift labels (1d)
oldSampleFeatures = parsedDataInit.take(1)[0].features
newSampleFeatures = parsedData.take(1)[0].features
Test.assertTrue(np.allclose(oldSampleFeatures, newSampleFeatures),
                'new features do not match old features')
sumFeatTwo = parsedData.map(lambda lp: lp.features[2]).sum()
Test.assertTrue(np.allclose(sumFeatTwo, 3158.96224351), 'parsedData has unexpected values')
minYearNew = parsedData.map(lambda lp: lp.label).min()
maxYearNew = parsedData.map(lambda lp: lp.label).max()
Test.assertTrue(minYearNew == 0, 'incorrect min year in shifted data')
Test.assertTrue(maxYearNew == 89, 'incorrect max year in shifted data')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(1e) Training, validation, and test sets
We're almost done parsing our dataset, and our final task involves split it into training, validation and test sets. Use the randomSplit method with the specified weights and seed to create RDDs storing each of these datasets. Next, cache each of these RDDs, as we will be accessing them multiple times in the remainder of this lab. Finally, compute the size of each dataset and verify that the sum of their sizes equals the value computed in Part (1a).
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
weights = [.8, .1, .1]
seed = 42
parsedTrainData, parsedValData, parsedTestData = parsedData.randomSplit(weights, seed)
parsedTrainData.cache()
parsedValData.cache()
parsedTestData.cache()
nTrain = parsedTrainData.count()
nVal = parsedValData.count()
nTest = parsedTestData.count()

print nTrain, nVal, nTest, nTrain + nVal + nTest
print parsedData.count()

# TEST Training, validation, and test sets (1e)
Test.assertEquals(parsedTrainData.getNumPartitions(), numPartitions,
                  'parsedTrainData has wrong number of partitions')
Test.assertEquals(parsedValData.getNumPartitions(), numPartitions,
                  'parsedValData has wrong number of partitions')
Test.assertEquals(parsedTestData.getNumPartitions(), numPartitions,
                  'parsedTestData has wrong number of partitions')
Test.assertEquals(len(parsedTrainData.take(1)[0].features), 12,
                  'parsedTrainData has wrong number of features')
sumFeatTwo = (parsedTrainData
              .map(lambda lp: lp.features[2])
              .sum())
sumFeatThree = (parsedValData
                .map(lambda lp: lp.features[3])
                .reduce(lambda x, y: x + y))
sumFeatFour = (parsedTestData
               .map(lambda lp: lp.features[4])
               .reduce(lambda x, y: x + y))
Test.assertTrue(np.allclose([sumFeatTwo, sumFeatThree, sumFeatFour],
                            2526.87757656, 297.340394298, 184.235876654),
                'parsed Train, Val, Test data has unexpected values')
Test.assertTrue(nTrain + nVal + nTest == 6724, 'unexpected Train, Val, Test data set size')
Test.assertEquals(nTrain, 5371, 'unexpected value for nTrain')
Test.assertEquals(nVal, 682, 'unexpected value for nVal')
Test.assertEquals(nTest, 671, 'unexpected value for nTest')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(2b) Root mean squared error
We naturally would like to see how well this naive baseline performs.  We will use root mean squared error (RMSE) for evaluation purposes.  Implement a function to compute RMSE given an RDD of (label, prediction) tuples, and test out this function on an example.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def squaredError(label, prediction):
    """Calculates the the squared error for a single prediction.

    Args:
        label (float): The correct value for this observation.
        prediction (float): The predicted value for this observation.

    Returns:
        float: The difference between the `label` and `prediction` squared.
    """
    return (label - prediction)**2.

def calcRMSE(labelsAndPreds):
    """Calculates the root mean squared error for an `RDD` of (label, prediction) tuples.

    Args:
        labelsAndPred (RDD of (float, float)): An `RDD` consisting of (label, prediction) tuples.

    Returns:
        float: The square root of the mean of the squared errors.
    """
    return np.sqrt(labelsAndPreds
                   .map(lambda x: squaredError(x[0], x[1]))
                   .mean())

labelsAndPreds = sc.parallelize([(3., 1.), (1., 2.), (2., 2.)])
# RMSE = sqrt[((3-1)^2 + (1-2)^2 + (2-2)^2) / 3] = 1.291
exampleRMSE = calcRMSE(labelsAndPreds)
print exampleRMSE

# TEST Root mean squared error (2b)
Test.assertTrue(np.allclose(squaredError(3, 1), 4.), 'incorrect definition of squaredError')
Test.assertTrue(np.allclose(exampleRMSE, 1.29099444874), 'incorrect value for exampleRMSE')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(2c) Training, validation and test RMSE
Now let's calculate the training, validation and test RMSE of our baseline model. To do this, first create RDDs of (label, prediction) tuples for each dataset, and then call calcRMSE. Note that each RMSE can be interpreted as the average prediction error for the given dataset (in terms of number of years).
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
labelsAndPredsTrain = parsedTrainData.map(lambda x: (x.label, averageTrainYear))
rmseTrainBase = calcRMSE(labelsAndPredsTrain)

labelsAndPredsVal = parsedValData.map(lambda x: (x.label, averageTrainYear))
rmseValBase = calcRMSE(labelsAndPredsVal)

labelsAndPredsTest = parsedTestData.map(lambda x: (x.label, averageTrainYear))
rmseTestBase = calcRMSE(labelsAndPredsTest)

print 'Baseline Train RMSE = {0:.3f}'.format(rmseTrainBase)
print 'Baseline Validation RMSE = {0:.3f}'.format(rmseValBase)
print 'Baseline Test RMSE = {0:.3f}'.format(rmseTestBase)

# TEST Training, validation and test RMSE (2c)
Test.assertTrue(np.allclose([rmseTrainBase, rmseValBase, rmseTestBase],
                            [21.305869, 21.586452, 22.136957]), 'incorrect RMSE value')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(3c) Gradient descent
Next, implement a gradient descent function for linear regression and test out this function on an example.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def linregGradientDescent(trainData, numIters):
    """Calculates the weights and error for a linear regression model trained with gradient descent.

    Note:
        `DenseVector` behaves similarly to a `numpy.ndarray` and they can be used interchangably
        within this function.  For example, they both implement the `dot` method.

    Args:
        trainData (RDD of LabeledPoint): The labeled data for use in training the model.
        numIters (int): The number of iterations of gradient descent to perform.

    Returns:
        (np.ndarray, np.ndarray): A tuple of (weights, training errors).  Weights will be the
            final weights (one weight per feature) for the model, and training errors will contain
            an error (RMSE) for each iteration of the algorithm.
    """
    # The length of the training data
    n = trainData.count()
    # The number of features in the training data
    d = len(trainData.take(1)[0].features)
    w = np.zeros(d)
    alpha = 1.0
    # We will compute and store the training error after each iteration
    errorTrain = np.zeros(numIters)
    for i in range(numIters):
        # Use getLabeledPrediction from (3b) with trainData to obtain an RDD of (label, prediction)
        # tuples.  Note that the weights all equal 0 for the first iteration, so the predictions will
        # have large errors to start.
        labelsAndPredsTrain = trainData.map(lambda x: getLabeledPrediction(w, x))
        errorTrain[i] = calcRMSE(labelsAndPredsTrain)

        # Calculate the `gradient`.  Make use of the `gradientSummand` function you wrote in (3a).
        # Note that `gradient` sould be a `DenseVector` of length `d`.
        gradient = trainData.map(lambda x: gradientSummand(w, x)).sum()

        # Update the weights
        alpha_i = alpha / (n * np.sqrt(i+1))
        w -= alpha_i * gradient
    return w, errorTrain

# create a toy dataset with n = 10, d = 3, and then run 5 iterations of gradient descent
# note: the resulting...


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(4b) Predict
Now use the LinearRegressionModel.predict() method to make a prediction on a sample point.  Pass the features from a LabeledPoint into the predict() method.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
samplePoint = parsedTrainData.take(1)[0]
samplePrediction = firstModel.predict(samplePoint.features)
print samplePrediction

# TEST Predict (4b)
Test.assertTrue(np.allclose(samplePrediction, 56.8013380112),
                'incorrect value for samplePrediction')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(4d) Grid search
We're already outperforming the baseline on the validation set by almost 2 years on average, but let's see if we can do better. Perform grid search to find a good regularization parameter.  Try regParam values 1e-10, 1e-5, and 1.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
bestRMSE = rmseValLR1
bestRegParam = reg
bestModel = firstModel

numIters = 500
alpha = 1.0
miniBatchFrac = 1.0
for reg in [1e-10, 1e-5, 1]:
    model = LinearRegressionWithSGD.train(parsedTrainData, numIters, alpha,
                                          miniBatchFrac, regParam=reg,
                                          regType='l2', intercept=True)
    labelsAndPreds = parsedValData.map(lambda lp: (lp.label, model.predict(lp.features)))
    rmseValGrid = calcRMSE(labelsAndPreds)
    print rmseValGrid

    if rmseValGrid < bestRMSE:
        bestRMSE = rmseValGrid
        bestRegParam = reg
        bestModel = model
rmseValLRGrid = bestRMSE

print ('Validation RMSE:\n\tBaseline = {0:.3f}\n\tLR0 = {1:.3f}\n\tLR1 = {2:.3f}\n' +
       '\tLRGrid = {3:.3f}').format(rmseValBase, rmseValLR0, rmseValLR1, rmseValLRGrid)

# TEST Grid search (4d)
Test.assertTrue(np.allclose(17.017170, rmseValLRGrid), 'incorrect value for rmseValLRGrid')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(4e) Vary alpha and the number of iterations
In the previous grid search, we set alpha = 1 for all experiments.  Now let's see what happens when we vary alpha.  Specifically, try 1e-5 and 10 as values for alpha and also try training models for 500 iterations (as before) but also for 5 iterations. Evaluate all models on the validation set.  Note that if we set alpha too small the gradient descent will require a huge number of steps to converge to the solution, and if we use too large of an alpha it can cause numerical problems, like you'll see below for alpha = 10.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
reg = bestRegParam
modelRMSEs = []

for alpha in [1e-5, 10]:
    for numIters in [500, 5]:
        model = LinearRegressionWithSGD.train(parsedTrainData, numIters, alpha,
                                              miniBatchFrac, regParam=reg,
                                              regType='l2', intercept=True)
        labelsAndPreds = parsedValData.map(lambda lp: (lp.label, model.predict(lp.features)))
        rmseVal = calcRMSE(labelsAndPreds)
        print 'alpha = {0:.0e}, numIters = {1}, RMSE = {2:.3f}'.format(alpha, numIters, rmseVal)
        modelRMSEs.append(rmseVal)

# TEST Vary alpha and the number of iterations (4e)
expectedResults = sorted([56.969705, 56.892949, 355124752.221221])
Test.assertTrue(np.allclose(sorted(modelRMSEs)[:3], expectedResults), 'incorrect value for modelRMSEs')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(5b) Build interaction model
Now, let's build the new model.  We've done this several times now.  To implement this for the new features, we need to change a few variable names.  Remember that we should build our model from the training data and evaluate it on the validation data.
Note that you should re-run your hyperparameter search after changing features, as using the best hyperparameters from your prior model will not necessary lead to the best model.  For this exercise, we have already preset the hyperparameters to reasonable values.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
numIters = 500
alpha = 1.0
miniBatchFrac = 1.0
reg = 1e-10

modelInteract = LinearRegressionWithSGD.train(trainDataInteract, numIters, alpha,
                                              miniBatchFrac, regParam=reg,
                                              regType='l2', intercept=True)
labelsAndPredsInteract = valDataInteract.map(lambda lp: (lp.label, modelInteract.predict(lp.features)))
rmseValInteract = calcRMSE(labelsAndPredsInteract)

print ('Validation RMSE:\n\tBaseline = {0:.3f}\n\tLR0 = {1:.3f}\n\tLR1 = {2:.3f}\n\tLRGrid = ' +
       '{3:.3f}\n\tLRInteract = {4:.3f}').format(rmseValBase, rmseValLR0, rmseValLR1,
                                                 rmseValLRGrid, rmseValInteract)

# TEST Build interaction model (5b)
Test.assertTrue(np.allclose(rmseValInteract, 15.6894664683), 'incorrect value for rmseValInteract')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
(5c) Evaluate interaction model on test data
Our final step is to evaluate the new model on the test dataset.  Note that we haven't used the test set to evaluate any of our models.  Because of this, our evaluation provides us with an unbiased estimate for how our model will perform on new data.  If we had changed our model based on viewing its performance on the test set, our estimate of RMSE would likely be overly optimistic.
We'll also print the RMSE for both the baseline model and our new model.  With this information, we can see how much better our model performs than the baseline model.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
labelsAndPredsTest = testDataInteract.map(lambda lp: (lp.label, modelInteract.predict(lp.features)))
rmseTestInteract = calcRMSE(labelsAndPredsTest)

print ('Test RMSE:\n\tBaseline = {0:.3f}\n\tLRInteract = {1:.3f}'
       .format(rmseTestBase, rmseTestInteract))

# TEST Evaluate interaction model on test data (5c)
Test.assertTrue(np.allclose(rmseTestInteract, 16.3272040537),
                'incorrect value for rmseTestInteract')


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
Part 5: Add interactions between features
 (5a) Add 2-way interactions
So far, we've used the features as they were provided.  Now, we will add features that capture the two-way interactions between our existing features.  Write a function twoWayInteractions that takes in a LabeledPoint and generates a new LabeledPoint that contains the old features and the two-way interactions between them.  Note that a dataset with three features would have nine ( $ \scriptsize 3^2 $ ) two-way interactions.
You might want to use itertools.product to generate tuples for each of the possible 2-way interactions.  Remember that you can combine two DenseVector or ndarray objects using np.hstack.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
import itertools

def twoWayInteractions(lp):
    """Creates a new `LabeledPoint` that includes two-way interactions.

    Note:
        For features [x, y] the two-way interactions would be [x^2, x*y, y*x, y^2] and these
        would be appended to the original [x, y] feature list.

    Args:
        lp (LabeledPoint): The label and features for this observation.

    Returns:
        LabeledPoint: The new `LabeledPoint` should have the same label as `lp`.  Its features
            should include the features from `lp` followed by the two-way interaction features.
    """
    interactions = np.outer(lp.features, lp.features).flatten()
    return LabeledPoint(lp.label, np.hstack((lp.features, interactions)))

print twoWayInteractions(LabeledPoint(0.0, [2, 3]))

# Transform the existing train, validation, and test sets to include two-way interactions.
trainDataInteract = parsedTrainData.map(twoWayInteractions)
valDataInteract = parsedValData.map(twoWayInteractions)
testDataInteract = parsedTestData.map(twoWayInteractions)

# TEST Add two-way interactions (5a)
twoWayExample = twoWayInteractions(LabeledPoint(0.0, [2, 3]))
Test.assertTrue(np.allclose(sorted(twoWayExample.features),
                            sorted([2.0, 3.0, 4.0, 6.0, 6.0, 9.0])),
                'incorrect features generatedBy twoWayInteractions')
twoWayPoint = twoWayInteractions(LabeledPoint(1.0, [1, 2, 3]))
Test.assertTrue(np.allclose(sorted(twoWayPoint.features),
                            sorted([1.0,2.0,3.0,1.0,2.0,3.0,2.0,4.0,6.0,3.0,6.0,9.0])),
                'incorrect features generated by twoWayInteractions')
Test.assertEquals(twoWayPoint.label, 1.0, 'incorrect label generated by twoWayInteractions')
Test.assertTrue(np.allclose(sum(trainDataInteract.take(1)[0].features), 40.821870576035529),
                'incorrect features in trainDataInteract')
Test.assertTrue(np.allclose(sum(valDataInteract.take(1)[0].features), 45.457719932695696),
                'incorrect features in...


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
Visualization 5: Best model's predictions
Next, we create a visualization similar to 'Visualization 3: Predicted vs. actual' from Part 2 using the predictions from the best model from Part (4d) on the validation dataset.  Specifically, we create a color-coded scatter plot visualizing tuples storing i) the predicted value from this model and ii) true label.
Can you write Python code for it?

predictions = np.asarray(parsedValData
                         .map(lambda lp: bestModel.predict(lp.features))
                         .collect())
actual = np.asarray(parsedValData
                    .map(lambda lp: lp.label)
                    .collect())
error = np.asarray(parsedValData
                   .map(lambda lp: (lp.label, bestModel.predict(lp.features)))
                   .map(lambda (l, p): squaredError(l, p))
                   .collect())

norm = Normalize()
clrs = cmap(np.asarray(norm(error)))[:,0:3]

fig, ax = preparePlot(np.arange(0, 120, 20), np.arange(0, 120, 20))
ax.set_xlim(15, 82), ax.set_ylim(-5, 105)
plt.scatter(predictions, actual, s=14**2, c=clrs, edgecolors='#888888', alpha=0.75, linewidths=.5)
ax.set_xlabel('Predicted'), ax.set_ylabel(r'Actual')
pass


I am working on the file "ML_lab3_linear_reg_student.ipynb".
The first task is:
Visualization 6: Hyperparameter heat map
Next, we perform a visualization of hyperparameter search using a larger set of hyperparameters (with precomputed results).  Specifically, we create a heat map where the brighter colors correspond to lower RMSE values.  The first plot has a large area with brighter colors.  In order to differentiate within the bright region, we generate a second plot corresponding to the hyperparameters found within that region.
Can you write Python code for it?

from matplotlib.colors import LinearSegmentedColormap

# Saved parameters and results, to save the time required to run 36 models
numItersParams = [10, 50, 100, 250, 500, 1000]
regParams = [1e-8, 1e-6, 1e-4, 1e-2, 1e-1, 1]
rmseVal = np.array([[  20.36769649,   20.36770128,   20.36818057,   20.41795354,  21.09778437,  301.54258421],
                    [  19.04948826,   19.0495    ,   19.05067418,   19.16517726,  19.97967727,   23.80077467],
                    [  18.40149024,   18.40150998,   18.40348326,   18.59457491,  19.82155716,   23.80077467],
                    [  17.5609346 ,   17.56096749,   17.56425511,   17.88442127,  19.71577117,   23.80077467],
                    [  17.0171705 ,   17.01721288,   17.02145207,   17.44510574,  19.69124734,   23.80077467],
                    [  16.58074813,   16.58079874,   16.58586512,   17.11466904,  19.6860931 ,   23.80077467]])

numRows, numCols = len(numItersParams), len(regParams)
rmseVal = np.array(rmseVal)
rmseVal.shape = (numRows, numCols)

fig, ax = preparePlot(np.arange(0, numCols, 1), np.arange(0, numRows, 1), figsize=(8, 7), hideLabels=True,
                      gridWidth=0.)
ax.set_xticklabels(regParams), ax.set_yticklabels(numItersParams)
ax.set_xlabel('Regularization Parameter'), ax.set_ylabel('Number of Iterations')

colors = LinearSegmentedColormap.from_list('blue', ['#0022ff', '#000055'], gamma=.2)
image = plt.imshow(rmseVal,interpolation='nearest', aspect='auto',
                    cmap = colors)

# Zoom into the bottom left
numItersParamsZoom, regParamsZoom = numItersParams[-3:], regParams[:4]
rmseValZoom = rmseVal[-3:, :4]

numRows, numCols = len(numItersParamsZoom), len(regParamsZoom)

fig, ax = preparePlot(np.arange(0, numCols, 1), np.arange(0, numRows, 1), figsize=(8, 7), hideLabels=True,
                      gridWidth=0.)
ax.set_xticklabels(regParamsZoom), ax.set_yticklabels(numItersParamsZoom)
ax.set_xlabel('Regularization Parameter'), ax.set_ylabel('Number of Iterations')

colors = LinearSegmentedColormap.from_list('blue',...


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(1c) OHE features as sparse vectors
Now let's see how we can represent the OHE features for points in our sample dataset.  Using the mapping defined by the OHE dictionary from Part (1a), manually define OHE features for the three sample data points using SparseVector format.  Any feature that occurs in a point should have the value 1.0.  For example, the DenseVector for a point with features 2 and 4 would be [0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0].
Can you write Python code for it?

# Reminder of the sample features
# sampleOne = [(0, 'mouse'), (1, 'black')]
# sampleTwo = [(0, 'cat'), (1, 'tabby'), (2, 'mouse')]
# sampleThree =  [(0, 'bear'), (1, 'black'), (2, 'salmon')]

# TODO: Replace <FILL IN> with appropriate code
sampleOneOHEFeatManual = SparseVector(7, [(2, 1), (3, 1)])
sampleTwoOHEFeatManual = SparseVector(7, [(1, 1), (4, 1), (5,1)])
sampleThreeOHEFeatManual = SparseVector(7, [(0, 1), (3, 1), (6,1)])

# TEST OHE Features as sparse vectors (1c)
Test.assertTrue(isinstance(sampleOneOHEFeatManual, SparseVector),
                'sampleOneOHEFeatManual needs to be a SparseVector')
Test.assertTrue(isinstance(sampleTwoOHEFeatManual, SparseVector),
                'sampleTwoOHEFeatManual needs to be a SparseVector')
Test.assertTrue(isinstance(sampleThreeOHEFeatManual, SparseVector),
                'sampleThreeOHEFeatManual needs to be a SparseVector')
Test.assertEqualsHashed(sampleOneOHEFeatManual,
                        'ecc00223d141b7bd0913d52377cee2cf5783abd6',
                        'incorrect value for sampleOneOHEFeatManual')
Test.assertEqualsHashed(sampleTwoOHEFeatManual,
                        '26b023f4109e3b8ab32241938e2e9b9e9d62720a',
                        'incorrect value for sampleTwoOHEFeatManual')
Test.assertEqualsHashed(sampleThreeOHEFeatManual,
                        'c04134fd603ae115395b29dcabe9d0c66fbdc8a7',
                        'incorrect value for sampleThreeOHEFeatManual')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(1e) Apply OHE to a dataset
Finally, use the function from Part (1d) to create OHE features for all 3 data points in the sample dataset.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
sampleOHEData = sampleDataRDD.map(lambda x: oneHotEncoding(x, sampleOHEDictManual, len(sampleOHEDictManual)))
print sampleOHEData.collect()

# TEST Apply OHE to a dataset (1e)
sampleOHEDataValues = sampleOHEData.collect()
Test.assertTrue(len(sampleOHEDataValues) == 3, 'sampleOHEData should have three elements')
Test.assertEquals(sampleOHEDataValues[0], SparseVector(7, {2: 1.0, 3: 1.0}),
                  'incorrect OHE for first sample')
Test.assertEquals(sampleOHEDataValues[1], SparseVector(7, {1: 1.0, 4: 1.0, 5: 1.0}),
                  'incorrect OHE for second sample')
Test.assertEquals(sampleOHEDataValues[2], SparseVector(7, {0: 1.0, 3: 1.0, 6: 1.0}),
                  'incorrect OHE for third sample')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(2b) OHE Dictionary from distinct features
Next, create an RDD of key-value tuples, where each (featureID, category) tuple in sampleDistinctFeats is a key and the values are distinct integers ranging from 0 to (number of keys - 1).  Then convert this RDD into a dictionary, which can be done using the collectAsMap action.  Note that there is no unique mapping from keys to values, as all we require is that each (featureID, category) key be mapped to a unique integer between 0 and the number of keys.  In this exercise, any valid mapping is acceptable.  Use zipWithIndex followed by collectAsMap.
In our sample dataset, one valid list of key-value tuples is: [((0, 'bear'), 0), ((2, 'salmon'), 1), ((1, 'tabby'), 2), ((2, 'mouse'), 3), ((0, 'mouse'), 4), ((0, 'cat'), 5), ((1, 'black'), 6)]. The dictionary defined in Part (1a) illustrates another valid mapping between keys and integers.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
sampleOHEDict = (sampleDistinctFeats
                 .zipWithIndex()
                 .collectAsMap())
print sampleOHEDict

# TEST OHE Dictionary from distinct features (2b)
Test.assertEquals(sorted(sampleOHEDict.keys()),
                  [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),
                   (1, 'tabby'), (2, 'mouse'), (2, 'salmon')],
                  'sampleOHEDict has unexpected keys')
Test.assertEquals(sorted(sampleOHEDict.values()), range(7), 'sampleOHEDict has unexpected values')