I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
As in Week 3, we will use the sqft_living variable. For plotting purposes (connecting the dots), you'll need to sort by the values of sqft_living. For houses with identical square footage, we break the tie by their prices.
Can you write Python code for it?

sales = sales.sort(['sqft_living', 'price'])


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Let's unpack that plt.plot() command. The first pair of SArrays we passed are the 1st power of sqft and the actual price we then ask it to print these as dots '.'. The next pair we pass is the 1st power of sqft and the predicted values from the linear model. We ask these to be plotted as a line '-'.
We can see, not surprisingly, that the predicted values all fall on a line, specifically the one with slope 280 and intercept -43579. What if we wanted to plot a second degree polynomial?
Can you write Python code for it?

poly2_data = polynomial_sframe(sales['sqft_living'], 2)
my_features = poly2_data.column_names() # get the name of the features
poly2_data['price'] = sales['price'] # add price to the data since it's the target
model2 = graphlab.linear_regression.create(poly2_data, target = 'price', features = my_features, validation_set = None)

model2.get("coefficients")

plt.plot(poly2_data['power_1'],poly2_data['price'],'.',
        poly2_data['power_1'], model2.predict(poly2_data),'-')


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Next you should write a loop that does the following:
* For degree in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] (to get this in python type range(1, 15+1))
    * Build an SFrame of polynomial data of train_data['sqft_living'] at the current degree
    * hint: my_features = poly_data.column_names() gives you a list e.g. ['power_1', 'power_2', 'power_3'] which you might find useful for graphlab.linear_regression.create( features = my_features)
    * Add train_data['price'] to the polynomial SFrame
    * Learn a polynomial regression model to sqft vs price with that degree on TRAIN data
    * Compute the RSS on VALIDATION data (here you will want to use .predict()) for that degree and you will need to make a polynmial SFrame using validation data.
* Report which degree had the lowest RSS on validation data (remember python indexes from 0)
(Note you can turn off the print out of linear_regression.create() with verbose = False)
Quiz Question: Which degree (1, 2, …, 15) had the lowest RSS on Validation data?
Now that you have chosen the degree of your polynomial using validation data, compute the RSS of this model on TEST data. Report the RSS on your quiz.
Can you write Python code for it?

for i in range(1,16):
    poly_data = polynomial_sframe(training['sqft_living'], i)
    my_features = poly_data.column_names()
    poly_data['price'] = training['price'] # add price to the data since it's the target
    model = graphlab.linear_regression.create(poly_data, target = 'price', features = my_features, validation_set = None, verbose = False)
    validation_data = polynomial_sframe(validation['sqft_living'], i)
    validation_data['price'] = validation['price']
    predictions = model.predict(validation_data)
    RSS = ((predictions - validation_data['price']) * (predictions - validation_data['price'])).sum()
    print str(RSS)


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Now try a 15th degree polynomial:
Can you write Python code for it?

poly1_data = polynomial_sframe(sales['sqft_living'], 15)
poly1_data['price'] = sales['price'] # add price to the data since it's the target

model2 = graphlab.linear_regression.create(poly1_data, target = 'price', features = ['power_15'], validation_set = None)
plt.plot(poly1_data['power_15'],poly1_data['price'],'.',
        poly1_data['power_15'], model2.predict(poly1_data),'-')


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Polynomial_sframe function
Using the hints above complete the following function to create an SFrame consisting of the powers of an SArray up to a specific degree:
Can you write Python code for it?

def polynomial_sframe(feature, degree):
    # assume that degree >= 1
    # initialize the SFrame:
    poly_sframe = graphlab.SFrame()
    # and set poly_sframe['power_1'] equal to the passed feature
    poly_sframe['power_1'] = feature
    # first check if degree > 1
    if degree > 1:
        # then loop over the remaining degrees:
        # range usually starts at 0 and stops at the endpoint-1. We want it to start at 2 and stop at degree
        for power in range(2, degree+1):
            # first we'll give the column a name:
            name = 'power_' + str(power)
            # then assign poly_sframe[name] to the appropriate power of feature
            poly_sframe[name] = feature**power
    return poly_sframe


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Quiz Question: what is the RSS on TEST data for the model with the degree selected from Validation data? (Make sure you got the correct degree from the previous question)
Can you write Python code for it?

poly_data = polynomial_sframe(testing['sqft_living'], 6)
my_features = poly_data.column_names()
poly_data['price'] = testing['price'] # add price to the data since it's the target
model = graphlab.linear_regression.create(poly_data, target = 'price', features = my_features, validation_set = None, verbose = False)
validation_data = polynomial_sframe(validation['sqft_living'], i)
validation_data['price'] = validation['price']
predictions = model.predict(validation_data)
RSS = ((predictions - validation_data['price']) * (predictions - validation_data['price'])).sum()
print str(RSS)


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
The resulting model looks like half a parabola. Try on your own to see what the cubic looks like:
Can you write Python code for it?

poly1_data = polynomial_sframe(sales['sqft_living'], 3)
poly1_data['price'] = sales['price'] # add price to the data since it's the target
model2 = graphlab.linear_regression.create(poly1_data, target = 'price', features = ['power_2'], validation_set = None)
plt.plot(poly1_data['power_2'],poly1_data['price'],'.',
        poly1_data['power_2'], model2.predict(poly1_data),'-')

model3 = graphlab.linear_regression.create(poly1_data, target = 'price', features = ['power_3'], validation_set = None)
plt.plot(poly1_data['power_3'],poly1_data['price'],'.',
        poly1_data['power_3'], model3.predict(poly1_data),'-')


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Visualizing polynomial regression
Let's use matplotlib to visualize what a polynomial regression looks like on some real data.
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
We can create an empty SFrame using graphlab.SFrame() and then add any columns to it with ex_sframe['column_name'] = value. For example we create an empty SFrame and make the column 'power_1' to be the first power of tmp (i.e. tmp itself).
Can you write Python code for it?

ex_sframe = graphlab.SFrame()
ex_sframe['power_1'] = tmp
print ex_sframe


I am working on the file "Regression/Assignment_three/.ipynb_checkpoints/week-3-polynomial-regression-assignment-blank-checkpoint.ipynb".
The first task is:
What do you think of the 15th degree polynomial? Do you think this is appropriate? If we were to change the data do you think you'd get pretty much the same curve? Let's take a look.
Changing the data and re-learning
We're going to split the sales data into four subsets of roughly equal size. Then you will estimate a 15th degree polynomial model on all four subsets of the data. Print the coefficients (you should use .print_rows(num_rows = 16) to view all of them) and plot the resulting fit (as we did above). The quiz will ask you some questions about these results.
To split the sales data into four subsets, we perform the following steps:
* First split sales into 2 subsets with .random_split(0.5, seed=0).
* Next split the resulting subsets into 2 more subsets each. Use .random_split(0.5, seed=0).
We set seed=0 in these steps so that different users get consistent results.
You should end up with 4 subsets (set_1, set_2, set_3, set_4) of approximately equal size.
Can you write Python code for it?

temp_set, temp_set_plus = sales.random_split(.5,seed=0)
set_1, set_2 = temp_set.random_split(.5,seed=0)
set_3, set_4 = temp_set_plus.random_split(.5,seed=0)


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
As in Week 3, we will use the sqft_living variable. For plotting purposes (connecting the dots), you'll need to sort by the values of sqft_living. For houses with identical square footage, we break the tie by their prices.
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Fit a 15th degree polynomial on set_1, set_2, set_3, and set_4 using sqft_living to predict prices. Print the coefficients and make a plot of the resulting model.
Can you write Python code for it?

poly_data = polynomial_sframe(set_1['sqft_living'], 15)
poly_data['price'] = set_1['price'] # add price to the data since it's the target
model = graphlab.linear_regression.create(poly_data, target = 'price', features = ['power_15'], validation_set = None)
model.get("coefficients")
# plt.plot(poly_data['power_15'],poly_data['price'],'.',
#         poly_data['power_15'], model.predict(poly_data),'-')

poly_data = polynomial_sframe(set_2['sqft_living'], 15)
poly_data['price'] = set_2['price'] # add price to the data since it's the target
model = graphlab.linear_regression.create(poly_data, target = 'price', features = ['power_15'], validation_set = None)
model.get("coefficients")
plt.plot(poly_data['power_15'],poly_data['price'],'.',
        poly_data['power_15'], model.predict(poly_data),'-')

poly_data = polynomial_sframe(set_3['sqft_living'], 15)
poly_data['price'] = set_3['price'] # add price to the data since it's the target
model = graphlab.linear_regression.create(poly_data, target = 'price', features = ['power_15'], validation_set = None)
model.get("coefficients")
plt.plot(poly_data['power_15'],poly_data['price'],'.',
        poly_data['power_15'], model.predict(poly_data),'-')

poly_data = polynomial_sframe(set_4['sqft_living'], 15)
poly_data['price'] = set_4['price'] # add price to the data since it's the target
model = graphlab.linear_regression.create(poly_data, target = 'price', features = ['power_15'], validation_set = None)
model.get("coefficients")
# plt.plot(poly_data['power_15'],poly_data['price'],'.',
#         poly_data['power_15'], model.predict(poly_data),'-')


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Let's unpack that plt.plot() command. The first pair of SArrays we passed are the 1st power of sqft and the actual price we then ask it to print these as dots '.'. The next pair we pass is the 1st power of sqft and the predicted values from the linear model. We ask these to be plotted as a line '-'.
We can see, not surprisingly, that the predicted values all fall on a line, specifically the one with slope 280 and intercept -43579. What if we wanted to plot a second degree polynomial?
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
NOTE: for all the models in this notebook use validation_set = None to ensure that all results are consistent across users.
Can you write Python code for it?

model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = ['power_1'], validation_set = None)

#let's take a look at the weights before we plot
model1.get("coefficients")

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(poly1_data['power_1'],poly1_data['price'],'.',
        poly1_data['power_1'], model1.predict(poly1_data),'-')


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Next we're going to write a polynomial function that takes an SArray and a maximal degree and returns an SFrame with columns containing the SArray to all the powers up to the maximal degree.
The easiest way to apply a power to an SArray is to use the .apply() and lambda x: functions.
For example to take the example array and compute the third power we can do as follows: (note running this cell the first time may take longer than expected since it loads graphlab)
Can you write Python code for it?

tmp = graphlab.SArray([1., 2., 3.])
tmp_cubed = tmp.apply(lambda x: x**3)
print tmp
print tmp_cubed


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Next you should write a loop that does the following:
* For degree in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] (to get this in python type range(1, 15+1))
    * Build an SFrame of polynomial data of train_data['sqft_living'] at the current degree
    * hint: my_features = poly_data.column_names() gives you a list e.g. ['power_1', 'power_2', 'power_3'] which you might find useful for graphlab.linear_regression.create( features = my_features)
    * Add train_data['price'] to the polynomial SFrame
    * Learn a polynomial regression model to sqft vs price with that degree on TRAIN data
    * Compute the RSS on VALIDATION data (here you will want to use .predict()) for that degree and you will need to make a polynmial SFrame using validation data.
* Report which degree had the lowest RSS on validation data (remember python indexes from 0)
(Note you can turn off the print out of linear_regression.create() with verbose = False)
Quiz Question: Which degree (1, 2, …, 15) had the lowest RSS on Validation data?
Now that you have chosen the degree of your polynomial using validation data, compute the RSS of this model on TEST data. Report the RSS on your quiz.
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Now try a 15th degree polynomial:
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Polynomial_sframe function
Using the hints above complete the following function to create an SFrame consisting of the powers of an SArray up to a specific degree:
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Quiz Question: what is the RSS on TEST data for the model with the degree selected from Validation data? (Make sure you got the correct degree from the previous question)
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
The resulting model looks like half a parabola. Try on your own to see what the cubic looks like:
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
To test your function consider the smaller tmp variable and what you would expect the outcome of the following call:
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Visualizing polynomial regression
Let's use matplotlib to visualize what a polynomial regression looks like on some real data.
Can you write Python code for it?


I am working on the file "Regression/Assignment_three/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
We can create an empty SFrame using graphlab.SFrame() and then add any columns to it with ex_sframe['column_name'] = value. For example we create an empty SFrame and make the column 'power_1' to be the first power of tmp (i.e. tmp itself).
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Compute RSS
Now that we can make predictions given the model, let's write a function to compute the RSS of the model. Complete the function below to calculate RSS given the model, data, and the outcome.
Can you write Python code for it?

import math
def get_residual_sum_of_squares(model, data, outcome):
    # First get the predictions
    predict = model.predict(data)
    # Then compute the residuals/errors
    residuals = []
    for i in range(0, len(predict)):
        error = outcome[i] - predict[i]
        residuals.append(math.pow(error,2))
    # Then square and add them up
    RSS = reduce(lambda x,y : x + y, residuals)
    return(RSS)


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Create some new features
Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms.
You will use the logarithm function to create a new feature. so first you should import it from the math library.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Learning Multiple Models
Now we will learn the weights for three (nested) models for predicting house prices. The first model will have the fewest features the second model will add one more feature and the third will add a few more:
* Model 1: squarefeet, # bedrooms, # bathrooms, latitude & longitude
* Model 2: add bedrooms*bathrooms
* Model 3: Add log squarefeet, bedrooms squared, and the (nonsensical) latitude + longitude
Can you write Python code for it?

model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long']
model_2_features = model_1_features + ['bed_bath_rooms']
model_3_features = model_2_features + ['bedrooms_squared', 'log_sqft_living', 'lat_plus_long']


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Learning a multiple regression model
Recall we can use the following code to learn a multiple regression model predicting 'price' based on the following features:
example_features = ['sqft_living', 'bedrooms', 'bathrooms'] on training data with the following code:
(Aside: We set validation_set = None to ensure that the results are always the same)
Can you write Python code for it?

example_features = ['sqft_living', 'bedrooms', 'bathrooms']
example_model = graphlab.linear_regression.create(train_data, target = 'price', features = example_features,
                                                  validation_set = None)


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Making Predictions
In the gradient descent notebook we use numpy to do our regression. In this book we will use existing graphlab create functions to analyze multiple regressions.
Recall that once a model is built we can use the .predict() function to find the predicted values for data we pass. For example using the example model above:
Can you write Python code for it?

example_predictions = example_model.predict(train_data)
print example_predictions[0] # should be 271789.505878


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Now that you have the features, learn the weights for the three different models for predicting target = 'price' using graphlab.linear_regression.create() and look at the value of the weights/coefficients:
Can you write Python code for it?

# Learn the three models: (don't forget to set validation_set = None)
model_1 = graphlab.linear_regression.create(train_data, target = 'price', features = model_1_features,
                                                  validation_set = None)
model_2 = graphlab.linear_regression.create(train_data, target = 'price', features = model_2_features,
                                                  validation_set = None)
model_3 = graphlab.linear_regression.create(train_data, target = 'price', features = model_3_features,
                                                  validation_set = None)

# Examine/extract each model's coefficients:
model_1_weight_summary = model_1.get("coefficients")
print model_1_weight_summary
model_2_weight_summary = model_2.get("coefficients")
print model_2_weight_summary
model_3_weight_summary = model_3.get("coefficients")
print model_3_weight_summary


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 1?
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 2?
Think about what this means.
Comparing multiple models
Now that you've learned three models and extracted the model weights we want to evaluate which model is best.
First use your functions from earlier to compute the RSS on TRAINING Data for each of the three models.
Can you write Python code for it?

# Compute the RSS on TRAINING data for each of the three models and record the values:
rss_model_1_train = get_residual_sum_of_squares(model_1, train_data, train_data['price'])
print rss_model_1_train
rss_model_2_train = get_residual_sum_of_squares(model_2, train_data, train_data['price'])
print rss_model_2_train
rss_model_3_train = get_residual_sum_of_squares(model_3, train_data, train_data['price'])
print rss_model_3_train


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Quiz Question: Which model (1, 2 or 3) has lowest RSS on TRAINING Data? Is this what you expected?
Now compute the RSS on on TEST data for each of the three models.
Can you write Python code for it?

# Compute the RSS on TESTING data for each of the three models and record the values:
rss_model_1_test = get_residual_sum_of_squares(model_1, test_data, test_data['price'])
print rss_model_1_test
rss_model_2_test = get_residual_sum_of_squares(model_2, test_data, test_data['price'])
print rss_model_2_test
rss_model_3_test = get_residual_sum_of_squares(model_3, test_data, test_data['price'])
print rss_model_3_test


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Test your function by computing the RSS on TEST data for the example model:
Can you write Python code for it?

rss_example_train = get_residual_sum_of_squares(example_model, test_data, test_data['price'])
print rss_example_train # should be 2.7376153833e+14


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Although the gradient descent is designed for multiple regression since the constant is now a feature we can use the gradient descent function to estimat the parameters in the simple regression on squarefeet. The folowing cell sets up the feature_matrix, output, initial weights and step size for the first model:
Can you write Python code for it?

# let's test out the gradient descent
simple_features = ['sqft_living']
my_output = 'price'
(simple_feature_matrix, output) = get_numpy_data(train_data, simple_features, my_output)
initial_weights = np.array([-47000., 1.])
step_size = 7e-12
tolerance = 2.5e7


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Computing the Derivative
We are now going to move to computing the derivative of the regression cost function. Recall that the cost function is the sum over the data points of the squared difference between an observed output and a predicted output.
Since the derivative of a sum is the sum of the derivatives we can compute the derivative for a single data point and then sum over data points. We can write the squared difference between the observed output and predicted output for a single point as follows:
(w[0]*[CONSTANT] + w[1]*[feature_1] + ... + w[i] *[feature_i] + ... +  w[k]*[feature_k] - output)^2
Where we have k features and a constant. So the derivative with respect to weight w[i] by the chain rule is:
2*(w[0]*[CONSTANT] + w[1]*[feature_1] + ... + w[i] *[feature_i] + ... +  w[k]*[feature_k] - output)* [feature_i]
The term inside the paranethesis is just the error (difference between prediction and output). So we can re-write this as:
2*error*[feature_i]
That is, the derivative for the weight for feature i is the sum (over data points) of 2 times the product of the error and the feature itself. In the case of the constant then this is just twice the sum of the errors!
Recall that twice the sum of the product of two vectors is just twice the dot product of the two vectors. Therefore the derivative for the weight for feature_i is just two times the dot product between the values of feature_i and the current errors.
With this in mind complete the following derivative function which computes the derivative of the weight given the value of the feature (over all data points) and the errors (over all data points).
Can you write Python code for it?

def feature_derivative(errors, feature):
    # Assume that errors and feature are both numpy arrays of the same length (number of data points)
    # compute twice the dot product of these vectors as 'derivative' and return the value
    derivative = 2*np.dot(errors, feature)
    return(derivative)


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
For testing let's use the 'sqft_living' feature and a constant as our features and price as our output:
Can you write Python code for it?

(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price') # the [] around 'sqft_living' makes it a list
print example_features[0,:] # this accesses the first row of the data the ':' indicates 'all columns'
print example_output[0] # and the corresponding output


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
How do your weights compare to those achieved in week 1 (don't expect them to be exactly the same)?
Quiz Question: What is the value of the weight for sqft_living -- the second element of ‘simple_weights’ (rounded to 1 decimal place)?
Use your newly estimated weights and your predict_output() function to compute the predictions on all the TEST data (you will need to create a numpy array of the test feature_matrix and test output first:
Can you write Python code for it?

(test_simple_feature_matrix, test_output) = get_numpy_data(test_data, simple_features, my_output)


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
If we want to do any "feature engineering" like creating new features or adjusting existing ones we should do this directly using the SFrames as seen in the other Week 2 notebook. For this notebook, however, we will work with the existing features.
Convert to Numpy Array
Although SFrames offer a number of benefits to users (especially when using Big Data and built-in graphlab functions) in order to understand the details of the implementation of algorithms it's important to work with a library that allows for direct (and optimized) matrix operations. Numpy is a Python solution to work with matrices (or any multi-dimensional "array").
Recall that the predicted value given the weights and the features is just the dot product between the feature and weight vector. Similarly, if we put all of the features row-by-row in a matrix then the predicted value for all the observations can be computed by right multiplying the "feature matrix" by the "weight vector".
First we need to take the SFrame of our data and convert it into a 2D numpy array (also called a matrix). To do this we use graphlab's built in .to_dataframe() which converts the SFrame into a Pandas (another python library) dataframe. We can then use Panda's .as_matrix() to convert the dataframe into a numpy matrix.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Now that you have the predictions on test data, compute the RSS on the test data set. Save this value for comparison later. Recall that RSS is the sum of the squared errors (difference between prediction and output).
Can you write Python code for it?

rss = 0
for i in range(0, len(test_predictions)):
    error = test_predictions[i] - test_data['price'][i]
    rss += error * error
print rss


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Quiz Question: What is the predicted price for the 1st house in the TEST data set for model 1 (round to nearest dollar)?
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Running a multiple regression
Now we will use more than one actual feature. Use the following code to produce the weights for a second model with the following parameters:
Can you write Python code for it?

model_features = ['sqft_living', 'sqft_living15'] # sqft_living15 is the average squarefeet for the nearest 15 neighbors.
my_output = 'price'
(feature_matrix, output) = get_numpy_data(train_data, model_features, my_output)
initial_weights = np.array([-100000., 1., 1.])
step_size = 4e-12
tolerance = 1e9


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Use the above parameters to estimate the model weights. Record these values for your quiz.
Can you write Python code for it?

simple_weights = regression_gradient_descent(feature_matrix, output,initial_weights, step_size, tolerance)
print simple_weights


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Use your newly estimated weights and the predict_output function to compute the predictions on the TEST data. Don't forget to create a numpy array for these features from the test set first!
Quiz Question: What is the predicted price for the 1st house in the TEST data set for model 2 (round to nearest dollar)?
Can you write Python code for it?

(test_simple_feature_matrix, test_output) = get_numpy_data(test_data, model_features, my_output)

test_predictions = predict_output(test_simple_feature_matrix, simple_weights)


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
What is the actual price for the 1st house in the test data set?
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/.ipynb_checkpoints/week-2-multiple-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
np.dot() also works when dealing with a matrix and a vector. Recall that the predictions from all the observations is just the RIGHT (as in weights on the right) dot product between the features matrix and the weights vector. With this in mind finish the following predict_output function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?

def predict_output(feature_matrix, weights):
    # assume feature_matrix is a numpy matrix containing the features as columns and weights is a corresponding numpy array
    # create the predictions vector by using np.dot()
    predictions = np.dot(feature_matrix, weights)
    return(predictions)


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Compute RSS
Now that we can make predictions given the model, let's write a function to compute the RSS of the model. Complete the function below to calculate RSS given the model, data, and the outcome.
Can you write Python code for it?

import math
def get_residual_sum_of_squares(model, data, outcome):
    # First get the predictions
    predict = model.predict(data)
    # Then compute the residuals/errors
    residuals = []
    for i in range(0, len(predict)):
        error = outcome[i] - predict[i]
        residuals.append(math.pow(error,2))
    # Then square and add them up
    RSS = reduce(lambda x,y : x + y, residuals)
    return(RSS)


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Create some new features
Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms.
You will use the logarithm function to create a new feature. so first you should import it from the math library.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Learning Multiple Models
Now we will learn the weights for three (nested) models for predicting house prices. The first model will have the fewest features the second model will add one more feature and the third will add a few more:
* Model 1: squarefeet, # bedrooms, # bathrooms, latitude & longitude
* Model 2: add bedrooms*bathrooms
* Model 3: Add log squarefeet, bedrooms squared, and the (nonsensical) latitude + longitude
Can you write Python code for it?

model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long']
model_2_features = model_1_features + ['bed_bath_rooms']
model_3_features = model_2_features + ['bedrooms_squared', 'log_sqft_living', 'lat_plus_long']


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Learning a multiple regression model
Recall we can use the following code to learn a multiple regression model predicting 'price' based on the following features:
example_features = ['sqft_living', 'bedrooms', 'bathrooms'] on training data with the following code:
(Aside: We set validation_set = None to ensure that the results are always the same)
Can you write Python code for it?

example_features = ['sqft_living', 'bedrooms', 'bathrooms']
example_model = graphlab.linear_regression.create(train_data, target = 'price', features = example_features,
                                                  validation_set = None)


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Making Predictions
In the gradient descent notebook we use numpy to do our regression. In this book we will use existing graphlab create functions to analyze multiple regressions.
Recall that once a model is built we can use the .predict() function to find the predicted values for data we pass. For example using the example model above:
Can you write Python code for it?

example_predictions = example_model.predict(train_data)
print example_predictions[0] # should be 271789.505878


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Now that you have the features, learn the weights for the three different models for predicting target = 'price' using graphlab.linear_regression.create() and look at the value of the weights/coefficients:
Can you write Python code for it?

# Learn the three models: (don't forget to set validation_set = None)
model_1 = graphlab.linear_regression.create(train_data, target = 'price', features = model_1_features,
                                                  validation_set = None)
model_2 = graphlab.linear_regression.create(train_data, target = 'price', features = model_2_features,
                                                  validation_set = None)
model_3 = graphlab.linear_regression.create(train_data, target = 'price', features = model_3_features,
                                                  validation_set = None)

# Examine/extract each model's coefficients:
model_1_weight_summary = model_1.get("coefficients")
print model_1_weight_summary
model_2_weight_summary = model_2.get("coefficients")
print model_2_weight_summary
model_3_weight_summary = model_3.get("coefficients")
print model_3_weight_summary


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 1?
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 2?
Think about what this means.
Comparing multiple models
Now that you've learned three models and extracted the model weights we want to evaluate which model is best.
First use your functions from earlier to compute the RSS on TRAINING Data for each of the three models.
Can you write Python code for it?

# Compute the RSS on TRAINING data for each of the three models and record the values:
rss_model_1_train = get_residual_sum_of_squares(model_1, train_data, train_data['price'])
print rss_model_1_train
rss_model_2_train = get_residual_sum_of_squares(model_2, train_data, train_data['price'])
print rss_model_2_train
rss_model_3_train = get_residual_sum_of_squares(model_3, train_data, train_data['price'])
print rss_model_3_train


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.
bedrooms times bathrooms gives what's called an "interaction" feature. It is large when both of them are large.
Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.
Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why)

Quiz Question: What is the mean (arithmetic average) value of your 4 new features on TEST data? (round to 2 digits)
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Test your function by computing the RSS on TEST data for the example model:
Can you write Python code for it?

rss_example_train = get_residual_sum_of_squares(example_model, test_data, test_data['price'])
print rss_example_train # should be 2.7376153833e+14


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
A few things to note before we run the gradient descent. Since the gradient is a sum over all the data points and involves a product of an error and a feature the gradient itself will be very large since the features are large (squarefeet) and the output is large (prices). So while you might expect "tolerance" to be small, small is only relative to the size of the features.
For similar reasons the step size will be much smaller than you might expect but this is because the gradient has such large values.
Running the Gradient Descent as Simple Regression
First let's split the data into training and test data.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Computing the Derivative
We are now going to move to computing the derivative of the regression cost function. Recall that the cost function is the sum over the data points of the squared difference between an observed output and a predicted output.
Since the derivative of a sum is the sum of the derivatives we can compute the derivative for a single data point and then sum over data points. We can write the squared difference between the observed output and predicted output for a single point as follows:
(w[0]*[CONSTANT] + w[1]*[feature_1] + ... + w[i] *[feature_i] + ... +  w[k]*[feature_k] - output)^2
Where we have k features and a constant. So the derivative with respect to weight w[i] by the chain rule is:
2*(w[0]*[CONSTANT] + w[1]*[feature_1] + ... + w[i] *[feature_i] + ... +  w[k]*[feature_k] - output)* [feature_i]
The term inside the paranethesis is just the error (difference between prediction and output). So we can re-write this as:
2*error*[feature_i]
That is, the derivative for the weight for feature i is the sum (over data points) of 2 times the product of the error and the feature itself. In the case of the constant then this is just twice the sum of the errors!
Recall that twice the sum of the product of two vectors is just twice the dot product of the two vectors. Therefore the derivative for the weight for feature_i is just two times the dot product between the values of feature_i and the current errors.
With this in mind complete the following derivative function which computes the derivative of the weight given the value of the feature (over all data points) and the errors (over all data points).
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
For testing let's use the 'sqft_living' feature and a constant as our features and price as our output:
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Gradient Descent
Now we will write a function that performs a gradient descent. The basic premise is simple. Given a starting point we update the current weights by moving in the negative gradient direction. Recall that the gradient is the direction of increase and therefore the negative gradient is the direction of decrease and we're trying to minimize a cost function.
The amount by which we move in the negative gradient direction  is called the 'step size'. We stop when we are 'sufficiently close' to the optimum. We define this by requiring that the magnitude (length) of the gradient vector to be smaller than a fixed 'tolerance'.
With this in mind, complete the following gradient descent function below using your derivative function above. For each step in the gradient descent we update the weight for each feature befofe computing our stopping criteria
Can you write Python code for it?

from math import sqrt # recall that the magnitude/length of a vector [g[0], g[1], g[2]] is sqrt(g[0]^2 + g[1]^2 + g[2]^2)

def regression_gradient_descent(feature_matrix, output, initial_weights, step_size, tolerance):
    converged = False
    weights = np.array(initial_weights) # make sure it's a numpy array
    gradient_magnitude = 0
    while not converged:
        # compute the predictions based on feature_matrix and weights using your predict_output() function
        predictions = predict_output(feature_matrix, weights)
        # compute the errors as predictions - output
        errors = predictions - output
        gradient_sum_squares = 0 # initialize the gradient sum of squares
        # while we haven't reached the tolerance yet, update each feature's weight
        for i in range(len(weights)): # loop over each weight
            # Recall that feature_matrix[:, i] is the feature column associated with weights[i]
            # compute the derivative for weight[i]:
            drivative = feature_derivative(errors, feature_matrix[:, i])
            # add the squared value of the derivative to the gradient magnitude (for assessing convergence)
            gradient_sum_squares += drivative * drivative
            # subtract the step size times the derivative from the current weight
            weights[i] -= step_size * drivative
        # compute the square-root of the gradient sum of squares to get the gradient matnigude:
        gradient_magnitude = sqrt(gradient_sum_squares)
        if gradient_magnitude < tolerance:
            converged = True
    return(weights)


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
How do your weights compare to those achieved in week 1 (don't expect them to be exactly the same)?
Quiz Question: What is the value of the weight for sqft_living -- the second element of ‘simple_weights’ (rounded to 1 decimal place)?
Use your newly estimated weights and your predict_output() function to compute the predictions on all the TEST data (you will need to create a numpy array of the test feature_matrix and test output first:
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Now compute your predictions using test_simple_feature_matrix and your weights from above.
Can you write Python code for it?

test_predictions = predict_output(test_simple_feature_matrix, simple_weights)


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Now that you have the predictions on test data, compute the RSS on the test data set. Save this value for comparison later. Recall that RSS is the sum of the squared errors (difference between prediction and output).
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Now we will write a function that will accept an SFrame, a list of feature names (e.g. ['sqft_living', 'bedrooms']) and an target feature e.g. ('price') and will return two things:
* A numpy matrix whose columns are the desired features plus a constant column (this is how we create an 'intercept')
* A numpy array containing the values of the output
With this in mind, complete the following function (where there's an empty line you should write a line of code that does what the comment above indicates)
Please note you will need GraphLab Create version at least 1.7.1 in order for .to_numpy() to work!
Can you write Python code for it?

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # this is how you add a constant column to an SFrame
    # add the column 'constant' to the front of the features list so that we can extract it along with the others:
    features = ['constant'] + features # this is how you combine two lists
    # select the columns of data_SFrame given by the features list into the SFrame features_sframe (now including constant):
    features_sframe = data_sframe[features]
    # the following line will convert the features_SFrame into a numpy matrix:
    feature_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the output to the SArray output_sarray
    output_sarray = data_sframe['price']
    # the following will convert the SArray into a numpy array by first converting it to a list
    output_array = output_sarray.to_numpy()
    return(feature_matrix, output_array)


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Quiz Question: Which estimate was closer to the true price for the 1st house on the Test data set, model 1 or model 2?
Now use your predictions and the output to compute the RSS for model 2 on TEST data.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Quiz Question: Which model (1 or 2) has lowest RSS on all of the TEST data?
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
To test your feature derivartive run the following:
Can you write Python code for it?

(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price')
my_weights = np.array([0., 0.]) # this makes all the predictions 0
test_predictions = predict_output(example_features, my_weights)
# just like SFrames 2 numpy arrays can be elementwise subtracted with '-':
errors = test_predictions - example_output # prediction errors in this case is just the -example_output
feature = example_features[:,0] # let's compute the derivative with respect to 'constant', the ":" indicates "all rows"
derivative = feature_derivative(errors, feature)
print derivative
print -np.sum(example_output)*2 # should be the same as derivative


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Use the above parameters to estimate the model weights. Record these values for your quiz.
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
Use your newly estimated weights and the predict_output function to compute the predictions on the TEST data. Don't forget to create a numpy array for these features from the test set first!
Quiz Question: What is the predicted price for the 1st house in the TEST data set for model 2 (round to nearest dollar)?
Can you write Python code for it?


I am working on the file "Regression/Assignment_two/week-2-multiple-regression-assignment-2-blank.ipynb".
The first task is:
What is the actual price for the 1st house in the test data set?
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Applying L1 penalty requires adding an extra parameter (l1_penalty) to the linear regression call in GraphLab Create. (Other tools may have separate implementations of LASSO.)  Note that it's important to set l2_penalty=0 to ensure we don't introduce an additional L2 penalty.
Can you write Python code for it?

model_all = graphlab.linear_regression.create(sales, target='price', features=all_features,
                                              validation_set=None,
                                              l2_penalty=0., l1_penalty=1e10)


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Create new features
As in Week 2, we consider features that are some transformations of inputs.
Can you write Python code for it?

from math import log, sqrt
sales['sqft_living_sqrt'] = sales['sqft_living'].apply(sqrt)
sales['sqft_lot_sqrt'] = sales['sqft_lot'].apply(sqrt)
sales['bedrooms_square'] = sales['bedrooms']*sales['bedrooms']

# In the dataset, 'floors' was defined with type string,
# so we'll convert them to float, before creating a new feature.
sales['floors'] = sales['floors'].astype(float)
sales['floors_square'] = sales['floors']*sales['floors']


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Exploring the larger range of values to find a narrow range with the desired sparsity
Let's define a wide range of possible l1_penalty_values:
Can you write Python code for it?

l1_penalty_values = np.logspace(8, 10, num=20)


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Find what features had non-zero weight.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Note that a majority of the weights have been set to zero. So by setting an L1 penalty that's large enough, we are performing a subset selection.
QUIZ QUESTION:
According to this list of weights, which of the features have been chosen?
Selecting an L1 penalty
To find a good L1 penalty, we will explore multiple values using a validation set. Let us do three way split into train, validation, and test sets:
* Split our sales data into 2 sets: training and test
* Further split our training data into two sets: train, validation
Be very careful that you use seed = 1 to ensure you get the same answer!
Can you write Python code for it?

(training_and_validation, testing) = sales.random_split(.9,seed=1) # initial train/test split
(training, validation) = training_and_validation.random_split(0.5, seed=1) # split training into train and validate


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Now, implement a loop that search through this space of possible l1_penalty values:

For l1_penalty in np.logspace(8, 10, num=20):
Fit a regression model with a given l1_penalty on TRAIN data. Specify l1_penalty=l1_penalty and l2_penalty=0. in the parameter list. When you call linear_regression.create() make sure you set validation_set = None
Extract the weights of the model and count the number of nonzeros. Save the number of nonzeros to a list.
Hint: model['coefficients']['value'] gives you an SArray with the parameters you learned.  If you call the method .nnz() on it, you will find the number of non-zero parameters!
Can you write Python code for it?

for l1_penalty in np.logspace(8, 10, num=20):
    A = 0
    predictions = 0
    print l1_penalty
    model_all = graphlab.linear_regression.create(training, target='price', features=all_features,
                                              validation_set=None, verbose = False,
                                              l2_penalty=0, l1_penalty=l1_penalty)
    predictions=model_all.predict(validation)
    A = get_RSS(predictions,validation['price'])
    print A
    print model_all['coefficients']['value'].nnz()


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Out of this large range, we want to find the two ends of our desired narrow range of l1_penalty.  At one end, we will have l1_penalty values that have too few non-zeros, and at the other end, we will have an l1_penalty that has too many non-zeros.
More formally, find:
* The largest l1_penalty that has more non-zeros than max_nonzero (if we pick a penalty smaller than this value, we will definitely have too many non-zero weights)
    * Store this value in the variable l1_penalty_min (we will use it later)
* The smallest l1_penalty that has fewer non-zeros than max_nonzero (if we pick a penalty larger than this value, we will definitely have too few non-zero weights)
    * Store this value in the variable l1_penalty_max (we will use it later)
Hint: there are many ways to do this, e.g.:
* Programmatically within the loop above
* Creating a list with the number of non-zeros for each value of l1_penalty and inspecting it to find the appropriate boundaries.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTIONS
What values did you find for l1_penalty_min andl1_penalty_max?
Exploring the narrow range of values to find the solution with the right number of non-zeros that has lowest RSS on the validation set
We will now explore the narrow region of l1_penalty values we found:
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-1-blank-checkpoint.ipynb".
The first task is:
Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this variable will mostly affect houses with many bedrooms.
On the other hand, taking square root of sqft_living will decrease the separation between big house and small house. The owner may not be exactly twice as happy for getting a house that is twice as big.

Learn regression weights with L1 penalty
Let us fit a model with all the features available, plus the features we just created above.
Can you write Python code for it?

all_features = ['bedrooms', 'bedrooms_square',
            'bathrooms',
            'sqft_living', 'sqft_living_sqrt',
            'sqft_lot', 'sqft_lot_sqrt',
            'floors', 'floors_square',
            'waterfront', 'view', 'condition', 'grade',
            'sqft_above',
            'sqft_basement',
            'yr_built', 'yr_renovated']


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Also, copy and paste the predict_output() function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?

def predict_output(feature_matrix, weights):
    # assume feature_matrix is a numpy matrix containing the features as columns and weights is a corresponding numpy array
    # create the predictions vector by using np.dot()
    predictions = np.dot(feature_matrix, weights)
    return(predictions)


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Compute the RSS of each of the three normalized weights on the (unnormalized) test_feature_matrix:
Can you write Python code for it?

prediction =  predict_output(test_feature_matrix, normalized_weights1e4)
rss = 0
for i in range(0, len(prediction)):
    error = prediction[i] - test_data['price'][i]
    rss += error * error
print rss

prediction =  predict_output(test_feature_matrix, normalized_weights1e7)
rss = 0
for i in range(0, len(prediction)):
    error = prediction[i] - test_data['price'][i]
    rss += error * error
print rss

prediction =  predict_output(test_feature_matrix, normalized_weights1e8)
rss = 0
for i in range(0, len(prediction)):
    error = prediction[i] - test_data['price'][i]
    rss += error * error
print rss


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Compute the values of ro[i] for each feature in this simple model, using the formula given above, using the formula:
ro[i] = SUM[ [feature_i]*(output - prediction + w[i]*[feature_i]) ]
Hint: You can get a Numpy vector for feature_i using:
simple_feature_matrix[:,i]
Can you write Python code for it?

ro = [0 for i in range((simple_feature_matrix.shape)[1])]
for j in range((simple_feature_matrix.shape)[1]):
    ro[j] = (simple_feature_matrix[:,j] * (output - prediction + (weights[j] * simple_feature_matrix[:,j]))).sum()
print ro


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Don't forget to normalize features:
Can you write Python code for it?

simple_feature_matrix, norms = normalize_features(simple_feature_matrix)


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
First create a normalized version of the feature matrix, normalized_simple_feature_matrix
Can you write Python code for it?

(simple_feature_matrix, output) = get_numpy_data(sales, simple_features, my_output)
(normalized_simple_feature_matrix, simple_norms) = normalize_features(simple_feature_matrix) # normalize features


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
First, learn the weights with l1_penalty=1e7, on the training data. Initialize weights to all zeros, and set the tolerance=1.  Call resulting weights weights1e7, you will need them later.
Can you write Python code for it?

weights1e7 = lasso_cyclical_coordinate_descent(normalized_all_feature_matrix, output,
                                            initial_weights, l1_penalty=1e7, tolerance=1)
print weights1e7


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
If we want to do any "feature engineering" like creating new features or adjusting existing ones we should do this directly using the SFrames as seen in the first notebook of Week 2. For this notebook, however, we will work with the existing features.
Import useful functions from previous notebook
As in Week 2, we convert the SFrame into a 2D Numpy array. Copy and paste get_num_data() from the second notebook of Week 2.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # this is how you add a constant column to an SFrame
    # add the column 'constant' to the front of the features list so that we can extract it along with the others:
    features = ['constant'] + features # this is how you combine two lists
    # select the columns of data_SFrame given by the features list into the SFrame features_sframe (now including constant):
    features_sframe = data_sframe[features]
    # the following line will convert the features_SFrame into a numpy matrix:
    feature_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the output to the SArray output_sarray
    output_sarray = data_sframe['price']
    # the following will convert the SArray into a numpy array by first converting it to a list
    output_array = output_sarray.to_numpy()
    return(feature_matrix, output_array)


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Let us consider the following set of features.
Can you write Python code for it?

all_features = ['bedrooms',
                'bathrooms',
                'sqft_living',
                'sqft_lot',
                'floors',
                'waterfront',
                'view',
                'condition',
                'grade',
                'sqft_above',
                'sqft_basement',
                'yr_built',
                'yr_renovated']


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Normalize features
In the house dataset, features vary wildly in their relative magnitude: sqft_living is very large overall compared to bedrooms, for instance. As a result, weight for sqft_living would be much smaller than weight for bedrooms. This is problematic because "small" weights are dropped first as l1_penalty goes up.
To give equal considerations for all features, we need to normalize features as discussed in the lectures: we divide each feature by its 2-norm so that the transformed feature has norm 1.
Let's see how we can do this normalization easily with Numpy: let us first consider a small matrix.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Numpy provides a shorthand for computing 2-norms of each column:
Can you write Python code for it?

norms = np.linalg.norm(X, axis=0) # gives [norm(X[:,0]), norm(X[:,1]), norm(X[:,2])]
print norms


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
Recall that, whenever ro[i] falls between -l1_penalty/2 and l1_penalty/2, the corresponding weight w[i] is sent to zero. Now suppose we were to take one step of coordinate descent on either feature 1 or feature 2. What range of values of l1_penalty would not set w[1] zero, but would set w[2] to zero, if we were to take a step in that coordinate?
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
What features had non-zero weight in this case?
Next, learn the weights with l1_penalty=1e8, on the training data. Initialize weights to all zeros, and set the tolerance=1.  Call resulting weights weights1e8, you will need them later.
Can you write Python code for it?

weights1e8 = lasso_cyclical_coordinate_descent(normalized_all_feature_matrix, output,
                                            initial_weights, l1_penalty=1e8, tolerance=1)
print weights1e8


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
What features had non-zero weight in this case?
Rescaling learned weights
Recall that we normalized our feature matrix, before learning the weights.  To use these weights on a test set, we must normalize the test data in the same way.
Alternatively, we can rescale the learned weights to include the normalization, so we never have to worry about normalizing the test data:
In this case, we must scale the resulting weights so that we can make predictions with original features:
 1. Store the norms of the original features to a vector called norms:
features, norms = normalize_features(features)
 2. Run Lasso on the normalized features and obtain a weights vector
 3. Compute the weights for the original features by performing element-wise division, i.e.
weights_normalized = weights / norms
Now, we can apply weights_normalized to the test data, without normalizing it!
Create a normalized version of each of the weights learned above. (weights1e4, weights1e7, weights1e8).
Can you write Python code for it?

# (normalized_simple_feature_matrix, simple_norms) = normalize_features(all_features) # normalize features
normalized_weights1e7 = weights1e7 / simple_norms
print normalized_weights1e7[3]
normalized_weights1e4 = weights1e4 / simple_norms
normalized_weights1e8 = weights1e8 / simple_norms


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTIONS
1. What is the RSS of the learned model on the normalized dataset?
2. Which features had weight zero at convergence?
Evaluating LASSO fit with more features
Let us split the sales dataset into training and test sets.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
To normalize, apply element-wise division:
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
To test the function, run the following:
Can you write Python code for it?

features, norms = normalize_features(np.array([[3.,6.,9.],[4.,8.,12.]]))
print features
# should print
# [[ 0.6  0.6  0.6]
#  [ 0.8  0.8  0.8]]
print norms
# should print
# [5.  10.  15.]


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Use predict_output() to make predictions on this data.
Can you write Python code for it?

prediction = predict_output(simple_feature_matrix, weights)


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Using the following parameters, learn the weights on the sales dataset.
Can you write Python code for it?

simple_features = ['sqft_living', 'bedrooms']
my_output = 'price'
initial_weights = np.zeros(3)
l1_penalty = 1e7
tolerance = 1.0


I am working on the file "Regression/Assignmet_five/.ipynb_checkpoints/week-5-lasso-assignment-2-blank-checkpoint.ipynb".
The first task is:
Using the shorthand we just covered, write a short function called normalize_features(feature_matrix), which normalizes columns of a given feature matrix. The function should return a pair (normalized_features, norms), where the second item contains the norms of original features. As discussed in the lectures, we will use these norms to normalize the test data in the same way as we normalized the training data.
Can you write Python code for it?

import numpy as np
def normalize_features(feature_matrix):
    norms = np.linalg.norm(feature_matrix, axis=0)
    features = feature_matrix / norms
    return features, norms


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
Find what features had non-zero weight.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
Next, we write a loop that does the following:
* For l1_penalty in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, type np.logspace(1, 7, num=13).)
    * Fit a regression model with a given l1_penalty on TRAIN data. Specify l1_penalty=l1_penalty and l2_penalty=0. in the parameter list.
    * Compute the RSS on VALIDATION data (here you will want to use .predict()) for that l1_penalty
* Report which l1_penalty produced the lowest RSS on validation data.
When you call linear_regression.create() make sure you set validation_set = None.
Note: you can turn off the print out of linear_regression.create() with verbose = False
Can you write Python code for it?

import numpy as np

def get_RSS(prediction, output):
    residual = output - prediction
    # square the residuals and add them up
    RS = residual*residual
    RSS = RS.sum()
    return(RSS)

for l1_penalty in np.logspace(1, 7, num=13):
    A = 0
    print l1_penalty
    model_all = graphlab.linear_regression.create(training, target='price', features=all_features,
                                              validation_set=None, verbose = False,
                                              l2_penalty=0, l1_penalty=l1_penalty)
    predictions=model_all.predict(validation)
    A = get_RSS(predictions,validation['price'])
    print A

model_all = graphlab.linear_regression.create(testing, target='price', features=all_features,
                                              validation_set=None, verbose = False,
                                              l2_penalty=0, l1_penalty=10)
model_all['coefficients']['value'].nnz()


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
Now, implement a loop that search through this space of possible l1_penalty values:

For l1_penalty in np.logspace(8, 10, num=20):
Fit a regression model with a given l1_penalty on TRAIN data. Specify l1_penalty=l1_penalty and l2_penalty=0. in the parameter list. When you call linear_regression.create() make sure you set validation_set = None
Extract the weights of the model and count the number of nonzeros. Save the number of nonzeros to a list.
Hint: model['coefficients']['value'] gives you an SArray with the parameters you learned.  If you call the method .nnz() on it, you will find the number of non-zero parameters!
Can you write Python code for it?

for l1_penalty in np.logspace(8, 10, num=20):
    A = 0
    predictions = 0
    print l1_penalty
    model_all = graphlab.linear_regression.create(training, target='price', features=all_features,
                                              validation_set=None, verbose = False,
                                              l2_penalty=0, l1_penalty=l1_penalty)
    predictions=model_all.predict(validation)
    A = get_RSS(predictions,validation['price'])
    print A
    print model_all['coefficients']['value'].nnz()


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
Out of this large range, we want to find the two ends of our desired narrow range of l1_penalty.  At one end, we will have l1_penalty values that have too few non-zeros, and at the other end, we will have an l1_penalty that has too many non-zeros.
More formally, find:
* The largest l1_penalty that has more non-zeros than max_nonzero (if we pick a penalty smaller than this value, we will definitely have too many non-zero weights)
    * Store this value in the variable l1_penalty_min (we will use it later)
* The smallest l1_penalty that has fewer non-zeros than max_nonzero (if we pick a penalty larger than this value, we will definitely have too few non-zero weights)
    * Store this value in the variable l1_penalty_max (we will use it later)
Hint: there are many ways to do this, e.g.:
* Programmatically within the loop above
* Creating a list with the number of non-zeros for each value of l1_penalty and inspecting it to find the appropriate boundaries.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
QUIZ QUESTIONS
1. What value of l1_penalty in our narrow range has the lowest RSS on the VALIDATION set and has sparsity equal to max_nonzeros?
2. What features in this model have non-zero coefficients?
Can you write Python code for it?

model_all = graphlab.linear_regression.create(training, target='price', features=all_features,
                                            validation_set=None, verbose = False,
                                            l2_penalty=0, l1_penalty=3448968612.16)
# model_all['coefficients']['value'].nnz()
# print np.linspace(l1_penalty_min,l1_penalty_max,20)[0]
print model_all['coefficients'].print_rows(num_rows=18, num_columns=3)


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-1-blank.ipynb".
The first task is:
QUIZ QUESTIONS
1. What was the best value for the l1_penalty?
2. What is the RSS on TEST data of the model with the best l1_penalty?
QUIZ QUESTION
Also, using this value of L1 penalty, how many nonzero weights do you have?
Limit the number of nonzero weights
What if we absolutely wanted to limit ourselves to, say, 7 features? This may be important if we want to derive "a rule of thumb" --- an interpretable model that has only a few features in them.
In this section, you are going to implement a simple, two phase procedure to achive this goal:
1. Explore a large range of l1_penalty values to find a narrow region of l1_penalty values where models are likely to have the desired number of non-zero weights.
2. Further explore the narrow region you found to find a good value for l1_penalty that achieves the desired sparsity.  Here, we will again use a validation set to choose the best value for l1_penalty.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Also, copy and paste the predict_output() function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?

def predict_output(feature_matrix, weights):
    # assume feature_matrix is a numpy matrix containing the features as columns and weights is a corresponding numpy array
    # create the predictions vector by using np.dot()
    predictions = np.dot(feature_matrix, weights)
    return(predictions)


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Compute the RSS of each of the three normalized weights on the (unnormalized) test_feature_matrix:
Can you write Python code for it?

prediction =  predict_output(test_feature_matrix, normalized_weights1e4)
rss = 0
for i in range(0, len(prediction)):
    error = prediction[i] - test_data['price'][i]
    rss += error * error
print rss

prediction =  predict_output(test_feature_matrix, normalized_weights1e7)
rss = 0
for i in range(0, len(prediction)):
    error = prediction[i] - test_data['price'][i]
    rss += error * error
print rss

prediction =  predict_output(test_feature_matrix, normalized_weights1e8)
rss = 0
for i in range(0, len(prediction)):
    error = prediction[i] - test_data['price'][i]
    rss += error * error
print rss


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Cyclical coordinate descent
Now that we have a function that optimizes the cost function over a single coordinate, let us implement cyclical coordinate descent where we optimize coordinates 0, 1, ..., (d-1) in order and repeat.
When do we know to stop? Each time we scan all the coordinates (features) once, we measure the change in weight for each coordinate. If no coordinate changes by more than a specified threshold, we stop.
For each iteration:
1. As you loop over features in order and perform coordinate descent, measure how much each coordinate changes.
2. After the loop, if the maximum change across all coordinates is falls below the tolerance, stop. Otherwise, go back to step 1.
Return weights
IMPORTANT: when computing a new weight for coordinate i, make sure to incorporate the new weights for coordinates 0, 1, ..., i-1. One good way is to update your weights variable in-place. See following pseudocode for illustration.
```
for i in range(len(weights)):
    old_weights_i = weights[i] # remember old value of weight[i], as it will be overwritten
    # the following line uses new values for weight[0], weight[1], ..., weight[i-1]
    #     and old values for weight[i], ..., weight[d-1]
    weights[i] = lasso_coordinate_descent_step(i, feature_matrix, output, weights, l1_penalty)
# use old_weights_i to compute change in coordinate
...

```
Can you write Python code for it?

def lasso_cyclical_coordinate_descent(feature_matrix, output, initial_weights, l1_penalty, tolerance):
    weights = initial_weights.copy()
    # converged condition variable
    converged  = False
    while not converged:
        max_change = 0
        for i in range(len(weights)):
            old_weights_i = weights[i]
            weights[i] = lasso_coordinate_descent_step(i, feature_matrix, output, weights, l1_penalty)
            change_i = np.abs(old_weights_i - weights[i])
            if change_i > max_change:
                max_change = change_i
        if max_change < tolerance:
            converged = True
    return weights


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
First create a normalized version of the feature matrix, normalized_simple_feature_matrix
Can you write Python code for it?

(simple_feature_matrix, output) = get_numpy_data(sales, simple_features, my_output)
(normalized_simple_feature_matrix, simple_norms) = normalize_features(simple_feature_matrix) # normalize features


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
First, learn the weights with l1_penalty=1e7, on the training data. Initialize weights to all zeros, and set the tolerance=1.  Call resulting weights weights1e7, you will need them later.
Can you write Python code for it?

weights1e7 = lasso_cyclical_coordinate_descent(normalized_all_feature_matrix, output,
                                            initial_weights, l1_penalty=1e7, tolerance=1)
print weights1e7


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
If we want to do any "feature engineering" like creating new features or adjusting existing ones we should do this directly using the SFrames as seen in the first notebook of Week 2. For this notebook, however, we will work with the existing features.
Import useful functions from previous notebook
As in Week 2, we convert the SFrame into a 2D Numpy array. Copy and paste get_num_data() from the second notebook of Week 2.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # this is how you add a constant column to an SFrame
    # add the column 'constant' to the front of the features list so that we can extract it along with the others:
    features = ['constant'] + features # this is how you combine two lists
    # select the columns of data_SFrame given by the features list into the SFrame features_sframe (now including constant):
    features_sframe = data_sframe[features]
    # the following line will convert the features_SFrame into a numpy matrix:
    feature_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the output to the SArray output_sarray
    output_sarray = data_sframe['price']
    # the following will convert the SArray into a numpy array by first converting it to a list
    output_array = output_sarray.to_numpy()
    return(feature_matrix, output_array)


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Normalize features
In the house dataset, features vary wildly in their relative magnitude: sqft_living is very large overall compared to bedrooms, for instance. As a result, weight for sqft_living would be much smaller than weight for bedrooms. This is problematic because "small" weights are dropped first as l1_penalty goes up.
To give equal considerations for all features, we need to normalize features as discussed in the lectures: we divide each feature by its 2-norm so that the transformed feature has norm 1.
Let's see how we can do this normalization easily with Numpy: let us first consider a small matrix.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Numpy provides a shorthand for computing 2-norms of each column:
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
QUIZ QUESTION
Recall that, whenever ro[i] falls between -l1_penalty/2 and l1_penalty/2, the corresponding weight w[i] is sent to zero. Now suppose we were to take one step of coordinate descent on either feature 1 or feature 2. What range of values of l1_penalty would not set w[1] zero, but would set w[2] to zero, if we were to take a step in that coordinate?
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
QUIZ QUESTION
What features had non-zero weight in this case?
Finally, learn the weights with l1_penalty=1e4, on the training data. Initialize weights to all zeros, and set the tolerance=5e5.  Call resulting weights weights1e4, you will need them later.  (This case will take quite a bit longer to converge than the others above.)
Can you write Python code for it?

weights1e4 = lasso_cyclical_coordinate_descent(normalized_all_feature_matrix, output,
                                            initial_weights, l1_penalty=1e4, tolerance=5e5)
print weights1e4


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
QUIZ QUESTION
What features had non-zero weight in this case?
Rescaling learned weights
Recall that we normalized our feature matrix, before learning the weights.  To use these weights on a test set, we must normalize the test data in the same way.
Alternatively, we can rescale the learned weights to include the normalization, so we never have to worry about normalizing the test data:
In this case, we must scale the resulting weights so that we can make predictions with original features:
 1. Store the norms of the original features to a vector called norms:
features, norms = normalize_features(features)
 2. Run Lasso on the normalized features and obtain a weights vector
 3. Compute the weights for the original features by performing element-wise division, i.e.
weights_normalized = weights / norms
Now, we can apply weights_normalized to the test data, without normalizing it!
Create a normalized version of each of the weights learned above. (weights1e4, weights1e7, weights1e8).
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
QUIZ QUESTION
What range of values of l1_penalty would set both w[1] and w[2] to zero, if we were to take a step in that coordinate?
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
QUIZ QUESTIONS
1. What is the RSS of the learned model on the normalized dataset?
2. Which features had weight zero at convergence?
Evaluating LASSO fit with more features
Let us split the sales dataset into training and test sets.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Then, run your implementation of LASSO coordinate descent:
Can you write Python code for it?

weights = lasso_cyclical_coordinate_descent(normalized_simple_feature_matrix, output,
                                            initial_weights, l1_penalty, tolerance)
print weights

# predictions = predict_output(normalized_simple_feature_matrix, weights)
# rss = 0
# for i in range(0, len(predictions)):
#     error = predictions[i] - sales['price'][i]
#     rss += error * error
# print rss


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
To normalize, apply element-wise division:
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
To test the function, run the following:
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Use predict_output() to make predictions on this data.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
Using the following parameters, learn the weights on the sales dataset.
Can you write Python code for it?


I am working on the file "Regression/Assignmet_five/week-5-lasso-assignment-2-blank.ipynb".
The first task is:
We assign some random set of initial weights and inspect the values of ro[i]:
Can you write Python code for it?


I am working on the file "Regression/Poisson Regression.ipynb".
The first task is:
and now look into this
something is not quite right with my undrstanding
Can you write Python code for it?

df = pd.read_csv('http://stats.idre.ucla.edu/stat/data/poisson_sim.csv', index_col=0)
df['x'] = df['math']
df['y'] = df['num_awards']
df.head()

df.plot(kind='scatter', x='math', y='num_awards')

with Model() as model:
    # specify glm and pass in data. The resulting linear model, its likelihood and
    # and all its parameters are automatically added to our model.
    glm.GLM.from_formula('y ~ x', df)
    trace = sample(3000, cores=2) # draw 3000 posterior samples using NUTS sampling

plt.figure(figsize=(7, 7))
traceplot(trace[100:])
plt.tight_layout();

fig, ax = plt.subplots(figsize=(7, 7))
df.plot(kind='scatter', x='x', y='y', ax=ax)
plot_posterior_predictive_glm(trace, eval=np.linspace(0, 80, 100), samples=100)

with Model() as model:
    # specify glm and pass in data. The resulting linear model, its likelihood and
    # and all its parameters are automatically added to our model.
    glm.GLM.from_formula('y ~ x', df, family=glm.families.NegativeBinomial())
    step = NUTS()
    trace = sample(3000, cores=2, step=step) # draw 3000 posterior samples using NUTS sampling

plt.figure(figsize=(7, 7))
traceplot(trace[100:])
plt.tight_layout();

autocorrplot(trace);

fig, ax = plt.subplots(figsize=(7, 7))
df.plot(kind='scatter', x='x', y='y', ax=ax)
plot_posterior_predictive_glm(trace, eval=np.linspace(0, 80, 100), samples=100)


I am working on the file "Regression/assignments/.ipynb_checkpoints/Ridge Regression Programming Assignment 1-checkpoint.ipynb".
The first task is:
Applying a higher L2 value
Can you write Python code for it?

l2_large_penalty=1.23e2

poly15_data_set1 = polynomial_dataframe(set_1['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model1 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model1.fit(poly15_data_set1, set_1['price'])

poly15_data_set2 = polynomial_dataframe(set_2['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model2 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model2.fit(poly15_data_set2, set_2['price'])

poly15_data_set3 = polynomial_dataframe(set_3['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model3 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model3.fit(poly15_data_set3, set_3['price'])

poly15_data_set4 = polynomial_dataframe(set_4['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model4 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model4.fit(poly15_data_set4, set_4['price'])

plt.plot(poly15_data_set1, model1.predict(poly15_data_set1), poly15_data_set1, set_1['price'])
plt.show()

plt.plot(poly15_data_set2, model2.predict(poly15_data_set2), poly15_data_set2, set_2['price'])
plt.show()

plt.plot(poly15_data_set3, model3.predict(poly15_data_set3), poly15_data_set3, set_3['price'])
plt.show()

plt.plot(poly15_data_set4, model4.predict(poly15_data_set4), poly15_data_set4, set_4['price'])
plt.show()

print('Model 1 coefficients: ', model1.coef_)
print('Model 2 coefficients: ', model2.coef_)
print('Model 3 coefficients: ', model3.coef_)
print('Model 4 coefficients: ', model4.coef_)


I am working on the file "Regression/assignments/.ipynb_checkpoints/Ridge Regression Programming Assignment 1-checkpoint.ipynb".
The first task is:
Ridge regression model fitting
Can you write Python code for it?

poly15_data = polynomial_dataframe(sales['sqft_living'], 15) # use equivalent of `polynomial_sframe`
print(poly15_data)

l2_small_penalty = 1.5e-5
model = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model.fit(poly15_data, sales['price'])

model.coef_

plt.plot(poly15_data, model.predict(poly15_data), poly15_data, sales['price'])
plt.show()


I am working on the file "Regression/assignments/.ipynb_checkpoints/Ridge Regression Programming Assignment 1-checkpoint.ipynb".
The first task is:
Ridge regression on subsets
Using ridge regression with small l2
Can you write Python code for it?

l2_small_penalty=1e-9

poly15_data_set1 = polynomial_dataframe(set_1['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model1 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model1.fit(poly15_data_set1, set_1['price'])

poly15_data_set2 = polynomial_dataframe(set_2['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model2 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model2.fit(poly15_data_set2, set_2['price'])

poly15_data_set3 = polynomial_dataframe(set_3['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model3 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model3.fit(poly15_data_set3, set_3['price'])

poly15_data_set4 = polynomial_dataframe(set_4['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model4 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model4.fit(poly15_data_set4, set_4['price'])

plt.plot(poly15_data_set1, model1.predict(poly15_data_set1), poly15_data_set1, set_1['price'])
plt.show()

plt.plot(poly15_data_set2, model2.predict(poly15_data_set2), poly15_data_set2, set_2['price'])
plt.show()

plt.plot(poly15_data_set3, model3.predict(poly15_data_set3), poly15_data_set3, set_3['price'])
plt.show()

plt.plot(poly15_data_set4, model4.predict(poly15_data_set4), poly15_data_set4, set_4['price'])
plt.show()

print('Model 1 coefficients: ', model1.coef_)
print('Model 2 coefficients: ', model2.coef_)
print('Model 3 coefficients: ', model3.coef_)
print('Model 4 coefficients: ', model4.coef_)


I am working on the file "Regression/assignments/.ipynb_checkpoints/Ridge Regression Programming Assignment 1-checkpoint.ipynb".
The first task is:
Selecting an L2 penalty via cross-validation
Just like the polynomial degree, the L2 penalty is a "magic" parameter we need to select. We could use the validation set approach as we did in the last module, but that approach has a major disadvantage: it leaves fewer observations available for training. Cross-validation seeks to overcome this issue by using all of the training set in a smart way.
We will implement a kind of cross-validation called k-fold cross-validation. The method gets its name because it involves dividing the training set into k segments of roughtly equal size. Similar to the validation set method, we measure the validation error with one of the segments designated as the validation set. The major difference is that we repeat the process k times as follows:
Set aside segment 0 as the validation set, and fit a model on rest of data, and evalutate it on this validation set
Set aside segment 1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set
...
Set aside segment k-1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set
After this process, we compute the average of the k validation errors, and use it as an estimate of the generalization error. Notice that all observations are used for both training and validation, as we iterate over segments of data.
Can you write Python code for it?

def k_fold_cross_validation(k, l2_penalty, data, output):
    n = len(data)
    sumRSS = 0

    for i in range(k):
        # Get the validation/training interval
        start = (n*i)/k
        end = (n*(i+1))/k-1
        #print (i, (ceil(start), ceil(end)))
        train_valid_shuffled[0:ceil(start)].append(train_valid_shuffled[ceil(end)+1:n])

        # Train the model
        model = linear_model.Ridge(alpha=l2_penalty, normalize=True)
        model.fit(data, output)

        # Calculate RSS
        RSS = (abs(output - model.predict(data)) ** 2).sum()

        # Add the RSS to the sum for computing the average
        sumRSS += RSS
    return (sumRSS / k)

print (k_fold_cross_validation(10, 1e-9, poly15_data_set2, set_2['price']))


I am working on the file "Regression/assignments/.ipynb_checkpoints/Ridge Regression Programming Assignment 1-checkpoint.ipynb".
The first task is:
Use the best l2 to train the model on all the data
Can you write Python code for it?

model = linear_model.Ridge(alpha=1000, normalize=True)
model.fit(training[['sqft_living']], training['price'])

print("Residual sum of squares: %.2f"
  % ((model.predict(test[['sqft_living']]) - test['price']) ** 2).sum())