I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now as before, we're labeling each green line as having a distance D, and each red point as having a coordinate of (X,Y). Then we can define our best fit line as the line having the property were:
$$ D_{1}^2 + D_{2}^2 + D_{3}^2 + D_{4}^2 + ....+ D_{N}^2$$
So how do we find this line? The least-square line approximating the set of points:
$$ (X,Y){1},(X,Y){2},(X,Y){3},(X,Y){4},(X,Y)_{5}, $$
has the equation:
$$ Y = a_{0} +a_{1}X $$
this is basically just a rewritten form of the standard equation for a line:
$$Y=mx+b$$
We can solve for these constants a0 and a1 by simultaneously solving these equations:
$$ \Sigma Y = a_{0}N + a_{1}\Sigma X $$
$$ \Sigma XY = a_{0}\Sigma X + a_{1}\Sigma X^2 $$
These are called the normal equations for the least squares line. There are further steps that can be taken in rearranging these equations  to solve for y, but we'll let scikit-learn do the rest of the heavy lifting here. If you want further informatino on the mathematics of the above formulas, check out this great video.
For now, we'll use numpy to do a simple single variable linear regression. Afterwards we'll unleash the power of scikit learn to do a full multivariate linear regression.
Step 4: Using Numpy for a Univariate Linear Regression
Numpy has a built in Least Square Method in its linear algebra library. We'll use this first for our Univariate regression and then move on to scikit learn for out Multi variate regression.
We will start by setting up the X and Y arrays for numpy to take in. An important note for the X array: Numpy expects a two-dimensional array, the first dimension is the different example values, and the second dimension is the attribute number. In this case we have our value as the mean number of rooms per house, and this is a single attribute so the second dimension of the array is just 1. So we'll need to create a (506,1) shape array. There are a few ways to do this, but an easy way to do this is by using numpy's built-in vertical stack tool, vstack.
Can you write Python code for it?

# Set up X as median room values
X = boston_df.RM

# Use v to make X two-dimensional
X = np.vstack(boston_df.RM)

# Set up Y as the target price of the houses.
Y = boston_df.Price


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now import dataset from scikit learn as well as the linear_model module. Note: You may have to run a download, scikit learn will read an error and prompt you to if you don't have the datasets.
Can you write Python code for it?

from sklearn.datasets import load_boston


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now let's see the resultign DataFrame!
Can you write Python code for it?


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now run a prediction on both the X training set and the testing set.
Can you write Python code for it?

# Predictions on training and testing sets
pred_train = lreg.predict(X_train)
pred_test = lreg.predict(X_test)


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now that we have our X and Y, let's go ahead and use numpy to create the single variable linear regression.
We know that a line has the equation:
$$y=mx+b$$
which we can rewrite using matrices:
$$y=Ap$$
where:
$$A = \begin{bmatrix}x & 1\end{bmatrix}$$
and
$$p= \begin{bmatrix}m \b\end{bmatrix}$$
This is the same as the first equation if you carry out the linear algebra.
So we'll start by creating the A matrix using numpy. We'll do this by creating a matrix in the form [X 1], so we'll call every value in our original X using a list comprehension and then set up an array in the form [X 1]
Can you write Python code for it?

# Create the X array in the form [X 1]
X = np.array( [ [value,1] for value in X ] )


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now we will get the mean square error
Can you write Python code for it?

print("Fit a model X_train, and calculate MSE with Y_train: %.2f"  % np.mean((Y_train - pred_train) ** 2))

print("Fit a model X_train, and calculate MSE with X_test and Y_test: %.2f"  %np.mean((Y_test - pred_test) ** 2))


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Now, you might be reminded of the seaborn lmplot function we used during the visualization lectures. You could use it here to do a linear fit automatically!
Can you write Python code for it?

# Using seabron to create a linear fit
sns.lmplot('RM','Price',data = boston_df)


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Since the root mean square error (RMSE) corresponds approximately to the standard deviation we can now say that the price of a house won't vary more than 2 times the RMSE 95% of the time. Note: Review the Normal Distribution Appendix lecture if this doesn't make sense to you or check out this link.
Thus we can reasonably expect a house price to be within $13,200 of our line fit.
Step 6: Using scikit learn to implement a multivariate regression
Now, we'll keep moving along with using scikit learn to do a multi variable regression. This will be a similar apporach to the above example, but sci kit learn will be able to take into account more than just a single data variable effecting the target!
We'll start by importing the linear regression library from the sklearn module.
The sklearn.linear_model.LinearRegression class is an estimator. Estimators predict a value based on the observed data. In scikit-learn, all estimators implement the fit() and predict() methods. The former method is used to learn the parameters of a model, and the latter method is used to predict the value of a response variable for an explanatory variable using the learned parameters. It is easy to experiment with different models using scikit-learn because all estimators implement the fit and predict methods.
Can you write Python code for it?

# Import for Linear Regression
import sklearn
from sklearn.linear_model import LinearRegression


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Step 5: Getting the error
Great! We've just completed a single variable regression using the least squares method with Python! Let's see if we can find the error in our fitted line. Checking out the documentation here, we see that the resulting array has the total squared error. For each element, it checks the the difference between the line and the true value (our original D value), squares it, and returns the sum of all these. This was the summed D^2 value we discussed earlier.
It's probably easier to understand the root mean squared error, which is similar to the standard deviation. In this case, to find the root mean square error we divide by the number of elements and then take the square root. There is also an issue of bias and an unbiased regression, but we'll delve into those topics later.
For now let's see how we can get the root mean squared error of the line we just fitted.
Can you write Python code for it?

# Get the resulting array
result = np.linalg.lstsq(X,Y)

# Get the total error
error_total = result[1]

# Get the root mean square error
rmse = np.sqrt(error_total/len(X) )

# Print
print("The root mean squared error was %.2f " %rmse)


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
The functions we will be using are:
lreg.fit() which fits a linear model
lreg.predict() which is used to predict Y using the linear model with estimated coefficients
lreg.score() which returns the coefficient of determination (R^2). A measure of how well observed outcomes are replicated by the model, learn more about it here
We'll start the multi variable regression analysis by seperating our boston dataframe into the data columns and the target columns:
Can you write Python code for it?

# Data Columns
X_multi = boston_df.drop('Price',1)

# Targets
Y_target = boston_df.Price


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
Create Training and Testing Sets
In order to see how well our classifier works, we need to divide our data into training and testing sets
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
Evaluate the Toy Model
Next, we evaluate how well our model worked on the training dataset. Unlike with discrete classifiers (e.g. KNN, SVM), the number of examples it got "correct" isn't meaningful here. We certainly care if the predicted value is off by 100, but do we care as much if it is off by 1? What about 0.01?
There are many ways to evaluate a linear classifier, but one popular way is the mean-squared error, or MSE. As the name implies, you find the error for each example (the distance between the point and the predicted line), square each of them, and then add them all together.
Scikit-learn has a function that does this for you easily.
Can you write Python code for it?

mse = mean_squared_error(y_test, predictions)

print('The MSE is ' + '%.2f' % mse)


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
Making Predictions
To make predictions on new data we use the predict() method. It expects a single input: an array-like object containing the features for the examples we want to predict.
Here, we get the predictions for two body mass index values: -.05 and 0.1.
Can you write Python code for it?

bmi = [[-0.05],[0.1]]
predictions = regr.predict(bmi)
print(predictions)


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
Sidenote: Randomness and Results
Every time you run this notebook, you will get slightly different results. Why? Because data is randomly divided among the training/testing/validation data sets. Running the code again will create a different division of the data, and will make the results slightly different. However, the overall outcome should remain consistent and have approximately the same values. If you have drastically different results when running an analysis multiple times, it suggests a problem with your model or that you need more data.
If it's important that you get the exact same results every time you run the code, you can specify a random state in the random_state argument of train_test_split() and KFold.
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
The MSE isn't as intuitive as the accuracy of a discrete classifier, but it is highly useful for comparing the effectiveness of different models. Another option is to look at the $R^2$ score, which you may already be familiar with if you've ever fit a line to data in Excel. A value of 1.0 is a perfect predictor, while 0.0 means there is no correlation between the input and output of the regression model.
Can you write Python code for it?

r2score = r2_score(y_test, predictions)

print('The R^2 score is ' + '%.2f' % r2score)


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
Train A Toy Model
Next, we train a linear regression algorithm on our data.
In scikit learn, we use linear_model.LinearRegression() to create our model. Models are trained using a fit() method. For linear regression, fit() expects two arguments: the training examples X_train and the corresponding labels y_train.
To start, we will train a toy model using only the body mass index feature.
Can you write Python code for it?

regr = linear_model.LinearRegression()
x_train = X_train[:,bmi_ind].reshape(-1, 1) # regression expects a (#examples,#features) array shape
regr.fit(x_train, y_train)


I am working on the file "LinearRegression_Tutorial.ipynb".
The first task is:
Visualize The Data
There are too many features to visualize the whole training set, but we can plot a single feature (e.g. body mass index) against the quantified disease progression.
Remember that the features have been normalized to center it around 0 and scaled by the standard deviation. So the values shown aren't the actual BMI levels, but a standardized version of them.
Can you write Python code for it?

bmi_ind = 2

plt.scatter(X_train[:,bmi_ind], y_train)
plt.ylabel('Disease Progression')
plt.xlabel('Body Mass Index')
plt.show()


I am working on the file "Linear_Algebra.ipynb".
The first task is:
(3) Feature reduction

We manually creat a matrix $\mathbf{A}$ such that all but its last column is heavily correlated.
Can you write Python code for it?

a = np.random.rand(8)
b = np.random.rand(8)
sigma = 0.01
At = np.array( [a + sigma*np.random.randn(8) for _ in range(4)])
At = np.vstack((At, b))
A = At.T
print(a)
print(b)
print()
print(A)

def approximate_one(A):
    U, s, Vt = linalg.svd(A, full_matrices=False)
    print('singular values are: \n', s)
    S = np.diag(s)
    V = Vt.T
    # only 1 summand
    U1 = U[:, 0]
    s1 = s[0]
    Vt1 = Vt[0, :]
    A1 = s1 * np.outer(U1, Vt1)
    return A1

A1 = approximate_one(A)
A1

print(A1 - A)

[ np.sum( (A1-A)[:, i] ) for i in range(A.shape[1]) ]


I am working on the file "Linear_Algebra.ipynb".
The first task is:
Impact to basis vectors
Can you write Python code for it?

U = np.mat("1 0; 0 1")
V = M @ U
V

U2 = np.hstack([np.array([i-1, j-1]).reshape(2,1) for i in range(3) for j in range(3)])
U2

V2 = M @ U2
V2

plt.plot(U2[0,:], U2[1,:], 'bx')
plt.title('original space')
plt.draw()
plt.show()

plt.plot(V2[0,:], V2[1,:], 'r+')
plt.title('new space')
plt.draw()
plt.show()

plt.plot(U2[0,:], U2[1,:], 'bx')
plt.plot(V2[0,:], V2[1,:], 'r+')
plt.title('plot together')
plt.draw()
plt.show()


I am working on the file "Linear_Algebra.ipynb".
The first task is:
We only do a reduced SVD: $$\mathbf{A} = \mathbf{U} \mathbf{S} \mathbf{V}^{\top}, $$
so that $\mathbf{U}$ is of the same shape as $\mathbf{A}$.
Can you write Python code for it?

U, s, Vt = linalg.svd(A, full_matrices=False)
S = np.diag(s)
V = Vt.T

U

s

S

Vt

V


I am working on the file "Linear_Algebra.ipynb".
The first task is:
With some more efforts, one can draw the original coordinate systems and the transformed one in a more visual way. This will hopefully strengthen the idea that a matrix multiplication represents a linear transformation.
SVD
Can you write Python code for it?


I am working on the file "Linear_Regression.ipynb".
The first task is:
Create train & test sets.
Can you write Python code for it?

train_X = data[:400, :-1]
train_y = data[:400, -1]

test_X = data[400:, :-1]
test_y = data[400:, -1]

m, n = train_X.shape


I am working on the file "Linear_Regression.ipynb".
The first task is:
Define the cost function and how to compute the gradient.<br>
Both are needed for the subsequent optimization procedure.
Can you write Python code for it?

def cost_function(theta, X, y):
    squared_errors = (X.dot(theta) - y) ** 2
    J = 0.5 * squared_errors.sum()
    return J

def gradient(theta, X, y):
    errors = X.dot(theta) - y
    return errors.dot(X)


I am working on the file "Linear_Regression.ipynb".
The first task is:
Finally, let's have a more intuitive look at the predictions.
Can you write Python code for it?

plt.figure(figsize=(10, 8))
plt.scatter(np.arange(test_y.size), sorted(test_y), c='b', edgecolor='None', alpha=0.5, label='actual')
plt.scatter(np.arange(test_y.size), sorted(test_X.dot(optimal_theta)), c='g', edgecolor='None', alpha=0.5, label='predicted')
plt.legend(loc='upper left')
plt.ylabel('House price ($1000s)')
plt.xlabel('House #')


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
As we see we get the same answer both ways
Can you write Python code for it?

# if we want to multiply every price by 0.5 it's a simple as:
half_prices = 0.5*prices
# Let's compute the sum of squares of price. We can multiply two SArrays of the same length elementwise also with *
prices_squared = prices*prices
sum_prices_squared = prices_squared.sum() # price_squared is an SArray of the squares and we want to add them up.
print "the sum of price squared is: " + str(sum_prices_squared)


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Load house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
New Model: estimate prices from bedrooms
We have made one model for predicting house prices using squarefeet, but there are many other features in the sales SFrame.
Use your simple linear regression function to estimate the regression parameters from predicting Prices based on number of bedrooms. Use the training data!
Can you write Python code for it?

# Estimate the slope and intercept for predicting 'price' based on 'bedrooms'
br_intercept, br_slope = simple_linear_regression(train_data['bedrooms'], train_data['price'])


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Now that we can calculate a prediction given the slope and intercept let's make a prediction. Use (or alter) the following to find out the estimated price for a house with 2650 squarefeet according to the squarefeet model we estiamted above.
Quiz Question: Using your Slope and Intercept from (4), What is the predicted price for a house with 2650 sqft?
Can you write Python code for it?

my_house_sqft = 2650
estimated_price = get_regression_predictions(my_house_sqft, sqft_intercept, sqft_slope)
print "The estimated price for a house with %d squarefeet is $%.2f" % (my_house_sqft, estimated_price)


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Now that we know it works let's build a regression model for predicting price based on sqft_living. Rembember that we train on train_data!
Can you write Python code for it?

sqft_intercept, sqft_slope = simple_linear_regression(train_data['sqft_living'], train_data['price'])

print "Intercept: " + str(sqft_intercept)
print "Slope: " + str(sqft_slope)


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Predict the squarefeet given price
What if we want to predict the squarefoot given the price? Since we have an equation y = a + b*x we can solve the function for x. So that if we have the intercept (a) and the slope (b) and the price (y) we can solve for the estimated squarefeet (x).
Comlplete the following function to compute the inverse regression estimate, i.e. predict the input_feature given the output!
Can you write Python code for it?

def inverse_regression_predictions(output, intercept, slope):
    # solve output = intercept + slope*input_feature for input_feature. Use this equation to compute the inverse predictions:
    estimated_feature = (output - intercept) * 1.0 / slope
    return estimated_feature


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Residual Sum of Squares
Now that we have a model and can make predictions let's evaluate our model using Residual Sum of Squares (RSS). Recall that RSS is the sum of the squares of the residuals and the residuals is just a fancy word for the difference between the predicted output and the true output.
Complete the following (or write your own) function to compute the RSS of a simple linear regression model given the input_feature, output, intercept and slope:
Can you write Python code for it?

def get_residual_sum_of_squares(input_feature, output, intercept, slope):
    # First get the predictions
    predictions = get_regression_predictions(input_feature, intercept, slope)

    # then compute the residuals (since we are squaring it doesn't matter which order you subtract)
    residuals = output - predictions

    # square the residuals and add them up
    RSS = (residuals * residuals).sum()

    return(RSS)


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Split data into training and testing
We use seed=0 so that everyone running this notebook gets the same results.  In practice, you may set a random seed (or let GraphLab Create pick a random seed for you).
Can you write Python code for it?


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
Test your Linear Regression Algorithm
Now we have two models for predicting the price of a house. How do we know which one is better? Calculate the RSS on the TEST data (remember this data wasn't involved in learning the model). Compute the RSS from predicting prices using bedrooms and from predicting prices using squarefeet.
Quiz Question: Which model (square feet or bedrooms) has lowest RSS on TEST data? Think about why this might be the case.
Can you write Python code for it?

# Compute RSS when using bedrooms on TEST data:
get_residual_sum_of_squares(test_data['bedrooms'], test_data['price'], br_intercept, br_slope)

# Compute RSS when using squarefeet on TEST data:
get_residual_sum_of_squares(test_data['sqft_living'], test_data['price'], sqft_intercept, sqft_slope)


I am working on the file "Linear_Regression_1_simple_regression.ipynb".
The first task is:
We can test that our function works by passing it something where we know the answer. In particular we can generate a feature and then put the output exactly on a line: output = 1 + 1*input_feature then we know both our slope and intercept should be 1
Can you write Python code for it?

test_feature = graphlab.SArray(range(5))
test_output = graphlab.SArray(1 + 1*test_feature)
(test_intercept, test_slope) =  simple_linear_regression(test_feature, test_output)
print "Intercept: " + str(test_intercept)
print "Slope: " + str(test_slope)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Create some new features
Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms.
You will use the logarithm function to create a new feature. so first you should import it from the math library.
Can you write Python code for it?


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Learning Multiple Models
Now we will learn the weights for three (nested) models for predicting house prices. The first model will have the fewest features the second model will add one more feature and the third will add a few more:
* Model 1: squarefeet, # bedrooms, # bathrooms, latitude & longitude
* Model 2: add bedrooms*bathrooms
* Model 3: Add log squarefeet, bedrooms squared, and the (nonsensical) latitude + longitude
Can you write Python code for it?

model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long']
model_2_features = model_1_features + ['bed_bath_rooms']
model_3_features = model_2_features + ['bedrooms_squared', 'log_sqft_living', 'lat_plus_long']


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Learning a multiple regression model
Recall we can use the following code to learn a multiple regression model predicting 'price' based on the following features:
example_features = ['sqft_living', 'bedrooms', 'bathrooms'] on training data with the following code:
(Aside: We set validation_set = None to ensure that the results are always the same)
Can you write Python code for it?

example_features = ['sqft_living', 'bedrooms', 'bathrooms']
example_model = graphlab.linear_regression.create(train_data, target = 'price', features = example_features,
                                                  validation_set = None)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Making Predictions
In the gradient descent notebook we use numpy to do our regression. In this book we will use existing graphlab create functions to analyze multiple regressions.
Recall that once a model is built we can use the .predict() function to find the predicted values for data we pass. For example using the example model above:
Can you write Python code for it?

example_predictions = example_model.predict(train_data)
print example_predictions[0] # should be 271789.505878


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Now that we have fitted the model we can extract the regression weights (coefficients) as an SFrame as follows:
Can you write Python code for it?

example_weight_summary = example_model.get("coefficients")
print example_weight_summary


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Now that you have the features, learn the weights for the three different models for predicting target = 'price' using graphlab.linear_regression.create() and look at the value of the weights/coefficients:
Can you write Python code for it?

# Learn the three models: (don't forget to set validation_set = None)
model_1 = graphlab.linear_regression.create(train_data, target='price', features=model_1_features, validation_set=None)
model_2 = graphlab.linear_regression.create(train_data, target='price', features=model_2_features, validation_set=None)
model_3 = graphlab.linear_regression.create(train_data, target='price', features=model_3_features, validation_set=None)

# Examine/extract each model's coefficients:
print model_1.get('coefficients')
print model_2.get('coefficients')


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 1?
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 2?
Think about what this means.
Comparing multiple models
Now that you've learned three models and extracted the model weights we want to evaluate which model is best.
First use your functions from earlier to compute the RSS on TRAINING Data for each of the three models.
Can you write Python code for it?

# Compute the RSS on TRAINING data for each of the three models and record the values:
print get_residual_sum_of_squares(model_1, train_data, train_data['price'])
print get_residual_sum_of_squares(model_2, train_data, train_data['price'])
print get_residual_sum_of_squares(model_3, train_data, train_data['price'])


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Quiz Question: Which model (1, 2 or 3) has lowest RSS on TRAINING Data? Is this what you expected?
Now compute the RSS on on TEST data for each of the three models.
Can you write Python code for it?

# Compute the RSS on TESTING data for each of the three models and record the values:
print get_residual_sum_of_squares(model_1, test_data, test_data['price'])
print get_residual_sum_of_squares(model_2, test_data, test_data['price'])
print get_residual_sum_of_squares(model_3, test_data, test_data['price'])


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.
bedrooms times bathrooms gives what's called an "interaction" feature. It is large when both of them are large.
Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.
Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why)

Quiz Question: What is the mean (arithmetic average) value of your 4 new features on TEST data? (round to 2 digits)
Can you write Python code for it?

test_data['bedrooms_squared'].mean()

test_data['bed_bath_rooms'].mean()

test_data['log_sqft_living'].mean()

test_data['lat_plus_long'].mean()


I am working on the file "Linear_Regression_2_multiple_regression_assignment_1.ipynb".
The first task is:
Test your function by computing the RSS on TEST data for the example model:
Can you write Python code for it?

rss_example_train = get_residual_sum_of_squares(example_model, test_data, test_data['price'])
print rss_example_train # should be 2.7376153833e+14


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
A few things to note before we run the gradient descent. Since the gradient is a sum over all the data points and involves a product of an error and a feature the gradient itself will be very large since the features are large (squarefeet) and the output is large (prices). So while you might expect "tolerance" to be small, small is only relative to the size of the features.
For similar reasons the step size will be much smaller than you might expect but this is because the gradient has such large values.
Running the Gradient Descent as Simple Regression
First let's split the data into training and test data.
Can you write Python code for it?


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Although the gradient descent is designed for multiple regression since the constant is now a feature we can use the gradient descent function to estimat the parameters in the simple regression on squarefeet. The folowing cell sets up the feature_matrix, output, initial weights and step size for the first model:
Can you write Python code for it?

# let's test out the gradient descent
simple_features = ['sqft_living']
my_output = 'price'
(simple_feature_matrix, output) = get_numpy_data(train_data, simple_features, my_output)
initial_weights = np.array([-47000., 1.])
step_size = 7e-12
tolerance = 2.5e7


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Computing the Derivative
We are now going to move to computing the derivative of the regression cost function. Recall that the cost function is the sum over the data points of the squared difference between an observed output and a predicted output.
Since the derivative of a sum is the sum of the derivatives we can compute the derivative for a single data point and then sum over data points. We can write the squared difference between the observed output and predicted output for a single point as follows:
(w[0]*[CONSTANT] + w[1]*[feature_1] + ... + w[i] *[feature_i] + ... +  w[k]*[feature_k] - output)^2
Where we have k features and a constant. So the derivative with respect to weight w[i] by the chain rule is:
2*(w[0]*[CONSTANT] + w[1]*[feature_1] + ... + w[i] *[feature_i] + ... +  w[k]*[feature_k] - output)* [feature_i]
The term inside the paranethesis is just the error (difference between prediction and output). So we can re-write this as:
2*error*[feature_i]
That is, the derivative for the weight for feature i is the sum (over data points) of 2 times the product of the error and the feature itself. In the case of the constant then this is just twice the sum of the errors!
Recall that twice the sum of the product of two vectors is just twice the dot product of the two vectors. Therefore the derivative for the weight for feature_i is just two times the dot product between the values of feature_i and the current errors.
With this in mind complete the following derivative function which computes the derivative of the weight given the value of the feature (over all data points) and the errors (over all data points).
Can you write Python code for it?

def feature_derivative(errors, feature):
    # Assume that errors and feature are both numpy arrays of the same length (number of data points)
    # compute twice the dot product of these vectors as 'derivative' and return the value
    derivative = 2 * np.dot(errors, feature)
    return(derivative)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
For testing let's use the 'sqft_living' feature and a constant as our features and price as our output:
Can you write Python code for it?

(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price') # the [] around 'sqft_living' makes it a list
print example_features[0,:] # this accesses the first row of the data the ':' indicates 'all columns'
print example_output[0] # and the corresponding output


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Gradient Descent
Now we will write a function that performs a gradient descent. The basic premise is simple. Given a starting point we update the current weights by moving in the negative gradient direction. Recall that the gradient is the direction of increase and therefore the negative gradient is the direction of decrease and we're trying to minimize a cost function.
The amount by which we move in the negative gradient direction  is called the 'step size'. We stop when we are 'sufficiently close' to the optimum. We define this by requiring that the magnitude (length) of the gradient vector to be smaller than a fixed 'tolerance'.
With this in mind, complete the following gradient descent function below using your derivative function above. For each step in the gradient descent we update the weight for each feature befofe computing our stopping criteria
Can you write Python code for it?

from math import sqrt # recall that the magnitude/length of a vector [g[0], g[1], g[2]] is sqrt(g[0]^2 + g[1]^2 + g[2]^2)

def regression_gradient_descent(feature_matrix, output, initial_weights, step_size, tolerance):
    converged = False
    weights = np.array(initial_weights) # make sure it's a numpy array
    while not converged:
        # compute the predictions based on feature_matrix and weights using your predict_output() function
        predictions = predict_output(feature_matrix, weights)
        # compute the errors as predictions - output
        errors = predictions - output

        gradient_sum_squares = 0 # initialize the gradient sum of squares
        # while we haven't reached the tolerance yet, update each feature's weight
        for i in range(len(weights)): # loop over each weight
            # Recall that feature_matrix[:, i] is the feature column associated with weights[i]
            # compute the derivative for weight[i]:
            derivative = feature_derivative(errors, feature_matrix[:, i])
            # add the squared value of the derivative to the gradient sum of squares (for assessing convergence)
            gradient_sum_squares = gradient_sum_squares + derivative * derivative
            # subtract the step size times the derivative from the current weight
            weights[i] = weights[i] - derivative * step_size

        # compute the square-root of the gradient sum of squares to get the gradient matnigude:
        gradient_magnitude = sqrt(gradient_sum_squares)
        if gradient_magnitude < tolerance:
            converged = True
    return(weights)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
If we want to do any "feature engineering" like creating new features or adjusting existing ones we should do this directly using the SFrames as seen in the other Week 2 notebook. For this notebook, however, we will work with the existing features.
Convert to Numpy Array
Although SFrames offer a number of benefits to users (especially when using Big Data and built-in graphlab functions) in order to understand the details of the implementation of algorithms it's important to work with a library that allows for direct (and optimized) matrix operations. Numpy is a Python solution to work with matrices (or any multi-dimensional "array").
Recall that the predicted value given the weights and the features is just the dot product between the feature and weight vector. Similarly, if we put all of the features row-by-row in a matrix then the predicted value for all the observations can be computed by right multiplying the "feature matrix" by the "weight vector".
First we need to take the SFrame of our data and convert it into a 2D numpy array (also called a matrix). To do this we use graphlab's built in .to_dataframe() which converts the SFrame into a Pandas (another python library) dataframe. We can then use Panda's .as_matrix() to convert the dataframe into a numpy matrix.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Next run your gradient descent with the above parameters.
Can you write Python code for it?

simple_weights = regression_gradient_descent(simple_feature_matrix, output, initial_weights, step_size, tolerance)

print simple_weights


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Now compute your predictions using test_simple_feature_matrix and your weights from above.
Can you write Python code for it?

predictions = predict_output(test_simple_feature_matrix, weights)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Now that you have the predictions on test data, compute the RSS on the test data set. Save this value for comparison later. Recall that RSS is the sum of the squared errors (difference between prediction and output).
Can you write Python code for it?

errors = predictions - test_output
RSS = np.dot(errors, errors)
print RSS


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Now we will write a function that will accept an SFrame, a list of feature names (e.g. ['sqft_living', 'bedrooms']) and an target feature e.g. ('price') and will return two things:
* A numpy matrix whose columns are the desired features plus a constant column (this is how we create an 'intercept')
* A numpy array containing the values of the output
With this in mind, complete the following function (where there's an empty line you should write a line of code that does what the comment above indicates)
Please note you will need GraphLab Create version at least 1.7.1 in order for .to_numpy() to work!
Can you write Python code for it?

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # this is how you add a constant column to an SFrame
    # add the column 'constant' to the front of the features list so that we can extract it along with the others:
    features = ['constant'] + features # this is how you combine two lists
    # select the columns of data_SFrame given by the features list into the SFrame features_sframe (now including constant):
    features_sframe = data_sframe[features]

    # the following line will convert the features_SFrame into a numpy matrix:
    feature_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the output to the SArray output_sarray
    output_sarray = data_sframe[output]

    # the following will convert the SArray into a numpy array by first converting it to a list
    output_array = output_sarray.to_numpy()
    return(feature_matrix, output_array)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Predicting output given regression weights
Suppose we had the weights [1.0, 1.0] and the features [1.0, 1180.0] and we wanted to compute the predicted output 1.0*1.0 + 1.0*1180.0 = 1181.0 this is the dot product between these two arrays. If they're numpy arrayws we can use np.dot() to compute this:
Can you write Python code for it?

my_weights = np.array([1., 1.]) # the example weights
my_features = example_features[0,] # we'll use the first data point
predicted_value = np.dot(my_features, my_weights)
print predicted_value


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Quiz Question: What is the predicted price for the 1st house in the TEST data set for model 1 (round to nearest dollar)?
Can you write Python code for it?


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Quiz Question: What is the predicted price for the 1st house in the TEST data set for model 2 (round to nearest dollar)?
Can you write Python code for it?


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Quiz Question: Which model (1 or 2) has lowest RSS on all of the TEST data?
Can you write Python code for it?


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Running a multiple regression
Now we will use more than one actual feature. Use the following code to produce the weights for a second model with the following parameters:
Can you write Python code for it?

model_features = ['sqft_living', 'sqft_living15'] # sqft_living15 is the average squarefeet for the nearest 15 neighbors.
my_output = 'price'
(feature_matrix, output) = get_numpy_data(train_data, model_features, my_output)
initial_weights = np.array([-100000., 1., 1.])
step_size = 4e-12
tolerance = 1e9


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
Use your newly estimated weights and the predict_output function to compute the predictions on the TEST data. Don't forget to create a numpy array for these features from the test set first!
Can you write Python code for it?

(test_feature_matrix, test_output) = get_numpy_data(test_data, model_features, my_output)
predictions_test = predict_output(test_feature_matrix, weights2)


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
What is the actual price for the 1st house in the test data set?
Can you write Python code for it?


I am working on the file "Linear_Regression_2_multiple_regression_assignment_2.ipynb".
The first task is:
np.dot() also works when dealing with a matrix and a vector. Recall that the predictions from all the observations is just the RIGHT (as in weights on the right) dot product between the features matrix and the weights vector. With this in mind finish the following predict_output function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?

def predict_output(feature_matrix, weights):
    # assume feature_matrix is a numpy matrix containing the features as columns and weights is a corresponding numpy array
    # create the predictions vector by using np.dot()
    predictions = np.dot(feature_matrix, weights)
    return(predictions)


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
As in Week 3, we will use the sqft_living variable. For plotting purposes (connecting the dots), you'll need to sort by the values of sqft_living. For houses with identical square footage, we break the tie by their prices.
Can you write Python code for it?

sales = sales.sort(['sqft_living', 'price'])


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Changing the data and re-learning
We're going to split the sales data into four subsets of roughly equal size. Then you will estimate a 15th degree polynomial model on all four subsets of the data. Print the coefficients (you should use .print_rows(num_rows = 16) to view all of them) and plot the resulting fit (as we did above). The quiz will ask you some questions about these results.
To split the sales data into four subsets, we perform the following steps:
* First split sales into 2 subsets with .random_split(0.5, seed=0).
* Next split the resulting subsets into 2 more subsets each. Use .random_split(0.5, seed=0).
We set seed=0 in these steps so that different users get consistent results.
You should end up with 4 subsets (set_1, set_2, set_3, set_4) of approximately equal size.
Can you write Python code for it?

sub1, sub2 = sales.random_split(0.5, seed=0)
set_1, set_2 = sub1.random_split(0.5, seed=0)
set_3, set_4 = sub2.random_split(0.5, seed=0)


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Fit a 15th degree polynomial on set_1, set_2, set_3, and set_4 using sqft_living to predict prices. Print the coefficients and make a plot of the resulting model.
Can you write Python code for it?

fitAndPlot(set_1, 15)

fitAndPlot(set_2, 15)

fitAndPlot(set_3, 15)

fitAndPlot(set_4, 15)


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Let's start with a degree 1 polynomial using 'sqft_living' (i.e. a line) to predict 'price' and plot what it looks like.
Can you write Python code for it?

poly1_data = polynomial_sframe(sales['sqft_living'], 1)
poly1_data['price'] = sales['price'] # add price to the data since it's the target


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
NOTE: for all the models in this notebook use validation_set = None to ensure that all results are consistent across users.
Can you write Python code for it?

model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = ['power_1'], validation_set = None)

#let's take a look at the weights before we plot
model1.get("coefficients")

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(poly1_data['power_1'],poly1_data['price'],'.',
        poly1_data['power_1'], model1.predict(poly1_data),'-')


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Next we're going to write a polynomial function that takes an SArray and a maximal degree and returns an SFrame with columns containing the SArray to all the powers up to the maximal degree.
The easiest way to apply a power to an SArray is to use the .apply() and lambda x: functions.
For example to take the example array and compute the third power we can do as follows: (note running this cell the first time may take longer than expected since it loads graphlab)
Can you write Python code for it?

tmp = graphlab.SArray([1., 2., 3.])
tmp_cubed = tmp.apply(lambda x: x**3)
print tmp
print tmp_cubed
print tmp*tmp


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Next you should write a loop that does the following:
* For degree in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] (to get this in python type range(1, 15+1))
    * Build an SFrame of polynomial data of train_data['sqft_living'] at the current degree
    * hint: my_features = poly_data.column_names() gives you a list e.g. ['power_1', 'power_2', 'power_3'] which you might find useful for graphlab.linear_regression.create( features = my_features)
    * Add train_data['price'] to the polynomial SFrame
    * Learn a polynomial regression model to sqft vs price with that degree on TRAIN data
    * Compute the RSS on VALIDATION data (here you will want to use .predict()) for that degree and you will need to make a polynmial SFrame using validation data.
* Report which degree had the lowest RSS on validation data (remember python indexes from 0)
(Note you can turn off the print out of linear_regression.create() with verbose = False)
Can you write Python code for it?

def fitAndValidate(training, validation, degree):
    poly_train = polynomial_sframe(training['sqft_living'], degree)
    my_features = poly_train.column_names() # get the name of the features
    poly_train['price'] = training['price'] # add price to the data since it's the target
    model = graphlab.linear_regression.create(poly_train, target = 'price', verbose=False,
                                              features = my_features, validation_set = None)
    poly_validation = polynomial_sframe(validation['sqft_living'], degree)
    predictions = model.predict(poly_validation)
    errors = predictions - validation['price']
    RSS = (errors * errors).sum()
    return model, RSS

lowest_RSS_degree = 0
RSS = float("inf")
for degree in range(1, 16):
    model, modelRSS = fitAndValidate(training, validation, degree)
    if modelRSS < RSS:
        lowest_RSS_degree = degree
        RSS = modelRSS
print lowest_RSS_degree


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Now try a 15th degree polynomial:
Can you write Python code for it?

poly15_data = polynomial_sframe(sales['sqft_living'], 15)
my_features = poly15_data.column_names() # get the name of the features
poly15_data['price'] = sales['price'] # add price to the data since it's the target
model15 = graphlab.linear_regression.create(poly15_data, target = 'price', features = my_features, validation_set = None)

plt.plot(poly15_data['power_1'],poly15_data['price'],'.',
        poly15_data['power_1'], model15.predict(poly15_data),'-')

def fitAndPlot(data, degree):
    poly_data = polynomial_sframe(data['sqft_living'], degree)
    my_features = poly_data.column_names() # get the name of the features
    poly_data['price'] = data['price'] # add price to the data since it's the target
    model = graphlab.linear_regression.create(poly_data, target = 'price', features = my_features, validation_set = None)
    plt.plot(poly_data['power_1'],poly_data['price'],'.',
        poly_data['power_1'], model.predict(poly_data),'-')
    model.get("coefficients").print_rows(num_rows = 16)


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Quiz Question: Which degree (1, 2, …, 15) had the lowest RSS on Validation data?
Now that you have chosen the degree of your polynomial using validation data, compute the RSS of this model on TEST data. Report the RSS on your quiz.
Can you write Python code for it?

model, RSS = fitAndValidate(training, validation, 6)


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
Some questions you will be asked on your quiz:
Quiz Question: Is the sign (positive or negative) for power_15 the same in all four models?
Quiz Question: (True/False) the plotted fitted lines look the same in all four plots
Selecting a Polynomial Degree
Whenever we have a "magic" parameter like the degree of the polynomial there is one well-known way to select these parameters: validation set. (We will explore another approach in week 4).
We split the sales dataset 3-way into training set, test set, and validation set as follows:

Split our sales data into 2 sets: training_and_validation and testing. Use random_split(0.9, seed=1).
Further split our training data into two sets: training and validation. Use random_split(0.5, seed=1).

Again, we set seed=1 to obtain consistent results for different users.
Can you write Python code for it?

training_and_validation, testing = sales.random_split(0.9, seed=1)
training, validation = training_and_validation.random_split(0.5, seed=1)


I am working on the file "Linear_Regression_3_polynomial_regression_assignment.ipynb".
The first task is:
To test your function consider the smaller tmp variable and what you would expect the outcome of the following call:
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
After designating one of the k segments as the validation set, we train a model using the rest of the data. To choose the remainder, we slice (0:start) and (end+1:n) of the data and paste them together. SFrame has append() method that pastes together two disjoint sets of rows originating from a common dataset. For instance, the following cell pastes together the first and last two rows of the train_valid_shuffled dataframe.
Can you write Python code for it?

n = len(train_valid_shuffled)
first_two = train_valid_shuffled[0:2]
last_two = train_valid_shuffled[n-2:n]
print first_two.append(last_two)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
As in Week 3, we will use the sqft_living variable. For plotting purposes (connecting the dots), you'll need to sort by the values of sqft_living. For houses with identical square footage, we break the tie by their prices.
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Extract the remainder of the data after excluding fourth segment (segment 3) and assign the subset to train4.
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Let us familiarize ourselves with array slicing with SFrame. To extract a continuous slice from an SFrame, use colon in square brackets. For instance, the following cell extracts rows 0 to 9 of train_valid_shuffled. Notice that the first index (0) is included in the slice but the last index (10) is omitted.
Can you write Python code for it?

train_valid_shuffled[0:10] # rows 0 to 9


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Let's use matplotlib to visualize what a polynomial regression looks like on the house data.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

sales = graphlab.SFrame('kc_house_data.gl/')


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Note: When we have so many features and so few data points, the solution can become highly numerically unstable, which can sometimes lead to strange unpredictable results.  Thus, rather than using no regularization, we will introduce a tiny amount of regularization (l2_penalty=1e-5) to make the solution numerically stable.  (In lecture, we discussed the fact that regularization can also help with numerical stability, and here we are seeing a practical example.)
With the L2 penalty specified above, fit the model and print out the learned weights.
Hint: make sure to add 'price' column to the new SFrame before calling graphlab.linear_regression.create(). Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set=None in this call.
Can you write Python code for it?

poly_sframe = polynomial_sframe(sales['sqft_living'], 15)
my_features = poly_sframe.column_names()
poly_sframe['price'] = sales['price']
model = graphlab.linear_regression.create(poly_sframe, 'price', features=my_features,
                                          validation_set=None, l2_penalty=1e-5)

model.get('coefficients')


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Now let us extract individual segments with array slicing. Consider the scenario where we group the houses in the train_valid_shuffled dataframe into k=10 segments of roughly equal size, with starting and ending indices computed as above.
Extract the fourth segment (segment 3) and assign it to a variable called validation4.
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Once the data is shuffled, we divide it into equal segments. Each segment should receive n/k elements, where n is the number of observations in the training set and k is the number of segments. Since the segment 0 starts at index 0 and contains n/k elements, it ends at index (n/k)-1. The segment 1 starts where the segment 0 left off, at index (n/k). With n/k elements, the segment 1 ends at index (n*2/k)-1. Continuing in this fashion, we deduce that the segment i starts at index (n*i/k) and ends at (n*(i+1)/k)-1.
With this pattern in mind, we write a short loop that prints the starting and ending indices of each segment, just to make sure you are getting the splits right.
Can you write Python code for it?

n = len(train_valid_shuffled)
k = 10 # 10-fold cross-validation

for i in xrange(k):
    start = (n*i)/k
    end = (n*(i+1))/k-1
    print i, (start, end)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Once you found the best value for the L2 penalty using cross-validation, it is important to retrain a final model on all of the training data using this value of l2_penalty.  This way, your final model will be trained on the entire dataset.
Can you write Python code for it?

final_model = graphlab.linear_regression.create(train_data, target='price', l2_penalty=1000,
                                                verbose=False, features=feature_list, validation_set=None)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
Polynomial regression, revisited
We build on the material from Week 3, where we wrote the function to produce an SFrame with columns containing the powers of a given input. Copy and paste the function polynomial_sframe from Week 3:
Can you write Python code for it?

def polynomial_sframe(feature, degree):
    # assume that degree >= 1
    # initialize the SFrame:
    poly_sframe = graphlab.SFrame()
    # and set poly_sframe['power_1'] equal to the passed feature
    poly_sframe['power_1'] = feature
    # first check if degree > 1
    if degree > 1:
        # then loop over the remaining degrees:
        # range usually starts at 0 and stops at the endpoint-1. We want it to start at 2 and stop at degree
        for power in range(2, degree+1):
            # first we'll give the column a name:
            name = 'power_' + str(power)
            name_left = 'power_' + str(power-1)
            # then assign poly_sframe[name] to the appropriate power of feature
            poly_sframe[name] = feature * poly_sframe[name_left]

    return poly_sframe


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
QUIZ QUESTION:  What's the learned value for the coefficient of feature power_1?
Observe overfitting
Recall from Week 3 that the polynomial fit of degree 15 changed wildly whenever the data changed. In particular, when we split the sales data into four subsets and fit the model of degree 15, the result came out to be very different for each subset. The model had a high variance. We will see in a moment that ridge regression reduces such variance. But first, we must reproduce the experiment we did in Week 3.
First, split the data into split the sales data into four subsets of roughly equal size and call them set_1, set_2, set_3, and set_4. Use .random_split function and make sure you set seed=0.
Can you write Python code for it?

(semi_split1, semi_split2) = sales.random_split(.5,seed=0)
(set_1, set_2) = semi_split1.random_split(0.5, seed=0)
(set_3, set_4) = semi_split2.random_split(0.5, seed=0)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
QUIZ QUESTION: Using the best L2 penalty found above, train a model using all training data. What is the RSS on the TEST data of the model you learn with this L2 penalty?
Can you write Python code for it?

test_data = polynomial_sframe(test['sqft_living'], 15)
test_data['price'] = test['price']
predictions = final_model.predict(test_data)

test_error = predictions - test['price']

RSS = (test_error * test_error).sum()

RSS


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
These curves should vary a lot less, now that you applied a high degree of regularization.
QUIZ QUESTION:  For the models learned with the high level of regularization in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature power_1? (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)
Selecting an L2 penalty via cross-validation
Just like the polynomial degree, the L2 penalty is a "magic" parameter we need to select. We could use the validation set approach as we did in the last module, but that approach has a major disadvantage: it leaves fewer observations available for training. Cross-validation seeks to overcome this issue by using all of the training set in a smart way.
We will implement a kind of cross-validation called k-fold cross-validation. The method gets its name because it involves dividing the training set into k segments of roughtly equal size. Similar to the validation set method, we measure the validation error with one of the segments designated as the validation set. The major difference is that we repeat the process k times as follows:
Set aside segment 0 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

Set aside segment 1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

...

Set aside segment k-1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set
After this process, we compute the average of the k validation errors, and use it as an estimate of the generalization error. Notice that  all observations are used for both training and validation, as we iterate over segments of data.
To estimate the generalization error well, it is crucial to shuffle the training data before dividing them into segments. GraphLab Create has a utility function for shuffling a given...
Can you write Python code for it?

(train_valid, test) = sales.random_split(.9, seed=1)
train_valid_shuffled = graphlab.toolkits.cross_validation.shuffle(train_valid, random_seed=1)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the data with fourth segment excluded. When rounded to nearest whole number, the average should be $539,450.
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_1.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the fourth segment. When rounded to nearest whole number, the average should be $536,234.
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Compute the RSS on the TEST data for the following three sets of weights:
1. The initial weights (all zeros)
2. The weights learned with no regularization
3. The weights learned with high regularization
Which weights perform best?
Can you write Python code for it?

print multiple_weights_0_penalty
print multiple_weights_high_penalty

print compute_RSS(test_feature_matrix, test_output, initial_weights)

print compute_RSS(test_feature_matrix, test_output, multiple_weights_0_penalty)

print compute_RSS(test_feature_matrix, test_output, multiple_weights_high_penalty)


def compute_RSS(feature_matrix, output, weights):
    predictions = predict_output(feature_matrix, weights)
    errors = predictions - output
    return np.dot(errors, errors)

print compute_RSS(simple_test_feature_matrix, test_output, initial_weights)

print compute_RSS(simple_test_feature_matrix, test_output, simple_weights_0_penalty)

print compute_RSS(simple_test_feature_matrix, test_output, simple_weights_high_penalty)

print simple_weights_0_penalty
print simple_weights_high_penalty


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Computing the Derivative
We are now going to move to computing the derivative of the regression cost function. Recall that the cost function is the sum over the data points of the squared difference between an observed output and a predicted output, plus the L2 penalty term.
Cost(w)
= SUM[ (prediction - output)^2 ]
+ l2_penalty*(w[0]^2 + w[1]^2 + ... + w[k]^2).
Since the derivative of a sum is the sum of the derivatives, we can take the derivative of the first part (the RSS) as we did in the notebook for the unregularized case in Week 2 and add the derivative of the regularization part.  As we saw, the derivative of the RSS with respect to w[i] can be written as:
2*SUM[ error*[feature_i] ].
The derivative of the regularization term with respect to w[i] is:
2*l2_penalty*w[i].
Summing both, we get
2*SUM[ error*[feature_i] ] + 2*l2_penalty*w[i].
That is, the derivative for the weight for feature i is the sum (over data points) of 2 times the product of the error and the feature itself, plus 2*l2_penalty*w[i].
We will not regularize the constant.  Thus, in the case of the constant, the derivative is just twice the sum of the errors (without the 2*l2_penalty*w[0] term).
Recall that twice the sum of the product of two vectors is just twice the dot product of the two vectors. Therefore the derivative for the weight for feature_i is just two times the dot product between the values of feature_i and the current errors, plus 2*l2_penalty*w[i].
With this in mind complete the following derivative function which computes the derivative of the weight given the value of the feature (over all data points) and the errors (over all data points).  To decide when to we are dealing with the constant (so we don't regularize it) we added the extra parameter to the call feature_is_constant which you should set to True when computing the derivative of the constant and False otherwise.
Can you write Python code for it?

def feature_derivative_ridge(errors, feature, weight, l2_penalty, feature_is_constant):
    derivative = 0
    # If feature_is_constant is True, derivative is twice the dot product of errors and feature
    if feature_is_constant:
        derivative = 2 * np.dot(errors, feature)
    # Otherwise, derivative is twice the dot product plus 2*l2_penalty*weight
    else:
        derivative = 2 * np.dot(errors, feature) + 2 * l2_penalty * weight
    return derivative


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
First, let's consider no regularization.  Set the l2_penalty to 0.0 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
multiple_weights_0_penalty
Can you write Python code for it?

multiple_weights_0_penalty = ridge_regression_gradient_descent(feature_matrix, output, initial_weights,
                                            step_size, 0.0, max_iterations)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Gradient Descent
Now we will write a function that performs a gradient descent. The basic premise is simple. Given a starting point we update the current weights by moving in the negative gradient direction. Recall that the gradient is the direction of increase and therefore the negative gradient is the direction of decrease and we're trying to minimize a cost function.
The amount by which we move in the negative gradient direction  is called the 'step size'. We stop when we are 'sufficiently close' to the optimum. Unlike in Week 2, this time we will set a maximum number of iterations and take gradient steps until we reach this maximum number. If no maximum number is supplied, the maximum should be set 100 by default. (Use default parameter values in Python.)
With this in mind, complete the following gradient descent function below using your derivative function above. For each step in the gradient descent, we update the weight for each feature before computing our stopping criteria.
Can you write Python code for it?

def ridge_regression_gradient_descent(feature_matrix, output, initial_weights, step_size, l2_penalty, max_iterations=100):
    weights = np.array(initial_weights) # make sure it's a numpy array

    iter = 0
    #while not reached maximum number of iterations:
    while iter <= max_iterations:
        iter = iter + 1
        # compute the predictions based on feature_matrix and weights using your predict_output() function
        predictions = predict_output(feature_matrix, weights)
        # compute the errors as predictions - output
        errors = predictions - output

        for i in xrange(len(weights)): # loop over each weight
            # Recall that feature_matrix[:,i] is the feature column associated with weights[i]
            # compute the derivative for weight[i].
            #(Remember: when i=0, you are computing the derivative of the constant!)
            derivative = feature_derivative_ridge(errors, feature_matrix[:, i], weights[i], l2_penalty, i==0)

            # subtract the step size times the derivative from the current weight
            weights[i] = weights[i] - derivative * step_size

    return weights


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Let us split the dataset into training set and test set. Make sure to use seed=0:
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Let's set the parameters for our optimization:
Can you write Python code for it?

initial_weights = np.array([0., 0.])
step_size = 1e-12
max_iterations=1000


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Next, let's consider high regularization.  Set the l2_penalty to 1e11 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
simple_weights_high_penalty
we'll use them later.
Can you write Python code for it?

simple_weights_high_penalty = ridge_regression_gradient_descent(simple_feature_matrix, output, initial_weights,
                                            step_size, 1e11, max_iterations)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
QUIZ QUESTIONS
1. What is the value of the coefficient for sqft_living that you learned with no regularization, rounded to 1 decimal place?  What about the one with high regularization?
2. Comparing the lines you fit with the with no regularization versus high regularization, which one is steeper?
3. What are the RSS on the test data for each of the set of weights above (initial, no regularization, high regularization)?
Running a multiple regression with L2 penalty
Let us now consider a model with 2 features: ['sqft_living', 'sqft_living15'].
First, create Numpy versions of your training and test data with these two features.
Can you write Python code for it?

model_features = ['sqft_living', 'sqft_living15'] # sqft_living15 is the average squarefeet for the nearest 15 neighbors.
my_output = 'price'
(feature_matrix, output) = get_numpy_data(train_data, model_features, my_output)
(test_feature_matrix, test_output) = get_numpy_data(test_data, model_features, my_output)


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
This code will plot the two learned models.  (The blue line is for the model with no regularization and the red line is for the one with high regularization.)
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(simple_feature_matrix,output,'k.',
         simple_feature_matrix,predict_output(simple_feature_matrix, simple_weights_0_penalty),'b-',
        simple_feature_matrix,predict_output(simple_feature_matrix, simple_weights_high_penalty),'r-')


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
To test your feature derivartive run the following:
Can you write Python code for it?

(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price')
my_weights = np.array([1., 10.])
test_predictions = predict_output(example_features, my_weights)
errors = test_predictions - example_output # prediction errors

# next two lines should print the same values
print feature_derivative_ridge(errors, example_features[:,1], my_weights[1], 1, False)
print np.sum(errors*example_features[:,1])*2+20.
print ''

# next two lines should print the same values
print feature_derivative_ridge(errors, example_features[:,0], my_weights[0], 1, True)
print np.sum(errors)*2.


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
Visualizing effect of L2 penalty
The L2 penalty gets its name because it causes weights to have small L2 norms than otherwise. Let's see how large weights get penalized. Let us consider a simple model with 1 feature:
Can you write Python code for it?

simple_features = ['sqft_living']
my_output = 'price'


I am working on the file "Linear_Regression_4_ridge_regression_assignment_2.ipynb".
The first task is:
We need to re-inialize the weights, since we have one extra parameter. Let us also set the step size and maximum number of iterations.
Can you write Python code for it?

initial_weights = np.array([0.0,0.0,0.0])
step_size = 1e-12
max_iterations = 1000


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Applying L1 penalty requires adding an extra parameter (l1_penalty) to the linear regression call in GraphLab Create. (Other tools may have separate implementations of LASSO.)  Note that it's important to set l2_penalty=0 to ensure we don't introduce an additional L2 penalty.
Can you write Python code for it?

model_all = graphlab.linear_regression.create(sales, target='price', features=all_features,
                                              validation_set=None,
                                              l2_penalty=0., l1_penalty=1e10)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Create new features
As in Week 2, we consider features that are some transformations of inputs.
Can you write Python code for it?

from math import log, sqrt
sales['sqft_living_sqrt'] = sales['sqft_living'].apply(sqrt)
sales['sqft_lot_sqrt'] = sales['sqft_lot'].apply(sqrt)
sales['bedrooms_square'] = sales['bedrooms']*sales['bedrooms']

# In the dataset, 'floors' was defined with type string,
# so we'll convert them to float, before creating a new feature.
sales['floors'] = sales['floors'].astype(float)
sales['floors_square'] = sales['floors']*sales['floors']


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Limit the number of nonzero weights
What if we absolutely wanted to limit ourselves to, say, 7 features? This may be important if we want to derive "a rule of thumb" --- an interpretable model that has only a few features in them.
In this section, you are going to implement a simple, two phase procedure to achive this goal:
1. Explore a large range of l1_penalty values to find a narrow region of l1_penalty values where models are likely to have the desired number of non-zero weights.
2. Further explore the narrow region you found to find a good value for l1_penalty that achieves the desired sparsity.  Here, we will again use a validation set to choose the best value for l1_penalty.
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Note that a majority of the weights have been set to zero. So by setting an L1 penalty that's large enough, we are performing a subset selection.
QUIZ QUESTION:
According to this list of weights, which of the features have been chosen?
Selecting an L1 penalty
To find a good L1 penalty, we will explore multiple values using a validation set. Let us do three way split into train, validation, and test sets:
* Split our sales data into 2 sets: training and test
* Further split our training data into two sets: train, validation
Be very careful that you use seed = 1 to ensure you get the same answer!
Can you write Python code for it?

(training_and_validation, testing) = sales.random_split(.9,seed=1) # initial train/test split
(training, validation) = training_and_validation.random_split(0.5, seed=1) # split training into train and validate


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Out of this large range, we want to find the two ends of our desired narrow range of l1_penalty.  At one end, we will have l1_penalty values that have too few non-zeros, and at the other end, we will have an l1_penalty that has too many non-zeros.
More formally, find:
* The largest l1_penalty that has more non-zeros than max_nonzero (if we pick a penalty smaller than this value, we will definitely have too many non-zero weights)
    * Store this value in the variable l1_penalty_min (we will use it later)
* The smallest l1_penalty that has fewer non-zeros than max_nonzero (if we pick a penalty larger than this value, we will definitely have too few non-zero weights)
    * Store this value in the variable l1_penalty_max (we will use it later)
Hint: there are many ways to do this, e.g.:
* Programmatically within the loop above
* Creating a list with the number of non-zeros for each value of l1_penalty and inspecting it to find the appropriate boundaries.
Can you write Python code for it?

l1_penalty_min = l1_penalty_values[-6]
l1_penalty_max = l1_penalty_values[-5]

print l1_penalty_min
print l1_penalty_max


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
QUIZ QUESTION
Also, using this value of L1 penalty, how many nonzero weights do you have?
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
QUIZ QUESTIONS
1. What value of l1_penalty in our narrow range has the lowest RSS on the VALIDATION set and has sparsity equal to max_nonzeros?
2. What features in this model have non-zero coefficients?
Can you write Python code for it?

the_model = find_best_lasso(l1_penalty_values, max_nonzeros, training, validation, all_features, 'price')

the_model['coefficients'].print_rows(18)

the_model


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
QUIZ QUESTIONS
What values did you find for l1_penalty_min andl1_penalty_max?
Exploring the narrow range of values to find the solution with the right number of non-zeros that has lowest RSS on the validation set
We will now explore the narrow region of l1_penalty values we found:
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
QUIZ QUESTIONS
1. What was the best value for the l1_penalty?
2. What is the RSS on TEST data of the model with the best l1_penalty?
Can you write Python code for it?

predictions = best_model.predict(testing)
errors = predictions - testing['price']
RSS = np.dot(errors, errors)
print RSS


I am working on the file "Linear_Regression_5_lasso_regression_assignment_1.ipynb".
The first task is:
Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this variable will mostly affect houses with many bedrooms.
On the other hand, taking square root of sqft_living will decrease the separation between big house and small house. The owner may not be exactly twice as happy for getting a house that is twice as big.

Learn regression weights with L1 penalty
Let us fit a model with all the features available, plus the features we just created above.
Can you write Python code for it?

all_features = ['bedrooms', 'bedrooms_square',
            'bathrooms',
            'sqft_living', 'sqft_living_sqrt',
            'sqft_lot', 'sqft_lot_sqrt',
            'floors', 'floors_square',
            'waterfront', 'view', 'condition', 'grade',
            'sqft_above',
            'sqft_basement',
            'yr_built', 'yr_renovated']


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Also, copy and paste the predict_output() function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?

def predict_output(feature_matrix, weights):
    # assume feature_matrix is a numpy matrix containing the features as columns and weights is a corresponding numpy array
    # create the predictions vector by using np.dot()
    predictions = np.dot(feature_matrix, weights)
    return(predictions)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Compute the RSS of each of the three normalized weights on the (unnormalized) test_feature_matrix:
Can you write Python code for it?

print compute_RSS(test_feature_matrix, test_output, weights1e4_normalized)

print compute_RSS(test_feature_matrix, test_output, weights1e7_normalized)

print compute_RSS(test_feature_matrix, test_output, weights1e8_normalized)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Cyclical coordinate descent
Now that we have a function that optimizes the cost function over a single coordinate, let us implement cyclical coordinate descent where we optimize coordinates 0, 1, ..., (d-1) in order and repeat.
When do we know to stop? Each time we scan all the coordinates (features) once, we measure the change in weight for each coordinate. If no coordinate changes by more than a specified threshold, we stop.
For each iteration:
1. As you loop over features in order and perform coordinate descent, measure how much each coordinate changes.
2. After the loop, if the maximum change across all coordinates is falls below the tolerance, stop. Otherwise, go back to step 1.
Return weights
IMPORTANT: when computing a new weight for coordinate i, make sure to incorporate the new weights for coordinates 0, 1, ..., i-1. One good way is to update your weights variable in-place. See following pseudocode for illustration.
```
for i in range(len(weights)):
    old_weights_i = weights[i] # remember old value of weight[i], as it will be overwritten
    # the following line uses new values for weight[0], weight[1], ..., weight[i-1]
    #     and old values for weight[i], ..., weight[d-1]
    weights[i] = lasso_coordinate_descent_step(i, feature_matrix, output, weights, l1_penalty)
# use old_weights_i to compute change in coordinate
...

```
Can you write Python code for it?

def lasso_cyclical_coordinate_descent(feature_matrix, output, initial_weights, l1_penalty, tolerance):
    weights = initial_weights
    changes = [float('inf') for i in range(len(weights))]

    while any(change > tolerance for change in changes):
        for i in range(len(weights)):
            old_weights_i = weights[i] # remember old value of weight[i], as it will be overwritten
            # the following line uses new values for weight[0], weight[1], ..., weight[i-1]
            #     and old values for weight[i], ..., weight[d-1]
            weights[i] = lasso_coordinate_descent_step(i, feature_matrix, output, weights, l1_penalty)

            # use old_weights_i to compute change in coordinate
            changes[i] = abs(old_weights_i - weights[i])
    return weights


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Don't forget to normalize features:
Can you write Python code for it?

simple_feature_matrix, norms = normalize_features(simple_feature_matrix)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Evaluating LASSO fit with more features
Let us split the sales dataset into training and test sets.
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Evaluating each of the learned models on the test data
Let's now evaluate the three models on the test data:
Can you write Python code for it?

(test_feature_matrix, test_output) = get_numpy_data(test_data, all_features, 'price')


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Finally, learn the weights with l1_penalty=1e4, on the training data. Initialize weights to all zeros, and set the tolerance=5e5.  Call resulting weights weights1e4, you will need them later.  (This case will take quite a bit longer to converge than the others above.)
Can you write Python code for it?

initial_weights = np.zeros(1+len(all_features)) # plus the intercept
l1_penalty=1e4
tolerance=5e5
weights1e4 = lasso_cyclical_coordinate_descent(norm_train_matrix, train_output,
                                            initial_weights, l1_penalty, tolerance)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
First create a normalized version of the feature matrix, normalized_simple_feature_matrix
Can you write Python code for it?

(simple_feature_matrix, output) = get_numpy_data(sales, simple_features, my_output)
(normalized_simple_feature_matrix, simple_norms) = normalize_features(simple_feature_matrix) # normalize features


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
First, learn the weights with l1_penalty=1e7, on the training data. Initialize weights to all zeros, and set the tolerance=1.  Call resulting weights weights1e7, you will need them later.
Can you write Python code for it?

initial_weights = np.zeros(1+len(all_features)) # plus the intercept
l1_penalty=1e7
tolerance=1
weights1e7 = lasso_cyclical_coordinate_descent(norm_train_matrix, train_output,
                                            initial_weights, l1_penalty, tolerance)

def find_zero_weight_features(weights, features):
    zero_indices, = np.where(weights == 0)
    for i in zero_indices - 1:
        if i>=0:
            print i, features[i]

def find_none_zero_weight_features(weights, features):
    zero_indices, = np.where(weights != 0)
    for i in zero_indices - 1:
        if i>=0:
            print i, features[i]


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Implementing Coordinate Descent with normalized features
We seek to obtain a sparse set of weights by minimizing the LASSO cost function
SUM[ (prediction - output)^2 ] + lambda*( |w[1]| + ... + |w[k]|).
(By convention, we do not include w[0] in the L1 penalty term. We never want to push the intercept to zero.)
The absolute value sign makes the cost function non-differentiable, so simple gradient descent is not viable (you would need to implement a method called subgradient descent). Instead, we will use coordinate descent: at each iteration, we will fix all weights but weight i and find the value of weight i that minimizes the objective. That is, we look for
argmin_{w[i]} [ SUM[ (prediction - output)^2 ] + lambda*( |w[1]| + ... + |w[k]|) ]
where all weights other than w[i] are held to be constant. We will optimize one w[i] at a time, circling through the weights multiple times.
  1. Pick a coordinate i
  2. Compute w[i] that minimizes the cost function SUM[ (prediction - output)^2 ] + lambda*( |w[1]| + ... + |w[k]|)
  3. Repeat Steps 1 and 2 for all coordinates, multiple times
For this notebook, we use cyclical coordinate descent with normalized features, where we cycle through coordinates 0 to (d-1) in order, and assume the features were normalized as discussed above. The formula for optimizing each coordinate is as follows:
┌ (ro[i] + lambda/2)     if ro[i] < -lambda/2
w[i] = ├ 0                      if -lambda/2 <= ro[i] <= lambda/2
       └ (ro[i] - lambda/2)     if ro[i] > lambda/2
where
ro[i] = SUM[ [feature_i]*(output - prediction + w[i]*[feature_i]) ].
Note that we do not regularize the weight of the constant feature (intercept) w[0], so, for this weight, the update is simply:
w[0] = ro[i]
Effect of L1 penalty
Let us consider a simple model with 2 features:
Can you write Python code for it?

simple_features = ['sqft_living', 'bedrooms']
my_output = 'price'
(simple_feature_matrix, output) = get_numpy_data(sales, simple_features, my_output)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Let us consider the following set of features.
Can you write Python code for it?

all_features = ['bedrooms',
                'bathrooms',
                'sqft_living',
                'sqft_lot',
                'floors',
                'waterfront',
                'view',
                'condition',
                'grade',
                'sqft_above',
                'sqft_basement',
                'yr_built',
                'yr_renovated']

(train_matrix, train_output) = get_numpy_data(train_data, all_features, 'price')
(test_matrix, test_output) = get_numpy_data(test_data, all_features, 'price')


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?

sales = graphlab.SFrame('kc_house_data.gl/')
# In the dataset, 'floors' was defined with type string,
# so we'll convert them to int, before using it below
sales['floors'] = sales['floors'].astype(int)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Normalize features
In the house dataset, features vary wildly in their relative magnitude: sqft_living is very large overall compared to bedrooms, for instance. As a result, weight for sqft_living would be much smaller than weight for bedrooms. This is problematic because "small" weights are dropped first as l1_penalty goes up.
To give equal considerations for all features, we need to normalize features as discussed in the lectures: we divide each feature by its 2-norm so that the transformed feature has norm 1.
Let's see how we can do this normalization easily with Numpy: let us first consider a small matrix.
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Numpy provides a shorthand for computing 2-norms of each column:
Can you write Python code for it?

norms = np.linalg.norm(X, axis=0) # gives [norm(X[:,0]), norm(X[:,1]), norm(X[:,2])]
print norms


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
QUIZ QUESTION
Recall that, whenever ro[i] falls between -l1_penalty/2 and l1_penalty/2, the corresponding weight w[i] is sent to zero. Now suppose we were to take one step of coordinate descent on either feature 1 or feature 2. What range of values of l1_penalty would not set w[1] zero, but would set w[2] to zero, if we were to take a step in that coordinate?
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
QUIZ QUESTION
What features had non-zero weight in this case?
Can you write Python code for it?


find_zero_weight_features(weights1e7, all_features)

find_none_zero_weight_features(weights1e7, all_features)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
QUIZ QUESTION
What range of values of l1_penalty would set both w[1] and w[2] to zero, if we were to take a step in that coordinate?
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
QUIZ QUESTIONS
1. What is the RSS of the learned model on the normalized dataset?
2. Which features had weight zero at convergence?
Can you write Python code for it?

def compute_RSS(feature_matrix, output, weights):
    errors = predict_output(feature_matrix, weights) - output
    return np.dot(errors, errors)

RSS = compute_RSS(normalized_simple_feature_matrix, output, weights)
'{:.3E}'.format(RSS)

weights


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Rescaling learned weights
Recall that we normalized our feature matrix, before learning the weights.  To use these weights on a test set, we must normalize the test data in the same way.
Alternatively, we can rescale the learned weights to include the normalization, so we never have to worry about normalizing the test data:
In this case, we must scale the resulting weights so that we can make predictions with original features:
 1. Store the norms of the original features to a vector called norms:
features, norms = normalize_features(features)
 2. Run Lasso on the normalized features and obtain a weights vector
 3. Compute the weights for the original features by performing element-wise division, i.e.
weights_normalized = weights / norms
Now, we can apply weights_normalized to the test data, without normalizing it!
Create a normalized version of each of the weights learned above. (weights1e4, weights1e7, weights1e8).
Can you write Python code for it?

weights1e4_normalized = weights1e4 / norms
weights1e7_normalized = weights1e7 / norms
weights1e8_normalized = weights1e8 / norms


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
So we can say that ro[i] quantifies the significance of the i-th feature: the larger ro[i] is, the more likely it is for the i-th feature to be retained.
Single Coordinate Descent Step
Using the formula above, implement coordinate descent that minimizes the cost function over a single feature i. Note that the intercept (weight 0) is not regularized. The function should accept feature matrix, output, current weights, l1 penalty, and index of feature to optimize over. The function should return new weight for feature i.
Can you write Python code for it?

def lasso_coordinate_descent_step(i, feature_matrix, output, weights, l1_penalty):
    # compute prediction
    prediction = predict_output(feature_matrix, weights)
    # compute ro[i] = SUM[ [feature_i]*(output - prediction + weight[i]*[feature_i]) ]
    errors = output - prediction
    feature = feature_matrix[:, i]
    ro_i = compute_ro(feature, errors, weights[i])

    if i == 0: # intercept -- do not regularize
        new_weight_i = ro_i
    elif ro_i < -l1_penalty/2.:
        new_weight_i = ro_i + l1_penalty/2
    elif ro_i > l1_penalty/2.:
        new_weight_i = ro_i - l1_penalty/2
    else:
        new_weight_i = 0.

    return new_weight_i


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Then, run your implementation of LASSO coordinate descent:
Can you write Python code for it?

weights = lasso_cyclical_coordinate_descent(normalized_simple_feature_matrix, output,
                                            initial_weights, l1_penalty, tolerance)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
To check your results, if you call normalized_weights1e7 the normalized version of weights1e7, then:
print normalized_weights1e7[3]
should return 161.31745624837794.
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
To normalize, apply element-wise division:
Can you write Python code for it?


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
To test the function, run the following cell:
Can you write Python code for it?

# should print 0.425558846691
import math
print lasso_coordinate_descent_step(1, np.array([[3./math.sqrt(13),1./math.sqrt(10)],[2./math.sqrt(13),3./math.sqrt(10)]]),
                                   np.array([1., 1.]), np.array([1., 4.]), 0.1)


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
To test the function, run the following:
Can you write Python code for it?

features, norms = normalize_features(np.array([[3.,6.,9.],[4.,8.,12.]]))
print features
# should print
# [[ 0.6  0.6  0.6]
#  [ 0.8  0.8  0.8]]
print norms
# should print
# [5.  10.  15.]


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
Using the following parameters, learn the weights on the sales dataset.
Can you write Python code for it?

simple_features = ['sqft_living', 'bedrooms']
my_output = 'price'
initial_weights = np.zeros(3)
l1_penalty = 1e7
tolerance = 1.0


I am working on the file "Linear_Regression_5_lasso_regression_assignment_2.ipynb".
The first task is:
We assign some random set of initial weights and inspect the values of ro[i]:
Can you write Python code for it?


I am working on the file "Linear_Regression_6_local_regression_assignment.ipynb".
The first task is:
Aside: it is a good idea to write tests like this cell whenever you are vectorizing a complicated operation.
Perform 1-nearest neighbor regression
Now that we have the element-wise differences, it is not too hard to compute the Euclidean distances between our query house and all of the training houses. First, write a single-line expression to define a variable diff such that diff[i] gives the element-wise difference between the features of the query house and the i-th training house.
Can you write Python code for it?