I am working on the file "Regression.ipynb".
The first task is:
Linear regression to predict nutrition score
Can you write Python code for it?

x = data.as_matrix()
y = target.as_matrix()

x = np.array([np.concatenate((v,[1])) for v in x]) #add column of ones to the end of the data set
print x

linreg = LinearRegression()
linreg.fit(x,y)

p = linreg.predict(x)
p

err = abs(p-y)
err

total_error = np.dot(err,err)
rmse_train = np.sqrt(total_error/len(p))
rmse_train

linreg.coef_ #Regression Coefficients

pl.plot(p, y,'ro')
pl.plot([-25,50],[-25,50], 'g-')
pl.xlabel('predicted')
pl.ylabel('real')
pl.show()

# RMSE with 10-Fold Cross Validation
kf = KFold(len(x), n_folds=10)
xval_err = 0
for train,test in kf:
    linreg.fit(x[train],y[train])
    p = linreg.predict(x[test])
    e = p-y[test]
    print e
    xval_err += np.dot(e,e)

rmse_10cv = np.sqrt(xval_err/len(x))

print('Method: Linear Regression')
print('RMSE on training: %.4f' %rmse_train)
print('RMSE on 10-fold CV: %.4f' %rmse_10cv)


I am working on the file "Regression.ipynb".
The first task is:
Load the Boston housing data. This data set is pretty out-of-date since it was collected in the 1970s. Each of the 506 entries represents a local district in the Boston area. The standard target variable is the median home value in the district (in 1000s of dollars). Let's print out the description along with a histogram of the target. Notice that the distribution of median value is roughly Gaussian with a significant outlier -- there are around 15 very wealthy districts.
Can you write Python code for it?

boston = load_boston()
X, Y = boston.data, boston.target
plt.hist(Y, 50)
plt.xlabel('Median value (in $1000)')
print boston.DESCR


I am working on the file "Regression.ipynb".
The first task is:
Ok, let's try fitting a model that uses more of the variables. Let's run just a few iterations and check the cost function.
Can you write Python code for it?

num_feats = 5
theta, costs = gradient_descent(train_data[:,0:num_feats], train_labels, .01, 10)
plt.plot(map(np.log, costs))
plt.xlabel('Iteration'), plt.ylabel('Log Cost')
plt.show()


I am working on the file "Regression.ipynb".
The first task is:
Ok, let's try gradient descent again. We can increase the learning rate and decrease the number of iterations.
Can you write Python code for it?

start_time = time.time()
theta, costs = gradient_descent(scaled_train_data[:,0:5], train_labels, .01, 5000)
train_time = time.time() - start_time
plt.plot(map(np.log, costs))
plt.xlabel('Iteration'), plt.ylabel('Log Cost')
plt.show()

print 'Training time: %.2f secs' %train_time
print 'Our estimated theta:', theta
print 'OLS estimated theta:', OLS(scaled_train_data[:,0:5], train_labels)


I am working on the file "Regression.ipynb".
The first task is:
Print $C$, $M$, $A$, $B$, and $\theta$ and make sure that you understand how these are constructed.
Then compare mu_out to mu_fit.
So, what we did was to replace
C = np.identity(len(z_sample))*(dmu*dmu)
M = np.column_stack((np.ones(len(z_sample)),z_sample))
A = np.dot(np.dot(M.transpose(),np.linalg.pinv(C)),M)
B = np.dot(np.dot(M.transpose(),np.linalg.pinv(C)),mu_sample)
theta = np.dot(np.linalg.pinv(A),B)
mu_out = theta[0] + theta[1]*z
with
LRmodel = LinearRegression()
LRmodel.fit(z_sample[:, None], mu_sample, dmu)
mu_fit = LRmodel.predict(z[:, None])
A Word of Caution
Anscombe's Quartet is a demonstration of why you should always visualize your data and not just blindly use the parameters of some black-box fitting algorithm.

Each of these data sets has 11 points and basic statistical properties that are identical.
Multivariate regression
In the above cases, we were doing 2-D linear regression with a univariate $X$.  If $X$ is instead multivariate, then we fit a hyperplane rather than a straight line
$$y_i =\theta_0 + \theta_1x_{i1} + \theta_2x_{i2} + \cdots +\theta_kx_{ik} + \epsilon_i.$$
The design matrix, $M$, is now
$$M = \left(
\begin{array}{ccccccc}
1 & x_{01} & x_{02} & . & x_{0k}\
1 & x_{11} & x_{12} & . & x_{1k}\
. & . & . & .  & . \
1 & x_{N1} & x_{N2} & . & x_{Nk}\
\end{array}
\right)$$
but the whole formalism is the same as before.
Scikit-Learn obviously has a LinearRegression routine, but it does not explicitly account for heteroscedastic errors, so above we used the AstroML routine instead as illustrated below.
Can you write Python code for it?

import numpy as np
from astroML.linear_model import LinearRegression
X = np.random.random((100,2)) # 100 points in 2D
dy = np.random.random(100) # heteroscedastic errors
y = np.random.normal(X[:,0] + X[:,1],dy)
model = LinearRegression()
model.fit(X,y,dy)
y_pred = model.predict(X)


I am working on the file "Regression.ipynb".
The first task is:
Recreate the supernovae figure from above now using the polynomial regression algorithm.  (Hint: Don't overthink it.)
Can you write Python code for it?

#fit data using standard package
order = # Complete
n_constraints = order+1
poly = PolynomialRegression( # Complete
poly.fit(# Complete
# Complete the rest

fig = plt.figure(figsize=(6, 6))

#plot the data
ax = fig.add_subplot(111)
ax.plot(z, mu_fit, '-k')
ax.plot(z, mu_true, '--', c='gray')
ax.errorbar(z_sample, mu_sample, dmu, fmt='.k', ecolor='gray', lw=1)

ax.text(0.5, 0.05, r"$\chi^2_{\rm dof} = %.2f$" % chi2_dof,
    ha='center', va='bottom', transform=ax.transAxes, fontsize=14)

ax.set_xlim(0.01, 1.8)
ax.set_ylim(36.01, 48)
ax.text(0.05, 0.95, 'Polynomial regression', ha='left', va='top',
                transform=ax.transAxes)

ax.set_ylabel(r'$\mu$')
ax.set_xlabel(r'$z$')

ax.plot(z, mu_out, '-k', color='red')

plt.show()

Can you make the same code do linear regression?


I am working on the file "Regression.ipynb".
The first task is:
Regularizadores
Can you write Python code for it?

# Ridge para x_1
ridge = Ridge(alpha=2.5)
ridge.fit(features, Y_1)

y1_ridge = ridge.predict(sample_x1_poly)
plt.scatter(x_1, Y_1)
plt.plot(sample_x1, y1_ridge, color='r')


I am working on the file "Regression.ipynb".
The first task is:
Regularization
We have to be a little careful when doing regression because if we progressively increase the number of terms in the fit we reach a regime where we are overfitting the data (i.e. there are not enough degrees of freedom).
For example, let's look at an example of Polynomial Regression.
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from astroML.linear_model import PolynomialRegression

def f(x):
    """ function to approximate by polynomial interpolation"""
    return np.sin(x)

# generate points used to plot
x_plot = np.linspace(0, 8, 100)

# generate points and keep a subset of them
x = np.linspace(0, 8, 10)
#rng = np.random.RandomState(0)
#rng.shuffle(x)
#x = np.sort(x[:10])
y = f(x)+0.25*(np.random.random(len(x))-0.5)

# create matrix versions of these arrays
X = x[:, None]
X_plot = x_plot[:, None]

colors = ['teal', 'yellowgreen', 'gold']
lw = 2
plt.figure(figsize=(8,8))
plt.plot(x_plot, f(x_plot), color='cornflowerblue', linewidth=lw, label="ground truth")
plt.scatter(x, y, color='navy', s=30, marker='o', label="training points")

for count, degree in enumerate([3,4,5]):
    poly = PolynomialRegression(degree)
    poly.fit(X,y)
    y_plot = poly.predict(X_plot)

    plt.plot(x_plot, y_plot, color=colors[count], linewidth=lw, label="degree %d" % degree)

plt.legend(loc='lower left')

plt.show()


I am working on the file "Regression.ipynb".
The first task is:
The following examples compares Gaussian Basis Regression with and without the constraints from Ridge Regression.  It uses 100 evenly spaced Gauassian, which we can see strongly overfits the problem and has very large coefficient values, until a constraint is imposed.
Can you write Python code for it?

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt
from scipy.stats import lognorm

from sklearn.linear_model import LinearRegression, Ridge

from astroML.cosmology import Cosmology
from astroML.datasets import generate_mu_z

#----------------------------------------------------------------------
# generate data
np.random.seed(0)
z_sample, mu_sample, dmu = generate_mu_z(100, random_state=0)
cosmo = Cosmology()
z = np.linspace(0.01, 2, 1000)
mu = np.asarray(map(cosmo.mu, z))

#------------------------------------------------------------
# Manually convert data to a gaussian basis
#  note that we're ignoring errors here, for the sake of example.
def gaussian_basis(x, mu, sigma):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2)

centers = np.linspace(0, 1.8, 100)
widths = 0.2
X = gaussian_basis(z_sample[:, None], centers, widths)

#------------------------------------------------------------
# Set up the figure to plot the results
fig = plt.figure(figsize=(12, 8))

classifier = [LinearRegression, Ridge]
kwargs = [dict(), dict(alpha=0.005)]
labels = ['Gaussian Basis Regression', 'Ridge Regression']

for i in range(2):
    clf = classifier[i](fit_intercept=True, **kwargs[i])
    clf.fit(X, mu_sample)
    w = clf.coef_
    fit = clf.predict(gaussian_basis(z[:, None], centers, widths))

    # plot fit
    ax = fig.add_subplot(221 + i)
    ax.xaxis.set_major_formatter(plt.NullFormatter())

    # plot curves for regularized fits
    if i == 0:
        ax.set_ylabel('$\mu$')
    else:
        ax.yaxis.set_major_formatter(plt.NullFormatter())
        curves = 37 + w * gaussian_basis(z[:, np.newaxis], centers, widths)
        curves = curves[:, abs(w) > 0.01]
       ...


I am working on the file "Regression.ipynb".
The first task is:
This is fit with order = 3, 4, and 5.  What happens if you make the order $\sim N_{\rm points}$?
In this case we are clearly overfitting the data.  For cases where we are concerned with overfitting, instead of computing
$$(Y - M \theta)^T(Y- M \theta),$$
we can apply constraints (usually of smoothness, number of coefficients, size of coefficients):
$$(Y - M \theta)^T(Y- M \theta) + \lambda |\theta^T \theta|,$$
with $\lambda$ as the regularization parameter.
This leads to a solution for the parameters of the model
$$\theta = (M^T C^{-1} M + \lambda I)^{-1} (M^T C^{-1} Y)$$
with $I$ the identity matrix.
From the Bayesian perspective this is the same as applying a prior to the regression coefficients:
$$p(\theta | I ) \propto \exp{\left(\frac{-(\lambda \theta^T \theta)}{2}\right)}$$
which, when multiplied by the likelihood for regression, gives the same posterior as described above.
Ridge regression
The case for a Gaussian prior corresponds to Ridge Regression, which
penalizes the regression coefficients according to
$$ |\theta |^2 < s.$$
That is the square of each parameter in the fitting is restricted to be less than some value, $s$.  We'll come back to exactly what $s$ is in a minute.  Doing this supresses large coefficients and limits the variance of the system---at the cost of increased bias.
The following figure illustrates the interaction of the prior and the posterior without the prior:

Scikit-Learn's Ridge is their implementation of ridge regression, while AstroML implements Ridge Regression as follows:
Can you write Python code for it?

import numpy as np
from sklearn.linear_model import Ridge
X = np.random.random((100,10))
y = np.dot(X, np.random.random(10))
model = Ridge(alpha=0.05)
model.fit(X,y)
y_pred = model.predict(X)


I am working on the file "Regression.ipynb".
The first task is:
This is getting pretty slow, and it looks like it hasn't yet converged (see the last value of theta, especially). The scale of the parameters can also make convergence difficult. Let's examine the distributions of the features we're using.
Can you write Python code for it?

plt.figure(figsize=(15, 3))
for feature in range(num_feats):
    plt.subplot(1, num_feats, feature+1)
    plt.hist(train_data[:,feature])
    plt.title(boston.feature_names[feature])


I am working on the file "Regression.ipynb".
The first task is:
Varianza esperada y sesgo esperado
Can you write Python code for it?

# Expected prediction
def E(f_x):
    return float(sum(f_x)) / float(len(f_x))

x_real = np.array([x for _, x in X_real])
exp_pred_1 = E(f1(x_real))
avg_y_real = E(Y_real)
avg_y_real - exp_pred_1


I am working on the file "Regression.ipynb".
The first task is:
Visualizing data
Can you write Python code for it?

BaseYear = 1995
x = np.matrix(df_arac.Year[0:]).T-BaseYear
y = np.matrix(df_arac.Car[0:]).T/1000000.

plt.plot(x+BaseYear, y, 'o-')
plt.xlabel('Yil')
plt.ylabel('Araba (Millions)')

plt.show()


I am working on the file "Regression.ipynb".
The first task is:
We'll now repeat the supernova data example using basis function regression.
Can you write Python code for it?

#------------------------------------------------------------
# Define our Gaussians
nGaussians = 10
basis_mu = np.linspace(0,2,nGaussians)[:, None]
basis_sigma = 1.0 * (basis_mu[1] - basis_mu[0])
n_constraints = nGaussians+1

#fit data using gaussian-based basis function regression
bfr = BasisFunctionRegression('gaussian', mu=basis_mu, sigma=basis_sigma)
bfr.fit(z_sample[:, None], mu_sample, dmu)
mu_fit = bfr.predict(z[:, None])
mu_sample_fit = bfr.predict(z_sample[:, None])
chi2_dof = (np.sum(((mu_sample_fit - mu_sample) / dmu) ** 2) / (len(mu_sample) - n_constraints))

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(6, 6))

ax = fig.add_subplot(111)

ax.plot(z, mu_fit, '-k')
ax.plot(z, mu_true, '--', c='gray')
ax.errorbar(z_sample, mu_sample, dmu, fmt='.k', ecolor='gray', lw=1)

ax.text(0.5, 0.05, r"$\chi^2_{\rm dof} = %.2f$" % chi2_dof,
    ha='center', va='bottom', transform=ax.transAxes, fontsize=14)

ax.set_xlim(0.01, 1.8)
ax.set_ylim(36.01, 48)
ax.text(0.05, 0.95, 'Basis Function regression', ha='left', va='top',
                transform=ax.transAxes)

ax.set_ylabel(r'$\mu$')
ax.set_xlabel(r'$z$')

#ax.plot(z, mu_out, '-k', color='red')
plt.show()

# Do it by hand so that we can overplot the Gaussians

def gaussian_basis(x, mu, sigma):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2)

#------------------------------------------------------------
M = np.zeros(shape=[nGaussians, z_sample.shape[0]])
for i in range(nGaussians):
    M[i] = gaussian_basis(z_sample, basis_mu[i], basis_sigma)

M = np.matrix(M).T
C = np.matrix(np.diagflat(dmu**2))
Y = np.matrix(mu_sample).T
coeff = (M.T * C.I * M).I * (M.T * C.I * Y)

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(6, 6))
fig.subplots_adjust(left=0.1, right=0.95, bottom=0.1, top=0.95, hspace=0.05, wspace=0.05)

ax = fig.add_subplot(111)

# Plot the gaussians and their sum
i=0
mu_fit = np.zeros(len(z))
for i in range(nGaussians):
    mu_fit...


I am working on the file "Regression.ipynb".
The first task is:
Why do we even bother with gradient descent when the closed form solution works just fine?
There are many answers. First, gradient descent is a general-purpose tool; we are applying it to the least squares problem which happens to have a closed form solution, but most other machine learning objectives do not have this luxury.
Also, if we want to add regularization, we can no longer use the algebraic formula.
Here's one more reason: You can't take the inverse of a singular matrix. That is, if two of our features are co-linear, the inverse function will fail. Gradient descent doesn't have this problem and should learn instead to share the weight appropriately between the two co-linear features. Let's test this by simply adding a copy of the crime feature.
Can you write Python code for it?

# Create an augmented training set that has 2 copies of the crime variable.
augmented_train_data = np.c_[scaled_train_data[:,0], scaled_train_data]

# Run gradient descent and OLS and compare the results.
theta, costs = gradient_descent(augmented_train_data[:,0:6], train_labels, .01, 5000)
print 'Our estimated theta:', theta

print 'OLS estimated theta:',
try: print OLS(augmented_train_data[:,0:6], train_labels)
except: print 'ERROR, singular matrix not invertible'


I am working on the file "Regression.ipynb".
The first task is:
\begin{eqnarray}
\left(
\begin{array}{c}
y_0 \
y_1 \
y_2 \
y_{3} \
y_{4} \
z_{5} \
z_{6} \
z_{7} \
\end{array}
\right)
\approx
\left(
\begin{array}{ccccc}
z_{-1} & z_{-2} \
y_{0} & z_{-1} \
y_{1} & y_{0} \
y_{2} & y_{1} \
y_{3} & y_{2} \
y_{4} & y_{3} \
z_{5} & y_{4} \
z_{6} & z_{5} \
\end{array}
\right)
\left(
\begin{array}{c}
 w_1  \ w_2
\end{array}
\right)
\end{eqnarray}
$$
0 \approx y_k - w_1 y_{k-1} - w_2 y_{k-2}
$$
$v_0 = 1, v_1 = -w_1, v_2 = -w_2$
$$
0 \approx \sum_{r} \sum_{s}  v_r y_{s} \delta(k-r, s)
$$
Can you write Python code for it?

%matplotlib inline
#from cvxpy import *
import cvxpy as cvx
import numpy as np
import matplotlib as mpl
import matplotlib.pylab as plt

np.set_printoptions(precision=3)

x = np.matrix('[0,1,2,3,4,5]').T
xx = np.matrix(np.arange(-1,6,0.1)).T
y = np.matrix('[2,4,6,-1,10,12]').T
N = len(x)
degree = 3
#A = np.hstack((np.power(x,0), np.power(x,1), np.power(x,2)))
A = np.hstack((np.power(x,i) for i in range(degree+1)))
A2 = np.hstack((np.power(xx,i) for i in range(degree+1)))

# Construct the problem.
w = cvx.Variable(degree+1)
objective = cvx.Minimize(cvx.norm(A*w - y, 1.1))
#constraints = [0 <= x, x <= 10]
#prob = Problem(objective, constraints)
prob = cvx.Problem(objective)

# The optimal objective is returned by prob.solve().
result = prob.solve()
# The optimal value for x is stored in x.value.
print(w.value)
# The optimal Lagrange multiplier for a constraint
# is stored in constraint.dual_value.
#print(constraints[0].dual_value)

plt.plot(x.T.tolist(), y.T.tolist(), 'o')
plt.plot(xx, A2*w.value, '-')
plt.show()


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
An interesting side effect of random forest classification, is that you can measure how 'important' each feature is when predicting the final result
Can you write Python code for it?

imp_features = (np.sort((clf_et.feature_importances_,features),axis=0))

for rank,f in zip(imp_features[0],imp_features[1]):
    print("{0:.3f} <-> {1}".format(float(rank), f))


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
Here we will try to predict the C6H6(GT) values. Hence we set it as our target variables
We split the dataset to 60% training and 40% testing sets.
Can you write Python code for it?

# split dataset to 60% training and 40% testing
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y, test_size=0.4, random_state=0)

print(X_train.shape, y_train.shape)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
Linear Regression
Can you write Python code for it?

regressor = LinearRegression()
regressor.fit(X_train, y_train)
y_predictions = regressor.predict(X_test)
print('R-squared:', regressor.score(X_test, y_test))


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
Random Forests for Regression Analysis
Finally, let's try again Random Forests, in their Extra Trees, and Regression version
Can you write Python code for it?

from sklearn import ensemble
clf_et=ensemble.ExtraTreesRegressor(n_estimators=10,random_state=42)
train_and_evaluate(clf_et,X_train,y_train)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
Selecting the best features
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.25, random_state=33)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
Sometimes there are a lot of features in the dataset, so before learning, we should try to see which features are more relevant for our learning task, i.e. which of them are better prize predictors. We will use the SelectKBest method from the feature_selection package, and plot the results.
Can you write Python code for it?

df.columns

feature_names = list(df.columns[2:])

feature_names.remove('PT08.S4(NO2)')

import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.feature_selection import *

fs=SelectKBest(score_func=f_regression,k=5)
X_new=fs.fit_transform(X_train,y_train)
print((fs.get_support(),feature_names))

x_min, x_max = X_new[:,0].min(), X_new[:, 0].max()
y_min, y_max = y_train.min(), y_train.max()

fig=plt.figure()
#fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)

# Two subplots, unpack the axes array immediately
fig, axes = plt.subplots(1,5)

fig.set_size_inches(12,12)

for i in range(5):
    axes[i].set_aspect('equal')
    axes[i].set_title('Feature ' + str(i))
    axes[i].set_xlabel('Feature')
    axes[i].set_ylabel('Target')
    axes[i].set_xlim(x_min, x_max)
    axes[i].set_ylim(y_min, y_max)
    plt.sca(axes[i])
    plt.scatter(X_new[:,i],y_train)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
Support Vector Machines for regression
The regression version of SVM can be used instead to find the hyperplane (note how easy is to change the classification method in scikit-learn!). We will try a linear kernel, a polynomial kernel, and finally, a rbf kernel. For more information on kernels, see http://scikit-learn.org/stable/modules/svm.html#svm-kernels
Can you write Python code for it?

from sklearn import svm
clf_svr= svm.SVR(kernel='linear')
train_and_evaluate(clf_svr,X_train,y_train)

clf_svr_poly= svm.SVR(kernel='poly')
train_and_evaluate(clf_svr_poly,X_train,y_train)

clf_svr_rbf= svm.SVR(kernel='rbf')
train_and_evaluate(clf_svr_rbf,X_train,y_train)

clf_svr_poly2= svm.SVR(kernel='poly',degree=2)
train_and_evaluate(clf_svr_poly2,X_train,y_train)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
The R-squared score of 1 indicates that 100 percent of the variance in the test set is explained by the model. The performance can change if a different set of 75 percent of the data is partitioned to the training set. Hence Cross-validation can be used to produce a better estimate of the estimator's performance. Each cross-validation round trains and tests different partitions of the data to reduce variability.
Can you write Python code for it?

scores = cross_val_score(regressor, X, y, cv=5)
print ("Average of scores: ", scores.mean())
print ("Cross validation scores: ", scores)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
The data contains null values. So we drop those rows and columns containing nulls.
Can you write Python code for it?


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
We will use the AirQualityUCI.csv file as our dataset. It is a ';' seperated file so we'll specify it as a parameter for the read_csv function. We'll also use parse_dates parameter so that pandas recognizes the 'Date' and 'Time' columns and format them accordingly
Can you write Python code for it?

df.head()

df.dropna(how="all",axis=1,inplace=True)

df.dropna(how="all",axis=0,inplace=True)


I am working on the file "Regression/Air-Quality-Prediction.ipynb".
The first task is:
You probably noted the penalty=None parameter when we called the method. The penalization parameter for linear regression methods is introduced to avoid overfitting. It does this by penalizing those hyperplanes having some of their coefficients too large, seeking hyperplanes where each feature contributes more or less the same to the predicted value. This parameter is generally the L2 norm (the squared sums of the coefficients) or the L1 norm (that is the sum of the absolute value of the coefficients). Let's see how our model works if we introduce an L2 or L1 penalty.
Can you write Python code for it?

clf_sgd1 = linear_model.SGDRegressor(loss='squared_loss', penalty='l2',  random_state=42)
train_and_evaluate(clf_sgd1,X_train,y_train)

clf_sgd2 = linear_model.SGDRegressor(loss='squared_loss', penalty='l1',  random_state=42)
train_and_evaluate(clf_sgd2,X_train,y_train)


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
After designating one of the k segments as the validation set, we train a model using the rest of the data. To choose the remainder, we slice (0:start) and (end+1:n) of the data and paste them together. SFrame has append() method that pastes together two disjoint sets of rows originating from a common dataset. For instance, the following cell pastes together the first and last two rows of the train_valid_shuffled dataframe.
Can you write Python code for it?

n = len(train_valid_shuffled)
first_two = train_valid_shuffled[0:2]
last_two = train_valid_shuffled[n-2:n]


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Extract the remainder of the data after excluding fourth segment (segment 3) and assign the subset to train4.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Let us familiarize ourselves with array slicing with SFrame. To extract a continuous slice from an SFrame, use colon in square brackets. For instance, the following cell extracts rows 0 to 9 of train_valid_shuffled. Notice that the first index (0) is included in the slice but the last index (10) is omitted.
Can you write Python code for it?

train_valid_shuffled[0:10] # rows 0 to 9


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Let us revisit the 15th-order polynomial model using the 'sqft_living' input. Generate polynomial features up to degree 15 using polynomial_sframe() and fit a model with these features. When fitting the model, use an L2 penalty of 1e-5:
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Let's use matplotlib to visualize what a polynomial regression looks like on the house data.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

sales = graphlab.SFrame('kc_house_data.gl/kc_house_data.gl')


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Next, fit a 15th degree polynomial on set_1, set_2, set_3, and set_4, using 'sqft_living' to predict prices. Print the weights and make a plot of the resulting model.
Hint: When calling graphlab.linear_regression.create(), use the same L2 penalty as before (i.e. l2_small_penalty).  Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set = None in this call.
Can you write Python code for it?

poly1_data = polynomial_sframe(set_1['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_1['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=l2_small_penalty)
model1.get("coefficients")

poly1_data = polynomial_sframe(set_2['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_2['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=l2_small_penalty)
model1.get("coefficients")

poly1_data = polynomial_sframe(set_3['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_3['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=l2_small_penalty)
model1.get("coefficients")

poly1_data = polynomial_sframe(set_4['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_4['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=l2_small_penalty)
model1.get("coefficients")


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Now let us extract individual segments with array slicing. Consider the scenario where we group the houses in the train_valid_shuffled dataframe into k=10 segments of roughly equal size, with starting and ending indices computed as above.
Extract the fourth segment (segment 3) and assign it to a variable called validation4.
Can you write Python code for it?

validation4 = train_valid_shuffled[5818:7758] #rows 0 to 9


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Once we have a function to compute the average validation error for a model, we can write a loop to find the model that minimizes the average validation error. Write a loop that does the following:
* We will again be aiming to fit a 15th-order polynomial model using the sqft_living input
* For l2_penalty in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, you can use this Numpy function: np.logspace(1, 7, num=13).)
    * Run 10-fold cross-validation with l2_penalty
* Report which L2 penalty produced the lowest average validation error.
Note: since the degree of the polynomial is now fixed to 15, to make things faster, you should generate polynomial features in advance and re-use them throughout the loop. Make sure to use train_valid_shuffled when generating polynomial features!
Can you write Python code for it?

import numpy as np

poly_data = polynomial_sframe(train_valid_shuffled['sqft_living'], 15)

my_features = poly_data.column_names()

poly_data['price'] = train_valid_shuffled['price']

for l2_penalty in np.logspace(1, 7, num=13):

    Val_err = k_fold_cross_validation(10, l2_penalty, poly_data, my_features)
    print l2_penalty
    print Val_err


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Once you found the best value for the L2 penalty using cross-validation, it is important to retrain a final model on all of the training data using this value of l2_penalty.  This way, your final model will be trained on the entire dataset.
Can you write Python code for it?

poly1_data = polynomial_sframe(train_valid_shuffled['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = train_valid_shuffled['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=1000)
Val_err = k_fold_cross_validation(10, 1000, poly1_data, my_features)


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
Polynomial regression, revisited
We build on the material from Week 3, where we wrote the function to produce an SFrame with columns containing the powers of a given input. Copy and paste the function polynomial_sframe from Week 3:
Can you write Python code for it?

def polynomial_sframe(feature, degree):
    # assume that degree >= 1
    # initialize the SFrame:
    poly_sframe = graphlab.SFrame()
    # and set poly_sframe['power_1'] equal to the passed feature
    poly_sframe['power_1'] = feature
    # first check if degree > 1
    if degree > 1:
        # then loop over the remaining degrees:
        # range usually starts at 0 and stops at the endpoint-1. We want it to start at 2 and stop at degree
        for power in range(2, degree+1):
            # first we'll give the column a name:
            name = 'power_' + str(power)
            # then assign poly_sframe[name] to the appropriate power of feature
            poly_sframe[name] = feature**power
    return poly_sframe


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTIONS:  What is the best value for the L2 penalty according to 10-fold validation?
You may find it useful to plot the k-fold cross-validation errors you have obtained to better understand the behavior of the method.
Can you write Python code for it?

# Plot the l2_penalty values in the x axis and the cross-validation error in the y axis.
# Using plt.xscale('log') will make your plot more intuitive.


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
The four curves should differ from one another a lot, as should the coefficients you learned.
QUIZ QUESTION:  For the models learned in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature power_1?  (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)
Ridge regression comes to rescue
Generally, whenever we see weights change so much in response to change in data, we believe the variance of our estimate to be large. Ridge regression aims to address this issue by penalizing "large" weights. (Weights of model15 looked quite small, but they are not that small because 'sqft_living' input is in the order of thousands.)
With the argument l2_penalty=1e5, fit a 15th-order polynomial model on set_1, set_2, set_3, and set_4. Other than the change in the l2_penalty parameter, the code should be the same as the experiment above. Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set = None in this call.
Can you write Python code for it?

poly1_data = polynomial_sframe(set_1['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_1['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=1e5)
model1.get("coefficients")

poly1_data = polynomial_sframe(set_2['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_2['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=1e5)
model1.get("coefficients")

poly1_data = polynomial_sframe(set_3['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_3['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=1e5)
model1.get("coefficients")

poly1_data = polynomial_sframe(set_4['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = set_4['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=1e5)
model1.get("coefficients")


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
These curves should vary a lot less, now that you applied a high degree of regularization.
QUIZ QUESTION:  For the models learned with the high level of regularization in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature power_1? (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)
Selecting an L2 penalty via cross-validation
Just like the polynomial degree, the L2 penalty is a "magic" parameter we need to select. We could use the validation set approach as we did in the last module, but that approach has a major disadvantage: it leaves fewer observations available for training. Cross-validation seeks to overcome this issue by using all of the training set in a smart way.
We will implement a kind of cross-validation called k-fold cross-validation. The method gets its name because it involves dividing the training set into k segments of roughtly equal size. Similar to the validation set method, we measure the validation error with one of the segments designated as the validation set. The major difference is that we repeat the process k times as follows:
Set aside segment 0 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

Set aside segment 1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set

...

Set aside segment k-1 as the validation set, and fit a model on rest of data, and evalutate it on this validation set
After this process, we compute the average of the k validation errors, and use it as an estimate of the generalization error. Notice that  all observations are used for both training and validation, as we iterate over segments of data.
To estimate the generalization error well, it is crucial to shuffle the training data before dividing them into segments. GraphLab Create has a utility function for shuffling a given...
Can you write Python code for it?

(train_valid, test) = sales.random_split(.9, seed=1)
train_valid_shuffled = graphlab.toolkits.cross_validation.shuffle(train_valid, random_seed=1)


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the data with fourth segment excluded. When rounded to nearest whole number, the average should be $539,450.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-1-blank-checkpoint.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the fourth segment. When rounded to nearest whole number, the average should be $536,234.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Also, copy and paste the predict_output() function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?

def predict_output(feature_matrix, weights):
    # assume feature_matrix is a numpy matrix containing the features as columns and weights is a corresponding numpy array
    # create the predictions vector by using np.dot()
    predictions = np.dot(feature_matrix, weights)
    return(predictions)


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
First, let's consider no regularization.  Set the l2_penalty to 0.0 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
simple_weights_0_penalty
we'll use them later.
Next, let's consider high regularization.  Set the l2_penalty to 1e11 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
simple_weights_high_penalty
we'll use them later.
This code will plot the two learned models.  (The blue line is for the model with no regularization and the red line is for the one with high regularization.)
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(simple_feature_matrix,output,'k.',
         simple_feature_matrix,predict_output(simple_feature_matrix, simple_weights_0_penalty),'b-',
        simple_feature_matrix,predict_output(simple_feature_matrix, simple_weights_high_penalty),'r-')


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
If we want to do any "feature engineering" like creating new features or adjusting existing ones we should do this directly using the SFrames as seen in the first notebook of Week 2. For this notebook, however, we will work with the existing features.
Import useful functions from previous notebook
As in Week 2, we convert the SFrame into a 2D Numpy array. Copy and paste get_num_data() from the second notebook of Week 2.
Can you write Python code for it?

import numpy as np # note this allows us to refer to numpy as np instead

def get_numpy_data(data_sframe, features, output):
    data_sframe['constant'] = 1 # this is how you add a constant column to an SFrame
    # add the column 'constant' to the front of the features list so that we can extract it along with the others:
    features = ['constant'] + features # this is how you combine two lists
    # select the columns of data_SFrame given by the features list into the SFrame features_sframe (now including constant):
    features_sframe = data_sframe[features]
    # the following line will convert the features_SFrame into a numpy matrix:
    feature_matrix = features_sframe.to_numpy()
    # assign the column of data_sframe associated with the output to the SArray output_sarray
    output_sarray = data_sframe['price']
    # the following will convert the SArray into a numpy array by first converting it to a list
    output_array = output_sarray.to_numpy()
    return(feature_matrix, output_array)


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Let us split the dataset into training set and test set. Make sure to use seed=0:
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Let's set the parameters for our optimization:
Can you write Python code for it?

initial_weights = np.array([0., 0.])
step_size = 1e-12
max_iterations=1000


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
To test your feature derivartive run the following:
Can you write Python code for it?

import numpy as np
(example_features, example_output) = get_numpy_data(sales, ['sqft_living'], 'price')
my_weights = np.array([1., 10.])
test_predictions = predict_output(example_features, my_weights)
errors = test_predictions - example_output # prediction errors

# next two lines should print the same values
print feature_derivative_ridge(errors, example_features[:,1], my_weights[1], 1, False)
print np.sum(errors*example_features[:,1])*2+20.
print ''

# next two lines should print the same values
print feature_derivative_ridge(errors, example_features[:,0], my_weights[0], 1, True)
print np.sum(errors)*2.


I am working on the file "Regression/Assignment_four/.ipynb_checkpoints/week-4-ridge-regression-assignment-2-blank-checkpoint.ipynb".
The first task is:
Visualizing effect of L2 penalty
The L2 penalty gets its name because it causes weights to have small L2 norms than otherwise. Let's see how large weights get penalized. Let us consider a simple model with 1 feature:
Can you write Python code for it?

simple_features = ['sqft_living']
my_output = 'price'


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
As in Week 3, we will use the sqft_living variable. For plotting purposes (connecting the dots), you'll need to sort by the values of sqft_living. For houses with identical square footage, we break the tie by their prices.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Let's use matplotlib to visualize what a polynomial regression looks like on the house data.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

sales = graphlab.SFrame('kc_house_data.gl/kc_house_data.gl')

len(sales[0])


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Next, fit a 15th degree polynomial on set_1, set_2, set_3, and set_4, using 'sqft_living' to predict prices. Print the weights and make a plot of the resulting model.
Hint: When calling graphlab.linear_regression.create(), use the same L2 penalty as before (i.e. l2_small_penalty).  Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set = None in this call.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Note: When we have so many features and so few data points, the solution can become highly numerically unstable, which can sometimes lead to strange unpredictable results.  Thus, rather than using no regularization, we will introduce a tiny amount of regularization (l2_penalty=1e-5) to make the solution numerically stable.  (In lecture, we discussed the fact that regularization can also help with numerical stability, and here we are seeing a practical example.)
With the L2 penalty specified above, fit the model and print out the learned weights.
Hint: make sure to add 'price' column to the new SFrame before calling graphlab.linear_regression.create(). Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set=None in this call.
Can you write Python code for it?

poly1_data = polynomial_sframe(sales['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = sales['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, validation_set = None, l2_penalty=l2_small_penalty)
model1.get("coefficients")


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Once the data is shuffled, we divide it into equal segments. Each segment should receive n/k elements, where n is the number of observations in the training set and k is the number of segments. Since the segment 0 starts at index 0 and contains n/k elements, it ends at index (n/k)-1. The segment 1 starts where the segment 0 left off, at index (n/k). With n/k elements, the segment 1 ends at index (n*2/k)-1. Continuing in this fashion, we deduce that the segment i starts at index (n*i/k) and ends at (n*(i+1)/k)-1.
With this pattern in mind, we write a short loop that prints the starting and ending indices of each segment, just to make sure you are getting the splits right.
Can you write Python code for it?

n = len(train_valid_shuffled)
k = 10 # 10-fold cross-validation

for i in xrange(k):
    start = (n*i)/k
    end = (n*(i+1))/k-1
    print i, (start, end)


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Once we have a function to compute the average validation error for a model, we can write a loop to find the model that minimizes the average validation error. Write a loop that does the following:
* We will again be aiming to fit a 15th-order polynomial model using the sqft_living input
* For l2_penalty in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, you can use this Numpy function: np.logspace(1, 7, num=13).)
    * Run 10-fold cross-validation with l2_penalty
* Report which L2 penalty produced the lowest average validation error.
Note: since the degree of the polynomial is now fixed to 15, to make things faster, you should generate polynomial features in advance and re-use them throughout the loop. Make sure to use train_valid_shuffled when generating polynomial features!
Can you write Python code for it?

import numpy as np
poly_data = polynomial_sframe(train_valid_shuffled['sqft_living'], 15)

my_features = poly_data.column_names()

poly_data['price'] = train_valid_shuffled['price']

for l2_penalty in np.logspace(1, 7, num=13):

    Val_err = k_fold_cross_validation(10, l2_penalty, poly_data, my_features)
    print l2_penalty
    print Val_err


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Once you found the best value for the L2 penalty using cross-validation, it is important to retrain a final model on all of the training data using this value of l2_penalty.  This way, your final model will be trained on the entire dataset.
Can you write Python code for it?

poly1_data = polynomial_sframe(train_valid_shuffled['sqft_living'], 15) # use equivalent of `polynomial_sframe`
my_features = poly1_data.column_names()
poly1_data['price'] = train_valid_shuffled['price']
model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = my_features, verbose = False, validation_set = None, l2_penalty=1000)
Val_err = k_fold_cross_validation(10, 1000, poly1_data, my_features)
print Val_err


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Polynomial regression, revisited
We build on the material from Week 3, where we wrote the function to produce an SFrame with columns containing the powers of a given input. Copy and paste the function polynomial_sframe from Week 3:
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
QUIZ QUESTION:  What's the learned value for the coefficient of feature power_1?
Observe overfitting
Recall from Week 3 that the polynomial fit of degree 15 changed wildly whenever the data changed. In particular, when we split the sales data into four subsets and fit the model of degree 15, the result came out to be very different for each subset. The model had a high variance. We will see in a moment that ridge regression reduces such variance. But first, we must reproduce the experiment we did in Week 3.
First, split the data into split the sales data into four subsets of roughly equal size and call them set_1, set_2, set_3, and set_4. Use .random_split function and make sure you set seed=0.
Can you write Python code for it?

(semi_split1, semi_split2) = sales.random_split(.5,seed=0)
(set_1, set_2) = semi_split1.random_split(0.5, seed=0)
(set_3, set_4) = semi_split2.random_split(0.5, seed=0)


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
QUIZ QUESTIONS:  What is the best value for the L2 penalty according to 10-fold validation?
You may find it useful to plot the k-fold cross-validation errors you have obtained to better understand the behavior of the method.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
The four curves should differ from one another a lot, as should the coefficients you learned.
QUIZ QUESTION:  For the models learned in each of these training sets, what are the smallest and largest values you learned for the coefficient of feature power_1?  (For the purpose of answering this question, negative numbers are considered "smaller" than positive numbers. So -5 is smaller than -3, and -3 is smaller than 5 and so forth.)
Ridge regression comes to rescue
Generally, whenever we see weights change so much in response to change in data, we believe the variance of our estimate to be large. Ridge regression aims to address this issue by penalizing "large" weights. (Weights of model15 looked quite small, but they are not that small because 'sqft_living' input is in the order of thousands.)
With the argument l2_penalty=1e5, fit a 15th-order polynomial model on set_1, set_2, set_3, and set_4. Other than the change in the l2_penalty parameter, the code should be the same as the experiment above. Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set = None in this call.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the data with fourth segment excluded. When rounded to nearest whole number, the average should be $539,450.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
To verify that we have the right elements extracted, run the following cell, which computes the average price of the fourth segment. When rounded to nearest whole number, the average should be $536,234.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
Also, copy and paste the predict_output() function to compute the predictions for an entire matrix of features given the matrix and the weights:
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
Computing the Derivative
We are now going to move to computing the derivative of the regression cost function. Recall that the cost function is the sum over the data points of the squared difference between an observed output and a predicted output, plus the L2 penalty term.
Cost(w)
= SUM[ (prediction - output)^2 ]
+ l2_penalty*(w[0]^2 + w[1]^2 + ... + w[k]^2).
Since the derivative of a sum is the sum of the derivatives, we can take the derivative of the first part (the RSS) as we did in the notebook for the unregularized case in Week 2 and add the derivative of the regularization part.  As we saw, the derivative of the RSS with respect to w[i] can be written as:
2*SUM[ error*[feature_i] ].
The derivative of the regularization term with respect to w[i] is:
2*l2_penalty*w[i].
Summing both, we get
2*SUM[ error*[feature_i] ] + 2*l2_penalty*w[i].
That is, the derivative for the weight for feature i is the sum (over data points) of 2 times the product of the error and the feature itself, plus 2*l2_penalty*w[i].
We will not regularize the constant.  Thus, in the case of the constant, the derivative is just twice the sum of the errors (without the 2*l2_penalty*w[0] term).
Recall that twice the sum of the product of two vectors is just twice the dot product of the two vectors. Therefore the derivative for the weight for feature_i is just two times the dot product between the values of feature_i and the current errors, plus 2*l2_penalty*w[i].
With this in mind complete the following derivative function which computes the derivative of the weight given the value of the feature (over all data points) and the errors (over all data points).  To decide when to we are dealing with the constant (so we don't regularize it) we added the extra parameter to the call feature_is_constant which you should set to True when computing the derivative of the constant and False otherwise.
Can you write Python code for it?

def feature_derivative_ridge(errors, feature, weight, l2_penalty, feature_is_constant):
    # If feature_is_constant is True, derivative is twice the dot product of errors and feature
    if(feature_is_constant):
        derivative = 2*np.dot(errors,feature)
    # Otherwise, derivative is twice the dot product plus 2*l2_penalty*weight
    else:
        derivative = 2*np.dot(errors,feature) + 2*l2_penalty*weight
    return derivative


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
First, let's consider no regularization.  Set the l2_penalty to 0.0 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
multiple_weights_0_penalty
Can you write Python code for it?

l2_penalty = 0.0
multiple_weights_0_penalty = ridge_regression_gradient_descent(feature_matrix, output, initial_weights, step_size, l2_penalty, max_iterations)
print multiple_weights_0_penalty


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
If we want to do any "feature engineering" like creating new features or adjusting existing ones we should do this directly using the SFrames as seen in the first notebook of Week 2. For this notebook, however, we will work with the existing features.
Import useful functions from previous notebook
As in Week 2, we convert the SFrame into a 2D Numpy array. Copy and paste get_num_data() from the second notebook of Week 2.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
Let us split the dataset into training set and test set. Make sure to use seed=0:
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
Next, let's consider high regularization.  Set the l2_penalty to 1e11 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
multiple_weights_high_penalty
Can you write Python code for it?

l2_penalty = 1e11
multiple_weights_high_penalty = ridge_regression_gradient_descent(feature_matrix, output, initial_weights, step_size, l2_penalty, max_iterations)
print multiple_weights_high_penalty


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
Next, let's consider high regularization.  Set the l2_penalty to 1e11 and run your ridge regression algorithm to learn the weights of your model.  Call your weights:
simple_weights_high_penalty
we'll use them later.
Can you write Python code for it?

l2_penalty = 1e11
simple_weights_high_penalty = ridge_regression_gradient_descent(simple_feature_matrix, output, initial_weights, step_size, l2_penalty, max_iterations)
print simple_weights_high_penalty


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
QUIZ QUESTIONS
1. What is the value of the coefficient for sqft_living that you learned with no regularization, rounded to 1 decimal place?  What about the one with high regularization?
2. Comparing the lines you fit with the with no regularization versus high regularization, which one is steeper?
3. What are the RSS on the test data for each of the set of weights above (initial, no regularization, high regularization)?
Running a multiple regression with L2 penalty
Let us now consider a model with 2 features: ['sqft_living', 'sqft_living15'].
First, create Numpy versions of your training and test data with these two features.
Can you write Python code for it?

model_features = ['sqft_living', 'sqft_living15'] # sqft_living15 is the average squarefeet for the nearest 15 neighbors.
my_output = 'price'
(feature_matrix, output) = get_numpy_data(train_data, model_features, my_output)
(test_feature_matrix, test_output) = get_numpy_data(test_data, model_features, my_output)


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
This code will plot the two learned models.  (The blue line is for the model with no regularization and the red line is for the one with high regularization.)
Can you write Python code for it?


I am working on the file "Regression/Assignment_four/week-4-ridge-regression-assignment-2-blank.ipynb".
The first task is:
To test your feature derivartive run the following:
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Above: We see that for the "no Center City" version, per unit increase in crime, the predicted decrease in house prices is 2,287.  In contrast, for the original dataset, the drop is only 576 per unit increase in crime.  This is significantly different!
High leverage points:
Center City is said to be a "high leverage" point because it is at an extreme x value where there are not other observations.  As a result, recalling the closed-form solution for simple regression, this point has the potential to dramatically change the least squares line since the center of x mass is heavily influenced by this one point and the least squares line will try to fit close to that outlying (in x) point.  If a high leverage point follows the trend of the other data, this might not have much effect.  On the other hand, if this point somehow differs, it can be strongly influential in the resulting fit.
Influential observations:
An influential observation is one where the removal of the point significantly changes the fit.  As discussed above, high leverage points are good candidates for being influential observations, but need not be.  Other observations that are not leverage points can also be influential observations (e.g., strongly outlying in y even if x is a typical value).
Remove high-value outlier neighborhoods and redo analysis
Based on the discussion above, a question is whether the outlying high-value towns are strongly influencing the fit.  Let's remove them and see what happens.
Can you write Python code for it?

sales_nohighend = sales_noCC[sales_noCC['HousePrice'] < 350000]
crime_model_nohighend = graphlab.linear_regression.create(sales_nohighend, target='HousePrice', features=['CrimeRate'],validation_set=None, verbose=False)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Above: We see that removing the outlying high-value neighborhoods has some effect on the fit, but not nearly as much as our high-leverage Center City datapoint.
Can you write Python code for it?

train_data,test_data = sales.random_split(.8,seed=0)
# input_feature = train_data[‘sqft_living’]
# output = train_data[‘price’]

def simple_linear_regression(input_feature, output):
    input_sum = input_feature.sum()
    output_sum = output.sum()
    N = input_feature.size()
    input_mean = input_sum/N
    output_mean = output_sum/N
    in_out_prod = input_feature * output
    in_out_prod_sum = in_out_prod.sum()
    prod_sum = output_sum * input_sum
    prod_mean = prod_sum/N
    sqr_test = input_feature * input_feature
    sqr_test_sum = sqr_test.sum()
    sqr_sum = input_sum * input_sum
    sqr_mean = sqr_sum/N
    slope = (in_out_prod_sum - prod_mean)/(sqr_test_sum - sqr_mean)
    intercept = output_mean - (input_mean * slope)
    return(intercept, slope)

test_feature = graphlab.SArray(range(5))
test_output = graphlab.SArray(5 + 3*test_feature)
(test_intercept, test_slope) =  simple_linear_regression(test_feature, test_output)
print "Intercept: " + str(test_intercept) # should print 5.0
print "Slope: " + str(test_slope) # should print 3.0

def get_regression_predictions(input_feature, intercept, slope):
    predicted_output = intercept + slope * input_feature
    return(predicted_output)

test_input_feature = 2650

print get_regression_predictions(test_input_feature, 5, 3)

def get_residual_sum_of_squares(input_feature, output, intercept,slope):
    [your code here]
    return(RSS)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Compare coefficients for full-data fit versus no-Center-City fit
Visually, the fit seems different, but let's quantify this by examining the estimated coefficients of our original fit and that of the modified dataset with Center City removed.
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Do the coefficients change much?
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Exploring the data
The house price in a town is correlated with the crime rate of that town. Low crime towns tend to be associated with higher house prices and vice versa.
Can you write Python code for it?

graphlab.canvas.set_target('ipynb')
sales.show(view="Scatter Plot", x="CrimeRate", y="HousePrice")


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Let's see what our fit looks like
Matplotlib is a Python plotting library that is also useful for plotting.  You can install it with:
'pip install matplotlib'
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(sales['CrimeRate'],sales['HousePrice'],'.',
        sales['CrimeRate'],crime_model.predict(sales),'-')


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/PhillyCrime-checkpoint.ipynb".
The first task is:
Look at the fit:
Can you write Python code for it?

plt.plot(sales_noCC['CrimeRate'],sales_noCC['HousePrice'],'.',
        sales_noCC['CrimeRate'],crime_model.predict(sales_noCC),'-')


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
As we see we get the same answer both ways
Can you write Python code for it?

# if we want to multiply every price by 0.5 it's a simple as:
half_prices = 0.5*prices
# Let's compute the sum of squares of price. We can multiply two SArrays of the same length elementwise also with *
prices_squared = prices*prices
sum_prices_squared = prices_squared.sum() # price_squared is an SArray of the squares and we want to add them up.
print "the sum of price squared is: " + str(sum_prices_squared)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Let's test our get_residual_sum_of_squares function by applying it to the test model where the data lie exactly on a line. Since they lie exactly on a line the residual sum of squares should be zero!
Can you write Python code for it?

print get_residual_sum_of_squares(test_feature, test_output, test_intercept, test_slope) # should be 0.0


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Load house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Now that we can calculate a prediction given the slop and intercept let's make a prediction. Use (or alter) the following to find out the estimated price for a house with 2650 squarefeet according to the squarefeet model we estiamted above.
Quiz Question: Using your Slope and Intercept from (4), What is the predicted price for a house with 2650 sqft?
Can you write Python code for it?

my_house_sqft = 2650
estimated_price = get_regression_predictions(my_house_sqft, sqft_intercept, sqft_slope)
print "The estimated price for a house with %d squarefeet is $%.2f" % (my_house_sqft, estimated_price)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Now that we know it works let's build a regression model for predicting price based on sqft_living. Rembember that we train on train_data!
Can you write Python code for it?

sqft_intercept, sqft_slope = simple_linear_regression(train_data['sqft_living'], train_data['price'])

print "Intercept: " + str(sqft_intercept)
print "Slope: " + str(sqft_slope)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Now use your function to calculate the RSS on training data from the squarefeet model calculated above.
Quiz Question: According to this function and the slope and intercept from the squarefeet model What is the RSS for the simple linear regression using squarefeet to predict prices on TRAINING data?
Can you write Python code for it?

rss_prices_on_sqft = get_residual_sum_of_squares(train_data['sqft_living'], train_data['price'], sqft_intercept, sqft_slope)
print 'The RSS of predicting Prices based on Square Feet is : ' + str(rss_prices_on_sqft)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Predict the squarefeet given price
What if we want to predict the squarefoot given the price? Since we have an equation y = a + b*x we can solve the function for x. So that if we have the intercept (a) and the slope (b) and the price (y) we can solve for the estimated squarefeet (x).
Comlplete the following function to compute the inverse regression estimate, i.e. predict the input_feature given the output!
Can you write Python code for it?

def inverse_regression_predictions(output, intercept, slope):
    # solve output = intercept + slope*input_feature for input_feature. Use this equation to compute the inverse predictions:
    estimated_feature = (output - intercept)/slope
    return estimated_feature


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Residual Sum of Squares
Now that we have a model and can make predictions let's evaluate our model using Residual Sum of Squares (RSS). Recall that RSS is the sum of the squares of the residuals and the residuals is just a fancy word for the difference between the predicted output and the true output.
Complete the following (or write your own) function to compute the RSS of a simple linear regression model given the input_feature, output, intercept and slope:
Can you write Python code for it?

import numpy as np
def get_residual_sum_of_squares(input_feature, output, intercept, slope):
    # First get the predictions
    predictions = input_feature.apply(lambda x:get_regression_predictions(x, intercept, slope))
    # then compute the residuals (since we are squaring it doesn't matter which order you subtract)
    diff = np.subtract(predictions,output)
    # square the residuals and add them up
    RSS = np.vdot(diff,diff)
    return(RSS)


I am working on the file "Regression/Assignment_one/.ipynb_checkpoints/week-1-simple-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Useful SFrame summary functions
In order to make use of the closed form soltion as well as take advantage of graphlab's built in functions we will review some important ones. In particular:
* Computing the sum of an SArray
* Computing the arithmetic average (mean) of an SArray
* multiplying SArrays by constants
* multiplying SArrays by other SArrays
Can you write Python code for it?

# Let's compute the mean of the House Prices in King County in 2 different ways.
prices = sales['price'] # extract the price column of the sales SFrame -- this is now an SArray

# recall that the arithmetic average (the mean) is the sum of the prices divided by the total number of houses:
sum_prices = prices.sum()
num_houses = prices.size() # when prices is an SArray .size() returns its length
avg_price_1 = sum_prices/num_houses
avg_price_2 = prices.mean() # if you just want the average, the .mean() function
print "average price via method 1: " + str(avg_price_1)
print "average price via method 2: " + str(avg_price_2)


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
As we see we get the same answer both ways
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
Let's test our get_residual_sum_of_squares function by applying it to the test model where the data lie exactly on a line. Since they lie exactly on a line the residual sum of squares should be zero!
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
New Model: estimate prices from bedrooms
We have made one model for predicting house prices using squarefeet, but there are many other features in the sales SFrame.
Use your simple linear regression function to estimate the regression parameters from predicting Prices based on number of bedrooms. Use the training data!
Can you write Python code for it?

# Estimate the slope and intercept for predicting 'price' based on 'bedrooms'
sqft_bed_intercept, sqft_bed_slope = simple_linear_regression(train_data['bedrooms'], train_data['price'])

print "Intercept: " + str(sqft_bed_intercept)
print "Slope: " + str(sqft_bed_slope)


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
Now use your function to calculate the RSS on training data from the squarefeet model calculated above.
Quiz Question: According to this function and the slope and intercept from the squarefeet model What is the RSS for the simple linear regression using squarefeet to predict prices on TRAINING data?
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
Predicting Values
Now that we have the model parameters: intercept & slope we can make predictions. Using SArrays it's easy to multiply an SArray by a constant and add a constant value. Complete the following function to return the predicted output given the input_feature, slope and intercept:
Can you write Python code for it?

def get_regression_predictions(input_feature, intercept, slope):
    predicted_values = intercept + slope * input_feature
    return predicted_values


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
Residual Sum of Squares
Now that we have a model and can make predictions let's evaluate our model using Residual Sum of Squares (RSS). Recall that RSS is the sum of the squares of the residuals and the residuals is just a fancy word for the difference between the predicted output and the true output.
Complete the following (or write your own) function to compute the RSS of a simple linear regression model given the input_feature, output, intercept and slope:
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
Useful SFrame summary functions
In order to make use of the closed form soltion as well as take advantage of graphlab's built in functions we will review some important ones. In particular:
* Computing the sum of an SArray
* Computing the arithmetic average (mean) of an SArray
* multiplying SArrays by constants
* multiplying SArrays by other SArrays
Can you write Python code for it?


I am working on the file "Regression/Assignment_one/week-1-simple-regression-assignment-blank.ipynb".
The first task is:
We can test that our function works by passing it something where we know the answer. In particular we can generate a feature and then put the output exactly on a line: output = 1 + 1*input_feature then we know both our slope and intercept should be 1
Can you write Python code for it?

test_feature = graphlab.SArray(range(5))
test_output = graphlab.SArray(3 + 5*test_feature)
(test_intercept, test_slope) =  simple_linear_regression(test_feature, test_output)
print "Intercept: " + str(test_intercept)
print "Slope: " + str(test_slope)


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Choosing the best value of k using a validation set
There remains a question of choosing the value of k to use in making predictions. Here, we use a validation set to choose this value. Write a loop that does the following:

For k in [1, 2, ..., 15]:
Makes predictions for each house in the VALIDATION set using the k-nearest neighbors from the TRAINING set.
Computes the RSS for these predictions on the VALIDATION set
Stores the RSS computed above in rss_all

Report which k produced the lowest RSS on VALIDATION set.

(Depending on your computing environment, this computation may take 10-15 minutes.)
To visualize the performance as a function of k, plot the RSS on the VALIDATION set for each considered k value:
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

kvals = range(1, 16)
plt.plot(kvals, rss_all,'bo-')


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Compare this predicted value using 4-nearest neighbors to the predicted value using 1-nearest neighbor computed earlier.
Make multiple predictions
Write a function to predict the value of each and every house in a query set. (The query set can be any subset of the dataset, be it the test set or validation set.) The idea is to have a loop where we take each house in the query set as the query house and make a prediction for that specific house. The new function should take the following parameters:
 * the value of k;
 * the feature matrix for the training houses;
 * the output values (prices) of the training houses; and
 * the feature matrix for the query set.
The function should return a set of predicted values, one for each house in the query set.
Hint: To get the number of houses in the query set, use the .shape field of the query features matrix. See the documentation.
Can you write Python code for it?

def predict_output(k, features_train, output_train, features_query):
    predictions = []
    for i in xrange(len(features_query)):
        prediction = predict_output_of_query(k, features_train, output_train, features_query[i])
        predictions.append(prediction)
    return predictions


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Compute a single distance
To start, let's just explore computing the "distance" between two given houses.  We will take our query house to be the first house of the test set and look at the distance between this house and the 10th house of the training set.
To see the features associated with the query house, print the first row (index 0) of the test feature matrix. You should get an 18-dimensional vector whose components are between 0 and 1.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Compute multiple distances
Of course, to do nearest neighbor regression, we need to compute the distance between our query house and all houses in the training set.
To visualize this nearest-neighbor search, let's first compute the distance from our query house (features_test[0]) to the first 10 houses of the training set (features_train[0:10]) and then search for the nearest neighbor within this small set of houses.  Through restricting ourselves to a small set of houses to begin with, we can visually scan the list of 10 distances to verify that our code for finding the nearest neighbor is working.
Write a loop to compute the Euclidean distance from the query house to each of the first 10 houses in the training set.
Can you write Python code for it?

for i in range(0,10):
    print str(i) + " : " + str(np.sqrt(np.sum((features_train[i]-features_test[0])**2)))


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Extract features and normalize
Using all of the numerical inputs listed in feature_list, transform the training, test, and validation SFrames into Numpy arrays:
Can you write Python code for it?

feature_list = ['bedrooms',
                'bathrooms',
                'sqft_living',
                'sqft_lot',
                'floors',
                'waterfront',
                'view',
                'condition',
                'grade',
                'sqft_above',
                'sqft_basement',
                'yr_built',
                'yr_renovated',
                'lat',
                'long',
                'sqft_living15',
                'sqft_lot15']
features_train, output_train = get_numpy_data(train, feature_list, 'price')
features_test, output_test = get_numpy_data(test, feature_list, 'price')
features_valid, output_valid = get_numpy_data(validation, feature_list, 'price')


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Load in house sales data
For this notebook, we use a subset of the King County housing dataset created by randomly selecting 40% of the houses in the full dataset.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Make a single prediction by averaging k nearest neighbor outputs
Now that we know how to find the k-nearest neighbors, write a function that predicts the value of a given query house. For simplicity, take the average of the prices of the k nearest neighbors in the training set. The function should have the following parameters:
 * the value of k;
 * the feature matrix for the training houses;
 * the output values (prices) of the training houses; and
 * the feature vector of the query house, whose price we are predicting.
The function should return a predicted value of the query house.
Hint: You can extract multiple items from a Numpy array using a list of indices. For instance, output_train[[6, 10]] returns the prices of the 7th and 11th training houses.
Can you write Python code for it?

def predict_output_of_query(k, features_train, output_train, features_query):
    neighbors = k_nearest_neighbors(k, features_train, features_query)
    prices = output_train[neighbors]
    prediction = np.sum(prices)/k
    return prediction


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
Now you are ready to write a function that computes the distances from a query house to all training houses. The function should take two parameters: (i) the matrix of training features and (ii) the single feature vector associated with the query.
Can you write Python code for it?

def compute_distances(features_instances, features_query):
    diff = features_instances[0:len(features_instances)] - features_query
    distances = np.sqrt(np.sum(diff**2, axis=1))
    return distances


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
Among the first 10 training houses, which house is the closest to the query house?
Can you write Python code for it?

for i in range(0,10):
    print str(i) + " : " + str(np.sqrt(np.sum((features_train[i]-features_test[2])**2)))


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
Make predictions for the first 10 houses in the test set using k-nearest neighbors with k=10.

What is the index of the house in this query set that has the lowest predicted value?
What is the predicted value of this house?
Can you write Python code for it?

print predict_output(10, features_train, output_train,features_test[0:10])


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
Take the query house to be third house of the test set (features_test[2]).  What are the indices of the 4 training houses closest to the query house?
Can you write Python code for it?

print k_nearest_neighbors(4, features_train, features_test[2])


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTION
What is the Euclidean distance between the query house and the 10th house of the training set?
Note: Do not use the np.linalg.norm function; use np.sqrt, np.sum, and the power operator (**) instead. The latter approach is more easily adapted to computing multiple distances at once.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
QUIZ QUESTIONS

Take the query house to be third ho
use of the test set (features_test[2]).  What is the index of the house in the training set that is closest to this query house?
What is the predicted value of the query house based on 1-nearest neighbor regression?
Can you write Python code for it?

distances = compute_distances(features_train, features_test[2])
min = distances[0]
index = 0
for i in xrange(len(distances)):
    if(distances[i] < min):
        min = distances[i]
        index = i
print min
print index

print output_train[382]


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
The next step in computing the Euclidean distances is to take these feature-by-feature differences in diff, square each, and take the sum over feature indices.  That is, compute the sum of square feature differences for each training house (row in diff).
By default, np.sum sums up everything in the matrix and returns a single number. To instead sum only over a row or column, we need to specifiy the axis parameter described in the np.sum documentation. In particular, axis=1 computes the sum across each row.
Below, we compute this sum of square feature differences for all training houses and verify that the output for the 16th house in the training set is equivalent to having examined only the 16th row of diff and computing the sum of squares on that row alone.
Can you write Python code for it?

print np.sum(diff**2, axis=1)[15] # take sum of squares across each row, and print the 16th sum
print np.sum(diff[15]**2) # print the sum of squares for the 16th row -- should be same as above


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
The subtraction operator (-) in Numpy is vectorized as follows:
Can you write Python code for it?

print features_train[0:3] - features_test[0]


I am working on the file "Regression/Assignment_six/.ipynb_checkpoints/week-6-local-regression-assignment-blank-checkpoint.ipynb".
The first task is:
With this result in mind, write a single-line expression to compute the Euclidean distances between the query house and all houses in the training set. Assign the result to a variable distances.
Hint: Do not forget to take the square root of the sum of squares.
Can you write Python code for it?

distances = np.sqrt(np.sum(diff**2, axis=1))


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Aside: it is a good idea to write tests like this cell whenever you are vectorizing a complicated operation.
Perform 1-nearest neighbor regression
Now that we have the element-wise differences, it is not too hard to compute the Euclidean distances between our query house and all of the training houses. First, write a single-line expression to define a variable diff such that diff[i] gives the element-wise difference between the features of the query house and the i-th training house.
Can you write Python code for it?

diff = features_train[0:len(features_train)] - features_test[0]


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Choosing the best value of k using a validation set
There remains a question of choosing the value of k to use in making predictions. Here, we use a validation set to choose this value. Write a loop that does the following:

For k in [1, 2, ..., 15]:
Makes predictions for each house in the VALIDATION set using the k-nearest neighbors from the TRAINING set.
Computes the RSS for these predictions on the VALIDATION set
Stores the RSS computed above in rss_all

Report which k produced the lowest RSS on VALIDATION set.

(Depending on your computing environment, this computation may take 10-15 minutes.)
To visualize the performance as a function of k, plot the RSS on the VALIDATION set for each considered k value:
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Compare this predicted value using 4-nearest neighbors to the predicted value using 1-nearest neighbor computed earlier.
Make multiple predictions
Write a function to predict the value of each and every house in a query set. (The query set can be any subset of the dataset, be it the test set or validation set.) The idea is to have a loop where we take each house in the query set as the query house and make a prediction for that specific house. The new function should take the following parameters:
 * the value of k;
 * the feature matrix for the training houses;
 * the output values (prices) of the training houses; and
 * the feature matrix for the query set.
The function should return a set of predicted values, one for each house in the query set.
Hint: To get the number of houses in the query set, use the .shape field of the query features matrix. See the documentation.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Compute multiple distances
Of course, to do nearest neighbor regression, we need to compute the distance between our query house and all houses in the training set.
To visualize this nearest-neighbor search, let's first compute the distance from our query house (features_test[0]) to the first 10 houses of the training set (features_train[0:10]) and then search for the nearest neighbor within this small set of houses.  Through restricting ourselves to a small set of houses to begin with, we can visually scan the list of 10 distances to verify that our code for finding the nearest neighbor is working.
Write a loop to compute the Euclidean distance from the query house to each of the first 10 houses in the training set.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
In computing distances, it is crucial to normalize features. Otherwise, for example, the sqft_living feature (typically on the order of thousands) would exert a much larger influence on distance than the bedrooms feature (typically on the order of ones). We divide each column of the training feature matrix by its 2-norm, so that the transformed column has unit norm.
IMPORTANT: Make sure to store the norms of the features in the training set. The features in the test and validation sets must be divided by these same norms, so that the training, test, and validation sets are normalized consistently.
Can you write Python code for it?

features_train, norms = normalize_features(features_train) # normalize training set features (columns)
features_test = features_test / norms # normalize test set by training set norms
features_valid = features_valid / norms # normalize validation set by training set norms


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Make a single prediction by averaging k nearest neighbor outputs
Now that we know how to find the k-nearest neighbors, write a function that predicts the value of a given query house. For simplicity, take the average of the prices of the k nearest neighbors in the training set. The function should have the following parameters:
 * the value of k;
 * the feature matrix for the training houses;
 * the output values (prices) of the training houses; and
 * the feature vector of the query house, whose price we are predicting.
The function should return a predicted value of the query house.
Hint: You can extract multiple items from a Numpy array using a list of indices. For instance, output_train[[6, 10]] returns the prices of the 7th and 11th training houses.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Note that the output of this vectorized operation is identical to that of the loop above, which can be verified below:
Can you write Python code for it?

# verify that vectorization works
results = features_train[0:3] - features_test[0]
print results[0] - (features_train[0]-features_test[0])
# should print all 0's if results[0] == (features_train[0]-features_test[0])
print results[1] - (features_train[1]-features_test[0])
# should print all 0's if results[1] == (features_train[1]-features_test[0])
print results[2] - (features_train[2]-features_test[0])
# should print all 0's if results[2] == (features_train[2]-features_test[0])


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
Perform k-nearest neighbor regression
For k-nearest neighbors, we need to find a set of k houses in the training set closest to a given query house. We then make predictions based on these k nearest neighbors.
Fetch k-nearest neighbors
Using the functions above, implement a function that takes in
 * the value of k;
 * the feature matrix for the training houses; and
 * the feature vector of the query house
and returns the indices of the k closest training houses. For instance, with 2-nearest neighbor, a return value of [5, 10] would indicate that the 6th and 11th training houses are closest to the query house.
Hint: Look at the documentation for np.argsort.
Can you write Python code for it?

def k_nearest_neighbors(k, feature_train, features_query):
    distances = compute_distances(features_train, features_query)
    neighbors = np.argsort(distances)[0:k]
    return neighbors


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
QUIZ QUESTION
Make predictions for the first 10 houses in the test set using k-nearest neighbors with k=10.

What is the index of the house in this query set that has the lowest predicted value?
What is the predicted value of this house?
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
The next step in computing the Euclidean distances is to take these feature-by-feature differences in diff, square each, and take the sum over feature indices.  That is, compute the sum of square feature differences for each training house (row in diff).
By default, np.sum sums up everything in the matrix and returns a single number. To instead sum only over a row or column, we need to specifiy the axis parameter described in the np.sum documentation. In particular, axis=1 computes the sum across each row.
Below, we compute this sum of square feature differences for all training houses and verify that the output for the 16th house in the training set is equivalent to having examined only the 16th row of diff and computing the sum of squares on that row alone.
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
The subtraction operator (-) in Numpy is vectorized as follows:
Can you write Python code for it?


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
To test the code above, run the following cell, which should output a value -0.0934339605842:
Can you write Python code for it?

print diff[-1].sum() # sum of the feature differences between the query and last training house
# should print -0.0934339605842


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
To test the code above, run the following cell, which should output a value 0.0237082324496:
Can you write Python code for it?

print distances[100] # Euclidean distance between the query house and the 101th training house
# should print 0.0237082324496


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
We will also need the normalize_features() function from Week 5 that normalizes all feature columns to unit norm. Paste this function below.
Can you write Python code for it?

def normalize_features(feature_matrix):
    norms = np.linalg.norm(feature_matrix, axis=0)
    features = feature_matrix / norms
    return features, norms


I am working on the file "Regression/Assignment_six/week-6-local-regression-assignment-blank.ipynb".
The first task is:
With this result in mind, write a single-line expression to compute the Euclidean distances between the query house and all houses in the training set. Assign the result to a variable distances.
Hint: Do not forget to take the square root of the sum of squares.
Can you write Python code for it?