I am working on the file "Regression/assignments/Feature selection and LASSO Programming Assignment 1.ipynb".
The first task is:
2 Phase LASSO for finding a desired number of nonzero features
First we loop through an array of L1 alues and find an interval where our model gets the desired number of nonzero coefficients. In phase 2 we loop once more through the interval (from the first phase) but with small steps, the goal of the second phase is to find the optimal L1 for our nonzero number.
Can you write Python code for it?

max_nonzeros = 7
L1_range = np.logspace(1, 4, num=20)
print (L1_range)

l1_penalty_min = L1_range[0]
l1_penalty_max = L1_range[19]

for L1 in L1_range:
    model = linear_model.Lasso(alpha=L1, normalize=True)
    model.fit(training[all_features], training['price']) # learn weights

    nonZeroes = np.count_nonzero(model.coef_) + np.count_nonzero(model.intercept_)

    # The largest l1_penalty that has more non-zeros than ‘max_nonzeros’
    if (nonZeroes > max_nonzeros) and (L1 > l1_penalty_min) :
        l1_penalty_min = L1

    # The smallest l1_penalty that has fewer non-zeros than ‘max_nonzeros’
    if (nonZeroes < max_nonzeros) and (L1 < l1_penalty_max):
        l1_penalty_max = L1

print ('l1_penalty_min: ', l1_penalty_min)
print ('l1_penalty_max: ', l1_penalty_max)

L1_narrow_interval = np.linspace(l1_penalty_min,l1_penalty_max,20)
print (L1_narrow_interval)

minRSS = sys.maxsize
bestL1 = 0

# Find the model with desired number of nonzeroes with the best RSS
for L1 in L1_narrow_interval:
    model = linear_model.Lasso(alpha=L1, normalize=True)
    model.fit(validation[all_features], validation['price']) # learn weights

    # Calculate the RSS
    RSS = ((model.predict(validation[all_features]) - validation.price) ** 2).sum()
    print ("RSS: ", RSS, " for L1: ", L1)

    # Remember the min RSS
    if RSS < minRSS:
        minRSS = RSS
        bestL1 = L1

print ('\nMinimum RSS:', minRSS, ' for L1: ', bestL1)

model = linear_model.Lasso(alpha=bestL1, normalize=True)
model.fit(validation[all_features], validation['price']) # learn weights

print (all_features)
print ()
print (model.coef_)
print ()
print (model.intercept_)


I am working on the file "Regression/assignments/Feature selection and LASSO Programming Assignment 1.ipynb".
The first task is:
Feature engineering
Can you write Python code for it?

sales['sqft_living_sqrt'] = sales['sqft_living'].apply(sqrt)
sales['sqft_lot_sqrt'] = sales['sqft_lot'].apply(sqrt)
sales['bedrooms_square'] = sales['bedrooms']*sales['bedrooms']
sales['floors_square'] = sales['floors']*sales['floors']

testing['sqft_living_sqrt'] = testing['sqft_living'].apply(sqrt)
testing['sqft_lot_sqrt'] = testing['sqft_lot'].apply(sqrt)
testing['bedrooms_square'] = testing['bedrooms']*testing['bedrooms']
testing['floors_square'] = testing['floors']*testing['floors']

training['sqft_living_sqrt'] = training['sqft_living'].apply(sqrt)
training['sqft_lot_sqrt'] = training['sqft_lot'].apply(sqrt)
training['bedrooms_square'] = training['bedrooms']*training['bedrooms']
training['floors_square'] = training['floors']*training['floors']

validation['sqft_living_sqrt'] = validation['sqft_living'].apply(sqrt)
validation['sqft_lot_sqrt'] = validation['sqft_lot'].apply(sqrt)
validation['bedrooms_square'] = validation['bedrooms']*validation['bedrooms']
validation['floors_square'] = validation['floors']*validation['floors']

all_features = ['bedrooms', 'bedrooms_square',
            'bathrooms',
            'sqft_living', 'sqft_living_sqrt',
            'sqft_lot', 'sqft_lot_sqrt',
            'floors', 'floors_square',
            'waterfront', 'view', 'condition', 'grade',
            'sqft_above',
            'sqft_basement',
            'yr_built', 'yr_renovated']


I am working on the file "Regression/assignments/Feature selection and LASSO Programming Assignment 1.ipynb".
The first task is:
LASSO
Can you write Python code for it?

model_all = linear_model.Lasso(alpha=5e2, normalize=True) # set parameters
model_all.fit(sales[all_features], sales['price']) # learn weights

print (all_features, '\n')
print (model_all.coef_)

print ('With intercetp, nonzeros: ', np.count_nonzero(model_all.coef_) + np.count_nonzero(model_all.intercept_))
print ('No intercept: ', np.count_nonzero(model_all.coef_))

l1pentalities = np.logspace(1, 7, num=13)
l1pentalities

import sys
minRSS = sys.maxsize
bestL1 = 0
for l1p in l1pentalities:
    model = linear_model.Lasso(alpha=l1p, normalize=True)
    model.fit(training[all_features], training['price']) # learn weights

    # Calculate the RSS
    RSS = ((model.predict(validation[all_features]) - validation.price) ** 2).sum()
    print ("RSS: ", RSS, " for L1: ", l1p)

    # Remember the min RSS
    if RSS < minRSS:
        minRSS = RSS
        bestL1 = l1p

print ('\nMinimum RSS:', minRSS, ' for L1: ', bestL1)


I am working on the file "Regression/assignments/Feature selection and LASSO Programming Assignment 1.ipynb".
The first task is:
Load the data
Can you write Python code for it?

dtype_dict = {'bathrooms':float, 'waterfront':int, 'sqft_above':int, 'sqft_living15':float, 'grade':int, 'yr_renovated':int, 'price':float, 'bedrooms':float, 'zipcode':str, 'long':float, 'sqft_lot15':float, 'sqft_living':float, 'floors':float, 'condition':int, 'lat':float, 'date':str, 'sqft_basement':int, 'yr_built':int, 'id':str, 'sqft_lot':int, 'view':int}

sales = pd.read_csv('../datasets/kc_house_data.csv', dtype=dtype_dict)

testing = pd.read_csv('../datasets/wk3_kc_house_test_data.csv', dtype=dtype_dict)
training = pd.read_csv('../datasets/wk3_kc_house_train_data.csv', dtype=dtype_dict)
validation = pd.read_csv('../datasets/wk3_kc_house_valid_data.csv', dtype=dtype_dict)


I am working on the file "Regression/assignments/Ridge Regression Programming Assignment 1.ipynb".
The first task is:
Applying a higher L2 value
Can you write Python code for it?

l2_large_penalty=1.23e2

poly15_data_set1 = polynomial_dataframe(set_1['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model1 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model1.fit(poly15_data_set1, set_1['price'])

poly15_data_set2 = polynomial_dataframe(set_2['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model2 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model2.fit(poly15_data_set2, set_2['price'])

poly15_data_set3 = polynomial_dataframe(set_3['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model3 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model3.fit(poly15_data_set3, set_3['price'])

poly15_data_set4 = polynomial_dataframe(set_4['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model4 = linear_model.Ridge(alpha=l2_large_penalty, normalize=True)
model4.fit(poly15_data_set4, set_4['price'])

plt.plot(poly15_data_set1, model1.predict(poly15_data_set1), poly15_data_set1, set_1['price'])
plt.show()

plt.plot(poly15_data_set2, model2.predict(poly15_data_set2), poly15_data_set2, set_2['price'])
plt.show()

plt.plot(poly15_data_set3, model3.predict(poly15_data_set3), poly15_data_set3, set_3['price'])
plt.show()

plt.plot(poly15_data_set4, model4.predict(poly15_data_set4), poly15_data_set4, set_4['price'])
plt.show()

print('Model 1 coefficients: ', model1.coef_)
print('Model 2 coefficients: ', model2.coef_)
print('Model 3 coefficients: ', model3.coef_)
print('Model 4 coefficients: ', model4.coef_)


I am working on the file "Regression/assignments/Ridge Regression Programming Assignment 1.ipynb".
The first task is:
Data exploration
Can you write Python code for it?

# Show plots in jupyter
%matplotlib inline

sales.head()

sales['price'].head()


I am working on the file "Regression/assignments/Ridge Regression Programming Assignment 1.ipynb".
The first task is:
Helper functions
Can you write Python code for it?

def polynomial_dataframe(feature, degree): # feature is pandas.Series type
    # assume that degree >= 1
    # initialize the dataframe:
    poly_dataframe = pd.DataFrame()
    # and set poly_dataframe['power_1'] equal to the passed feature
    poly_dataframe['power_1'] = feature

    # first check if degree > 1
    if degree > 1:
        # then loop over the remaining degrees:
        for power in range(2, degree+1):
            # first we'll give the column a name:
            name = 'power_' + str(power)
            # assign poly_dataframe[name] to be feature^power; use apply(*)
            poly_dataframe[name] = feature;
            poly_dataframe[name] = poly_dataframe[name].apply(lambda x: x**power)
    return poly_dataframe


I am working on the file "Regression/assignments/Ridge Regression Programming Assignment 1.ipynb".
The first task is:
Load the data
Can you write Python code for it?

dtype_dict = {'bathrooms':float, 'waterfront':int, 'sqft_above':int, 'sqft_living15':float, 'grade':int, 'yr_renovated':int, 'price':float, 'bedrooms':float, 'zipcode':str, 'long':float, 'sqft_lot15':float, 'sqft_living':float, 'floors':float, 'condition':int, 'lat':float, 'date':str, 'sqft_basement':int, 'yr_built':int, 'id':str, 'sqft_lot':int, 'view':int}

regressionDir = '/home/weenkus/workspace/Machine Learning - University of Washington/Regression/datasets/'

sales = pd.read_csv(regressionDir + 'kc_house_data.csv', dtype = dtype_dict)
sales = sales.sort(['sqft_living','price'])

# dtype_dict same as above
set_1 = pd.read_csv(regressionDir + 'wk3_kc_house_set_1_data.csv', dtype=dtype_dict)
set_2 = pd.read_csv(regressionDir + 'wk3_kc_house_set_2_data.csv', dtype=dtype_dict)
set_3 = pd.read_csv(regressionDir + 'wk3_kc_house_set_3_data.csv', dtype=dtype_dict)
set_4 = pd.read_csv(regressionDir + 'wk3_kc_house_set_4_data.csv', dtype=dtype_dict)

train_valid_shuffled = pd.read_csv(regressionDir + 'wk3_kc_house_train_valid_shuffled.csv', dtype=dtype_dict)
test = pd.read_csv(regressionDir + 'wk3_kc_house_test_data.csv', dtype=dtype_dict)
training =  pd.read_csv(regressionDir + 'wk3_kc_house_train_data.csv', dtype=dtype_dict)


I am working on the file "Regression/assignments/Ridge Regression Programming Assignment 1.ipynb".
The first task is:
Ridge regression model fitting
Can you write Python code for it?

poly15_data = polynomial_dataframe(sales['sqft_living'], 15) # use equivalent of `polynomial_sframe`
print(poly15_data)

l2_small_penalty = 1.5e-5
model = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model.fit(poly15_data, sales['price'])

model.coef_

plt.plot(poly15_data, model.predict(poly15_data), poly15_data, sales['price'])
plt.show()


I am working on the file "Regression/assignments/Ridge Regression Programming Assignment 1.ipynb".
The first task is:
Ridge regression on subsets
Using ridge regression with small l2
Can you write Python code for it?

l2_small_penalty=1e-9

poly15_data_set1 = polynomial_dataframe(set_1['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model1 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model1.fit(poly15_data_set1, set_1['price'])

poly15_data_set2 = polynomial_dataframe(set_2['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model2 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model2.fit(poly15_data_set2, set_2['price'])

poly15_data_set3 = polynomial_dataframe(set_3['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model3 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model3.fit(poly15_data_set3, set_3['price'])

poly15_data_set4 = polynomial_dataframe(set_4['sqft_living'], 15) # use equivalent of `polynomial_sframe`
model4 = linear_model.Ridge(alpha=l2_small_penalty, normalize=True)
model4.fit(poly15_data_set4, set_4['price'])

plt.plot(poly15_data_set1, model1.predict(poly15_data_set1), poly15_data_set1, set_1['price'])
plt.show()

plt.plot(poly15_data_set2, model2.predict(poly15_data_set2), poly15_data_set2, set_2['price'])
plt.show()

plt.plot(poly15_data_set3, model3.predict(poly15_data_set3), poly15_data_set3, set_3['price'])
plt.show()

plt.plot(poly15_data_set4, model4.predict(poly15_data_set4), poly15_data_set4, set_4['price'])
plt.show()

print('Model 1 coefficients: ', model1.coef_)
print('Model 2 coefficients: ', model2.coef_)
print('Model 3 coefficients: ', model3.coef_)
print('Model 4 coefficients: ', model4.coef_)


I am working on the file "Regression/assignments/Simple Linear Regression slides.ipynb".
The first task is:
Compare coefficients for full-data fit versus no-Center-City fit¶
Visually, the fit seems different, but let's quantify this by examining the estimated coefficients of our original fit and that of the modified dataset with Center City removed.
Can you write Python code for it?

print ('slope: ', crime_model.coef_)
print ('intercept: ', crime_model.intercept_)

print ('slope: ', crime_model_noCC.coef_)
print ('intercept: ', crime_model_noCC.intercept_)


I am working on the file "Regression/assignments/Simple Linear Regression slides.ipynb".
The first task is:
Exploring the data
The house price in a town is correlated with the crime rate of that town. Low crime towns tend to be associated with higher house prices and vice versa.
Can you write Python code for it?

plt.scatter(sales.CrimeRate, sales.HousePrice, alpha=0.5)
plt.ylabel('House price')
plt.xlabel('Crime rate')


I am working on the file "Regression/assignments/Simple Linear Regression slides.ipynb".
The first task is:
Fit the regression model using crime as the feature
Can you write Python code for it?

# Check the type and shape
X = sales[['CrimeRate']]
print (type(X))
print (X.shape)

y = sales['HousePrice']
print (type(y))
print (y.shape)

crime_model = linear_model.LinearRegression()
crime_model.fit(X, y)


I am working on the file "Regression/assignments/Simple Linear Regression slides.ipynb".
The first task is:
Let's see what our fit looks like
Can you write Python code for it?

plt.plot(sales.CrimeRate, sales.HousePrice, '.',
    X, crime_model.predict(X), '-',
         linewidth=3)
plt.ylabel('House price')
plt.xlabel('Crime rate')


I am working on the file "Regression/assignments/Simple Linear Regression slides.ipynb".
The first task is:
Load house value vs. crime rate data
Dataset is from Philadelphia, PA and includes average house sales price in a number of neighborhoods. The attributes of each neighborhood we have include the crime rate ('CrimeRate'), miles from Center City ('MilesPhila'), town name ('Name'), and county name ('County').
Can you write Python code for it?

regressionDir = '/home/weenkus/workspace/Machine Learning - University of Washington/Regression'
sales = pa.read_csv(regressionDir + '/datasets/Philadelphia_Crime_Rate_noNA.csv')
sales

# Show plots in jupyter
%matplotlib inline


I am working on the file "Regression/assignments/Simple Linear Regression slides.ipynb".
The first task is:
Remove Center City and redo the analysis
Center City is the one observation with an extremely high crime rate, yet house prices are not very low. This point does not follow the trend of the rest of the data very well. A question is how much including Center City is influencing our fit on the other datapoints. Let's remove this datapoint and see what happens.
Can you write Python code for it?

sales_noCC = sales[sales['MilesPhila'] != 0.0]

plt.scatter(sales_noCC.CrimeRate, sales_noCC.HousePrice, alpha=0.5)
plt.ylabel('House price')
plt.xlabel('Crime rate')

crime_model_noCC = linear_model.LinearRegression()
crime_model_noCC.fit(sales_noCC[['CrimeRate']], sales_noCC['HousePrice'])

plt.plot(sales_noCC.CrimeRate, sales_noCC.HousePrice, '.',
    sales_noCC[['CrimeRate']], crime_model_noCC.predict(sales_noCC[['CrimeRate']]), '-',
         linewidth=3)
plt.ylabel('House price')
plt.xlabel('Crime rate')


I am working on the file "Regression/auto-mpg/Auto-MPG.ipynb".
The first task is:
Clean Data (remove NaNs)
Can you write Python code for it?

#Drop nas (labelled as ? in this dataset) -> dropped 6 rows
data_clean=data.applymap(lambda x: np.nan if x == '?' else x).dropna()

data_clean.shape


I am working on the file "Regression/auto-mpg/Auto-MPG.ipynb".
The first task is:
Diagnostic Plot (errors vs. predicted)
Can you write Python code for it?

%pylab inline
#Predicted vs. errors plot -> demonstrates an issue with this fit (high bias)
plt.scatter(regr.predict(X_train), regr.predict(X_train)-y_train)
plt.plot([-5,40],[0,0], color = "red")

#place testing data on the plot as well
plt.scatter(regr.predict(X_test), regr.predict(X_test)-y_test, color = "yellow")


I am working on the file "Regression/auto-mpg/Auto-MPG.ipynb".
The first task is:
Model Evaluation
Can you write Python code for it?

#Mean Squared error and R-squared on the training set
preds = regr.predict(X_train)
mse = np.mean((preds - y_train) ** 2)
rsq = regr.score(X_train, y_train)

print("Mean Squared Error: %.4f \n R-squared: %.4f" % (mse,rsq))

#Test model on held out test set
#Mean Squared error on the testing set
preds_ = regr.predict(X_test)
mse_ = np.mean((preds_ - y_test) ** 2)
rsq_ = regr.score(X_test, y_test)

print("Mean Squared Error: %.4f \n R-squared: %.4f" % (mse_,rsq_))


I am working on the file "Regression/auto-mpg/Auto-MPG.ipynb".
The first task is:
Separate predictors from labels
Can you write Python code for it?

#Select Predictor columns
X = data_clean[['cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'model_year', "origin"]]

#Select target column
y = data_clean['mpg']


I am working on the file "Regression/auto-mpg/Auto-MPG.ipynb".
The first task is:
Split into Training and Testing Sets
Can you write Python code for it?

#Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, test_size=0.2)


I am working on the file "Regression/auto-mpg/Auto-MPG.ipynb".
The first task is:
Train Simple Linear Regression
Can you write Python code for it?

#Train a simple linear regression model
regr = linear_model.LinearRegression()
regr.fit(X_train, y_train)

#Print coefficients
list(zip(names[1:8], regr.coef_))


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
Accessing array elements
Accessing an array is straight forward. For vectors you access the index by referring to it inside square brackets. Recall that indices in Python start with 0.
Can you write Python code for it?


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
Creating Numpy Arrays
New arrays can be made in several ways. We can take an existing list and convert it to a numpy array:
Can you write Python code for it?

mylist = [1., 2., 3., 4.]
mynparray = np.array(mylist)
mynparray


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
For 2D arrays you can select specific columns and specific rows. Passing ':' selects all rows/columns
Can you write Python code for it?

select_cols = np.array([True, False, True]) # 1st and 3rd column
select_rows = np.array([False, True]) # 2nd row

print my_matrix[select_rows, :] # just 2nd row but all columns

print my_matrix[:, select_cols] # all rows and just the 1st and 3rd column


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
Recall that the Euclidean length (or magnitude) of a vector is the squareroot of the sum of the squares of the components. This is just the squareroot of the dot product of the vector with itself:
Can you write Python code for it?

array1_mag = np.sqrt(np.dot(array1, array1))
print array1_mag

print np.sqrt(np.sum(array1*array1))


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
Sequences of indices can be accessed using ':' for example
Can you write Python code for it?

print my_matrix[0:2, 2] # recall 0:2 = [0, 1]

print my_matrix[0, 0:3]


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
The dot product
An important mathematical operation in linear algebra is the dot product.
When we compute the dot product between two vectors we are simply multiplying them elementwise and adding them up. In numpy you can do this with np.dot()
Can you write Python code for it?

array1 = np.array([1., 2., 3., 4.])
array2 = np.array([2., 3., 4., 5.])
print np.dot(array1, array2)

print np.sum(array1*array2)


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
You can also initialize an empty array which will be filled with values. This is the fastest way to initialize a fixed-size numpy array however you must ensure that you replace all of the values.
Can you write Python code for it?

empty_vector = np.empty(5)
print empty_vector


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
You can also pass a list of indices.
Can you write Python code for it?

fib_indices = np.array([1, 1, 2, 3])
random_vector = np.random.random(10) # 10 random numbers between 0 and 1
print random_vector

print random_vector[fib_indices]


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
You can also use true/false values to select values
Can you write Python code for it?

my_vector = np.array([1, 2, 3, 4])
select_index = np.array([True, False, True, False])
print my_vector[select_index]


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
You can compute the sum with np.sum() and the average with np.average()
Can you write Python code for it?

print np.sum(my_array)

print np.average(my_array)

print np.sum(my_array)/len(my_array)


I am working on the file "Regression/examples/.ipynb_checkpoints/numpy-tutorial-checkpoint.ipynb".
The first task is:
You can initialize an array (of any dimension) of all ones or all zeroes with the ones() and zeros() functions:
Can you write Python code for it?

one_vector = np.ones(4)
print one_vector # using print removes the array() portion

one2Darray = np.ones((2, 4)) # an 2D array with 2 "rows" and 4 "columns"
print one2Darray

zero_vector = np.zeros(4)
print zero_vector


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Above: Fit looks pretty wild, too.  Here's a clear example of how overfitting is associated with very large magnitude estimated coefficients.

#
#
Ridge Regression
Ridge regression aims to avoid overfitting by adding a cost to the RSS term of standard least squares that depends on the 2-norm of the coefficients $\|w\|$.  The result is penalizing fits with large coefficients.  The strength of this penalty, and thus the fit vs. model complexity balance, is controled by a parameter lambda (here called "L2_penalty").
Define our function to solve the ridge objective for a polynomial regression model of any degree:
Can you write Python code for it?

def polynomial_ridge_regression(data, deg, l2_penalty):
    model = graphlab.linear_regression.create(polynomial_features(data,deg),
                                              target='Y', l2_penalty=l2_penalty,
                                              validation_set=None,verbose=False)
    return model


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Add random Gaussian noise to y
Can you write Python code for it?

random.seed(1)
e = graphlab.SArray([random.gauss(0,1.0/3.0) for i in range(n)])
y = y + e


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Compute y
Can you write Python code for it?


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Create a function to plot the data, since we'll do it many times
Can you write Python code for it?

def plot_data(data):
    plt.plot(data['X1'],data['Y'],'k.')
    plt.xlabel('x')
    plt.ylabel('y')

plot_data(data)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Create random values for x in interval [0,1)
Can you write Python code for it?

random.seed(98103)
n = 30
x = graphlab.SArray([random.random() for i in range(n)]).sort()


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Define function to plot data and predictions made, since we are going to use it many times.
Can you write Python code for it?

def plot_poly_predictions(data, model):
    plot_data(data)

    # Get the degree of the polynomial
    deg = len(model.coefficients['value'])-1

    # Create 200 points in the x axis and compute the predicted value for each point
    x_pred = graphlab.SFrame({'X1':[i/200.0 for i in range(200)]})
    y_pred = model.predict(polynomial_features(x_pred,deg))

    # plot predictions
    plt.plot(x_pred['X1'], y_pred, 'g-', label='degree ' + str(deg) + ' fit')
    plt.legend(loc='upper left')
    plt.axis([0,1,-1.5,2])


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Explore the lasso solution as a function of a few different penalty strengths
We refer to lambda in the lasso case below as "l1_penalty"
Can you write Python code for it?

for l1_penalty in [0.0001, 0.01, 0.1, 10]:
    model = polynomial_lasso_regression(data, deg=16, l1_penalty=l1_penalty)
    print 'l1_penalty = %e' % l1_penalty
    print 'number of nonzeros = %d' % (model.coefficients['value']).nnz()
    print_coefficients(model)
    print '\n'
    plt.figure()
    plot_poly_predictions(data,model)
    plt.title('LASSO, lambda = %.2e, # nonzeros = %d' % (l1_penalty, (model.coefficients['value']).nnz()))


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Find the value of lambda, $\lambda_{\mathrm{CV}}$, that minimizes the LOO cross validation error, and plot resulting fit
Can you write Python code for it?

best_l2_penalty

model = polynomial_ridge_regression(data, deg=16, l2_penalty=best_l2_penalty)
print_coefficients(model)

plot_poly_predictions(data,model)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Fit a degree-2 polynomial
Fit our degree-2 polynomial to the data generated above:
Can you write Python code for it?

model = polynomial_regression(data, deg=2)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Fit a degree-4 polynomial
Can you write Python code for it?

model = polynomial_regression(data, deg=4)
print_coefficients(model)
plot_poly_predictions(data,model)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Form and plot our predictions along a grid of x values:
Can you write Python code for it?


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Lasso Regression
Lasso regression jointly shrinks coefficients to avoid overfitting, and implicitly performs feature selection by setting some coefficients exactly to 0 for sufficiently large penalty strength lambda (here called "L1_penalty").  In particular, lasso takes the RSS term of standard least squares and adds a 1-norm cost of the coefficients $\|w\|$.
Define our function to solve the lasso objective for a polynomial regression model of any degree:
Can you write Python code for it?

def polynomial_lasso_regression(data, deg, l1_penalty):
    model = graphlab.linear_regression.create(polynomial_features(data,deg),
                                              target='Y', l2_penalty=0.,
                                              l1_penalty=l1_penalty,
                                              validation_set=None,
                                              solver='fista', verbose=False,
                                              max_iterations=3000, convergence_threshold=1e-10)
    return model


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Let's look at fits for a sequence of increasing lambda values
Can you write Python code for it?

for l2_penalty in [1e-25, 1e-10, 1e-6, 1e-3, 1e2]:
    model = polynomial_ridge_regression(data, deg=16, l2_penalty=l2_penalty)
    print 'lambda = %.2e' % l2_penalty
    print_coefficients(model)
    print '\n'
    plt.figure()
    plot_poly_predictions(data,model)
    plt.title('Ridge, lambda = %.2e' % l2_penalty)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Perform a ridge fit of a degree-16 polynomial using a "good" penalty strength
We will learn about cross validation later in this course as a way to select a good value of the tuning parameter (penalty strength) lambda.  Here, we consider "leave one out" (LOO) cross validation, which one can show approximates average mean square error (MSE).  As a result, choosing lambda to minimize the LOO error is equivalent to choosing lambda to minimize an approximation to average MSE.
Can you write Python code for it?

# LOO cross validation -- return the average MSE
def loo(data, deg, l2_penalty_values):
    # Create polynomial features
    polynomial_features(data, deg)

    # Create as many folds for cross validatation as number of data points
    num_folds = len(data)
    folds = graphlab.cross_validation.KFold(data,num_folds)

    # for each value of l2_penalty, fit a model for each fold and compute average MSE
    l2_penalty_mse = []
    min_mse = None
    best_l2_penalty = None
    for l2_penalty in l2_penalty_values:
        next_mse = 0.0
        for train_set, validation_set in folds:
            # train model
            model = graphlab.linear_regression.create(train_set,target='Y',
                                                      l2_penalty=l2_penalty,
                                                      validation_set=None,verbose=False)

            # predict on validation set
            y_test_predicted = model.predict(validation_set)
            # compute squared error
            next_mse += ((y_test_predicted-validation_set['Y'])**2).sum()

        # save squared error in list of MSE for each l2_penalty
        next_mse = next_mse/num_folds
        l2_penalty_mse.append(next_mse)
        if min_mse is None or next_mse < min_mse:
            min_mse = next_mse
            best_l2_penalty = l2_penalty

    return l2_penalty_mse,best_l2_penalty


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Perform a ridge fit of a degree-16 polynomial using a very large penalty strength
Can you write Python code for it?

model = polynomial_ridge_regression(data, deg=16, l2_penalty=100)
print_coefficients(model)

plot_poly_predictions(data,model)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Put data into an SFrame to manipulate later
Can you write Python code for it?

data = graphlab.SFrame({'X1':x,'Y':y})
data


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Run LOO cross validation for "num" values of lambda, on a log scale
Can you write Python code for it?

l2_penalty_values = numpy.logspace(-4, 10, num=10)
l2_penalty_mse,best_l2_penalty = loo(data, 16, l2_penalty_values)


I am working on the file "Regression/examples/Overfitting_Demo_Ridge_Lasso.ipynb".
The first task is:
Woah!!!!  Those coefficients are crazy!  On the order of 10^6.
Can you write Python code for it?


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
2D arrays are accessed similarly by referring to the row and column index separated by a comma:
Can you write Python code for it?

my_matrix = np.array([[1, 2, 3], [4, 5, 6]])
print (my_matrix)

print (my_matrix[1, 2])


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
Accessing array elements
Accessing an array is straight forward. For vectors you access the index by referring to it inside square brackets. Recall that indices in Python start with 0.
Can you write Python code for it?


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
Creating Numpy Arrays
New arrays can be made in several ways. We can take an existing list and convert it to a numpy array:
Can you write Python code for it?

mylist = [1., 2., 3., 4.]
mynparray = np.array(mylist)
mynparray


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
For 2D arrays you can select specific columns and specific rows. Passing ':' selects all rows/columns
Can you write Python code for it?

select_cols = np.array([True, False, True]) # 1st and 3rd column
select_rows = np.array([False, True]) # 2nd row

print (my_matrix[select_rows, :]) # just 2nd row but all columns

print (my_matrix[:, select_cols]) # all rows and just the 1st and 3rd column


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
Operations on Arrays
You can use the operations '*', '**', '\', '+' and '-' on numpy arrays and they operate elementwise.
Can you write Python code for it?

my_array = np.array([1., 2., 3., 4.])
print (my_array*my_array)

print (my_array**2)

print (my_array - np.ones(4))

print (my_array + np.ones(4))

print (my_array / 3)

print (my_array / np.array([2., 3., 4., 5.])) # = [1.0/2.0, 2.0/3.0, 3.0/4.0, 4.0/5.0]


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
Recall that the Euclidean length (or magnitude) of a vector is the squareroot of the sum of the squares of the components. This is just the squareroot of the dot product of the vector with itself:
Can you write Python code for it?

array1_mag = np.sqrt(np.dot(array1, array1))
print (array1_mag)

print (np.sqrt(np.sum(array1*array1)))


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
Sequences of indices can be accessed using ':' for example
Can you write Python code for it?

print (my_matrix[0:2, 2]) # recall 0:2 = [0, 1]

print (my_matrix[0, 0:3])


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
Similarly if you have a vector with the same number of elements as the matrix has rows you can left multiply them.
Can you write Python code for it?

my_matrix = my_features
my_array = np.array([0.3, 0.4, 0.5, 0.6])

print (np.dot(my_array, my_matrix)) # which has 2 elements because my_matrix has 2 columns


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
The dot product
An important mathematical operation in linear algebra is the dot product.
When we compute the dot product between two vectors we are simply multiplying them elementwise and adding them up. In numpy you can do this with np.dot()
Can you write Python code for it?

array1 = np.array([1., 2., 3., 4.])
array2 = np.array([2., 3., 4., 5.])
print (np.dot(array1, array2))

print (np.sum(array1*array2))


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
We can also use the dot product when we have a 2D array (or matrix). When you have an vector with the same number of elements as the matrix (2D array) has columns you can right-multiply the matrix by the vector to get another vector with the same number of elements as the matrix has rows. For example this is how you compute the predicted values given a matrix of features and an array of weights.
Can you write Python code for it?

my_features = np.array([[1., 2.], [3., 4.], [5., 6.], [7., 8.]])
print (my_features)

my_weights = np.array([0.4, 0.5])
print (my_weights)

my_predictions = np.dot(my_features, my_weights) # note that the weights are on the right
print (my_predictions) # which has 4 elements since my_features has 4 rows(


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
You can also pass a list of indices.
Can you write Python code for it?

fib_indices = np.array([1, 1, 2, 3])
random_vector = np.random.random(10) # 10 random numbers between 0 and 1
print (random_vector)

print (random_vector[fib_indices])


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
You can also use true/false values to select values
Can you write Python code for it?

my_vector = np.array([1, 2, 3, 4])
select_index = np.array([True, False, True, False])
print (my_vector[select_index])


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
You can compute the sum with np.sum() and the average with np.average()
Can you write Python code for it?

print (np.sum(my_array))

print (np.average(my_array))

print (np.sum(my_array)/len(my_array))


I am working on the file "Regression/examples/numpy-tutorial.ipynb".
The first task is:
You can initialize an array (of any dimension) of all ones or all zeroes with the ones() and zeros() functions:
Can you write Python code for it?

one_vector = np.ones(4)
print (one_vector) # using print removes the array() portion

one2Darray = np.ones((2, 4)) # an 2D array with 2 "rows" and 4 "columns"
print (one2Darray)

zero_vector = np.zeros(4)
print (zero_vector)


I am working on the file "Regression/examples/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Compute RSS
Now that we can make predictions given the model, let's write a function to compute the RSS of the model. Complete the function below to calculate RSS given the model, data, and the outcome.
Can you write Python code for it?

def get_residual_sum_of_squares(model, data, outcome):
    # First get the predictions

    # Then compute the residuals/errors

    # Then square and add them up

    return(RSS)


I am working on the file "Regression/examples/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Create some new features
Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms.
You will use the logarithm function to create a new feature. so first you should import it from the math library.
Can you write Python code for it?


I am working on the file "Regression/examples/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Load in house sales data
Dataset is from house sales in King County, the region where the city of Seattle, WA is located.
Can you write Python code for it?


I am working on the file "Regression/examples/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Next create the following 4 new features as column in both TEST and TRAIN data:
* bedrooms_squared = bedrooms*bedrooms
* bed_bath_rooms = bedrooms*bathrooms
* log_sqft_living = log(sqft_living)
* lat_plus_long = lat + long
As an example here's the first one:
Can you write Python code for it?

train_data['bedrooms_squared'] = train_data['bedrooms'].apply(lambda x: x**2)
test_data['bedrooms_squared'] = test_data['bedrooms'].apply(lambda x: x**2)

# create the remaining 3 features in both TEST and TRAIN data


I am working on the file "Regression/examples/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 1?
Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 2?
Think about what this means.
Comparing multiple models
Now that you've learned three models and extracted the model weights we want to evaluate which model is best.
First use your functions from earlier to compute the RSS on TRAINING Data for each of the three models.
Can you write Python code for it?

# Compute the RSS on TRAINING data for each of the three models and record the values:


I am working on the file "Regression/examples/week-2-multiple-regression-assignment-1-blank.ipynb".
The first task is:
Test your function by computing the RSS on TEST data for the example model:
Can you write Python code for it?

rss_example_train = get_residual_sum_of_squares(example_model, test_data, test_data['price'])
print rss_example_train # should be 2.7376153833e+14


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Let's start with a degree 1 polynomial using 'sqft_living' (i.e. a line) to predict 'price' and plot what it looks like.
Can you write Python code for it?

poly1_data = polynomial_sframe(sales['sqft_living'], 1)
poly1_data['price'] = sales['price'] # add price to the data since it's the target


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Let's unpack that plt.plot() command. The first pair of SArrays we passed are the 1st power of sqft and the actual price we then ask it to print these as dots '.'. The next pair we pass is the 1st power of sqft and the predicted values from the linear model. We ask these to be plotted as a line '-'.
We can see, not surprisingly, that the predicted values all fall on a line, specifically the one with slope 280 and intercept -43579. What if we wanted to plot a second degree polynomial?
Can you write Python code for it?

poly2_data = polynomial_sframe(sales['sqft_living'], 2)
my_features = poly2_data.column_names() # get the name of the features
poly2_data['price'] = sales['price'] # add price to the data since it's the target
model2 = graphlab.linear_regression.create(poly2_data, target = 'price', features = my_features, validation_set = None)

model2.get("coefficients")

plt.plot(poly2_data['power_1'],poly2_data['price'],'.',
        poly2_data['power_1'], model2.predict(poly2_data),'-')


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
NOTE: for all the models in this notebook use validation_set = None to ensure that all results are consistent across users.
Can you write Python code for it?

model1 = graphlab.linear_regression.create(poly1_data, target = 'price', features = ['power_1'], validation_set = None)

#let's take a look at the weights before we plot
model1.get("coefficients")

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(poly1_data['power_1'],poly1_data['price'],'.',
        poly1_data['power_1'], model1.predict(poly1_data),'-')


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Next we're going to write a polynomial function that takes an SArray and a maximal degree and returns an SFrame with columns containing the SArray to all the powers up to the maximal degree.
The easiest way to apply a power to an SArray is to use the .apply() and lambda x: functions.
For example to take the example array and compute the third power we can do as follows: (note running this cell the first time may take longer than expected since it loads graphlab)
Can you write Python code for it?

tmp = graphlab.SArray([1., 2., 3.])
tmp_cubed = tmp.apply(lambda x: x**3)
print tmp
print tmp_cubed


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
Polynomial_sframe function
Using the hints above complete the following function to create an SFrame consisting of the powers of an SArray up to a specific degree:
Can you write Python code for it?

def polynomial_sframe(feature, degree):
    # assume that degree >= 1
    # initialize the SFrame:
    poly_sframe = graphlab.SFrame()
    # and set poly_sframe['power_1'] equal to the passed feature

    # first check if degree > 1
    if degree > 1:
        # then loop over the remaining degrees:
        # range usually starts at 0 and stops at the endpoint-1. We want it to start at 2 and stop at degree
        for power in range(2, degree+1):
            # first we'll give the column a name:
            name = 'power_' + str(power)
            # then assign poly_sframe[name] to the appropriate power of feature

    return poly_sframe


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
To test your function consider the smaller tmp variable and what you would expect the outcome of the following call:
Can you write Python code for it?


I am working on the file "Regression/examples/week-3-polynomial-regression-assignment-blank.ipynb".
The first task is:
We can create an empty SFrame using graphlab.SFrame() and then add any columns to it with ex_sframe['column_name'] = value. For example we create an empty SFrame and make the column 'power_1' to be the first power of tmp (i.e. tmp itself).
Can you write Python code for it?

ex_sframe = graphlab.SFrame()
ex_sframe['power_1'] = tmp
print ex_sframe


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Extract the remainder of the data after excluding fourth segment (segment 3) and assign the subset to train4.
To verify that we have the right elements extracted, run the following cell, which computes the average price of the data with fourth segment excluded. When rounded to nearest whole number, the average should be $539,450.
Can you write Python code for it?


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Let us familiarize ourselves with array slicing with SFrame. To extract a continuous slice from an SFrame, use colon in square brackets. For instance, the following cell extracts rows 0 to 9 of train_valid_shuffled. Notice that the first index (0) is included in the slice but the last index (10) is omitted.
Can you write Python code for it?

train_valid_shuffled[0:10] # rows 0 to 9


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Let us revisit the 15th-order polynomial model using the 'sqft_living' input. Generate polynomial features up to degree 15 using polynomial_sframe() and fit a model with these features. When fitting the model, use an L2 penalty of 1e-5:
Can you write Python code for it?


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Let's use matplotlib to visualize what a polynomial regression looks like on the house data.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

sales = graphlab.SFrame('kc_house_data.gl/')


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Note: When we have so many features and so few data points, the solution can become highly numerically unstable, which can sometimes lead to strange unpredictable results.  Thus, rather than using no regularization, we will introduce a tiny amount of regularization (l2_penalty=1e-5) to make the solution numerically stable.  (In lecture, we discussed the fact that regularization can also help with numerical stability, and here we are seeing a practical example.)
With the L2 penalty specified above, fit the model and print out the learned weights.
Hint: make sure to add 'price' column to the new SFrame before calling graphlab.linear_regression.create(). Also, make sure GraphLab Create doesn't create its own validation set by using the option validation_set=None in this call.
QUIZ QUESTION:  What's the learned value for the coefficient of feature power_1?
Observe overfitting
Recall from Week 3 that the polynomial fit of degree 15 changed wildly whenever the data changed. In particular, when we split the sales data into four subsets and fit the model of degree 15, the result came out to be very different for each subset. The model had a high variance. We will see in a moment that ridge regression reduces such variance. But first, we must reproduce the experiment we did in Week 3.
First, split the data into split the sales data into four subsets of roughly equal size and call them set_1, set_2, set_3, and set_4. Use .random_split function and make sure you set seed=0.
Can you write Python code for it?

(semi_split1, semi_split2) = sales.random_split(.5,seed=0)
(set_1, set_2) = semi_split1.random_split(0.5, seed=0)
(set_3, set_4) = semi_split2.random_split(0.5, seed=0)


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Now let us extract individual segments with array slicing. Consider the scenario where we group the houses in the train_valid_shuffled dataframe into k=10 segments of roughly equal size, with starting and ending indices computed as above.
Extract the fourth segment (segment 3) and assign it to a variable called validation4.
To verify that we have the right elements extracted, run the following cell, which computes the average price of the fourth segment. When rounded to nearest whole number, the average should be $536,234.
Can you write Python code for it?


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Now we are ready to implement k-fold cross-validation. Write a function that computes k validation errors by designating each of the k segments as the validation set. It accepts as parameters (i) k, (ii) l2_penalty, (iii) dataframe, (iv) name of output column (e.g. price) and (v) list of feature names. The function returns the average validation error using k segments as validation sets.

For each i in [0, 1, ..., k-1]:
Compute starting and ending indices of segment i and call 'start' and 'end'
Form validation set by taking a slice (start:end+1) from the data.
Form training set by appending slice (end+1:n) to the end of slice (0:start).
Train a linear model using training set just formed, with a given l2_penalty
Compute validation error using validation set just formed
Can you write Python code for it?

def k_fold_cross_validation(k, l2_penalty, data, output_name, features_list):


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Once the data is shuffled, we divide it into equal segments. Each segment should receive n/k elements, where n is the number of observations in the training set and k is the number of segments. Since the segment 0 starts at index 0 and contains n/k elements, it ends at index (n/k)-1. The segment 1 starts where the segment 0 left off, at index (n/k). With n/k elements, the segment 1 ends at index (n*2/k)-1. Continuing in this fashion, we deduce that the segment i starts at index (n*i/k) and ends at (n*(i+1)/k)-1.
With this pattern in mind, we write a short loop that prints the starting and ending indices of each segment, just to make sure you are getting the splits right.
Can you write Python code for it?

n = len(train_valid_shuffled)
k = 10 # 10-fold cross-validation

for i in xrange(k):
    start = (n*i)/k
    end = (n*(i+1))/k-1
    print i, (start, end)


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Once we have a function to compute the average validation error for a model, we can write a loop to find the model that minimizes the average validation error. Write a loop that does the following:
* We will again be aiming to fit a 15th-order polynomial model using the sqft_living input
* For l2_penalty in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, you can use this Numpy function: np.logspace(1, 7, num=13).)
    * Run 10-fold cross-validation with l2_penalty
* Report which L2 penalty produced the lowest average validation error.
Note: since the degree of the polynomial is now fixed to 15, to make things faster, you should generate polynomial features in advance and re-use them throughout the loop. Make sure to use train_valid_shuffled when generating polynomial features!
QUIZ QUESTIONS:  What is the best value for the L2 penalty according to 10-fold validation?
You may find it useful to plot the k-fold cross-validation errors you have obtained to better understand the behavior of the method.
Can you write Python code for it?

# Plot the l2_penalty values in the x axis and the cross-validation error in the y axis.
# Using plt.xscale('log') will make your plot more intuitive.


I am working on the file "Regression/examples/week-4-ridge-regression-assignment-1-blank.ipynb".
The first task is:
Polynomial regression, revisited
We build on the material from Week 3, where we wrote the function to produce an SFrame with columns containing the powers of a given input. Copy and paste the function polynomial_sframe from Week 3:
Can you write Python code for it?


I am working on the file "Regression/intro_regression.ipynb".
The first task is:
Batting
Summary Stats
Can you write Python code for it?


I am working on the file "Regression/intro_regression.ipynb".
The first task is:
EDA
Function to print data info and summary stats
Can you write Python code for it?

def summary(data):
    # data info
    print('DATA INFO \n')
    data.info()
    print(50*'-', '\n')
    # numeric summary
    print('NUMERIC \n')
    print(data.describe().T)
    print(50*'-', '\n')
    # categorical summary
    print('CATEGORICAL \n')
    print(data.describe(include=['O']).T)

# import pandas scatter_matrix to create pair-plots
from pandas.plotting import scatter_matrix


I am working on the file "Regression/intro_regression.ipynb".
The first task is:
Read data files to dict
Can you write Python code for it?

sets = ['batting',
        'pitching',
        'player',
        'salary']
data = {}
for s in sets:
    file = s + '.csv'
    data[s] = pd.read_csv(file)


I am working on the file "RegressionAnalysisWithPython/Chap_6 - Achieving Generalization.ipynb".
The first task is:
Cross validation
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score, KFold, StratifiedKFold
from sklearn.metrics import make_scorer
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np

def RMSE(y_true, y_pred):
    return np.sum((y_true -y_pred)**2)

lm = LinearRegression()
cv_iterator = KFold(n=len(X), n_folds=10, shuffle=True, random_state=101)

edges = np.histogram(y, bins=5)[1]
binning = np.digitize(y, edges)
stratified_cv_iterator = StratifiedKFold(binning, n_folds=10, shuffle=True, random_state=101)

second_order=PolynomialFeatures(degree=2, interaction_only=False)
third_order=PolynomialFeatures(degree=3, interaction_only=True)

over_param_X = second_order.fit_transform(X)
extra_over_param_X = third_order.fit_transform(X)
cv_score = cross_val_score(lm, over_param_X, y, cv=cv_iterator, scoring='mean_squared_error', n_jobs=1)

print (cv_score)

print ('Cv score: mean %0.3f std %0.3f' % (np.mean(np.abs(cv_score)), np.std(cv_score)))

cv_score = cross_val_score(lm, over_param_X, y, cv=stratified_cv_iterator, scoring='mean_squared_error', n_jobs=1)
print ('Cv score: mean %0.3f std %0.3f' % (np.mean(np.abs(cv_score)), np.std(cv_score)))


I am working on the file "RegressionAnalysisWithPython/Chap_6 - Achieving Generalization.ipynb".
The first task is:
Lasso
Can you write Python code for it?

from sklearn.linear_model import Lasso
lasso = Lasso(alpha=1.0, normalize=True, max_iter=2*10**5)
#The following comment shows an example of L1 logistic regression
#lr_l1 = LogisticRegression(C=1.0, penalty='l1', tol=0.01)

from sklearn.grid_search import RandomizedSearchCV
from scipy.stats import expon
np.random.seed(101)
stratified_cv_iterator = StratifiedKFold(binning, n_folds=10, shuffle=True, random_state=101)
search_func=RandomizedSearchCV(estimator=lasso, param_distributions={'alpha':np.logspace(-5,2,100)}, n_iter=10,
                               scoring='mean_squared_error', n_jobs=1, iid=False, refit=True, cv=stratified_cv_iterator)
search_func.fit(second_order.fit_transform(X), y)
print ('Best alpha: %0.5f' % search_func.best_params_['alpha'])
print ('Best CV mean squared error: %0.3f' % np.abs(search_func.best_score_))

 print ('Zero value coefficients: %i out of %i' % (np.sum(~(search_func.best_estimator_.coef_==0.0)),
                                                 len(search_func.best_estimator_.coef_)))

# Alternative: sklearn.linear_model.LassoCV
# Execution time: 54.9 s
from sklearn.linear_model import LassoCV
auto_lasso = LassoCV(alphas=np.logspace(-5,2,100), normalize=True, n_jobs=1, cv=None, max_iter=10**6)
auto_lasso.fit(second_order.fit_transform(X), y)
print ('Best alpha: %0.5f' % auto_lasso.alpha_)


I am working on the file "RegressionAnalysisWithPython/Chap_6 - Achieving Generalization.ipynb".
The first task is:
Madelon dataset
Can you write Python code for it?

try:
    import urllib.request as urllib2
except:
    import urllib2
import numpy as np
train_data = 'https://archive.ics.uci.edu/ml/machine-learning-databases/madelon/MADELON/madelon_train.data'
validation_data = 'https://archive.ics.uci.edu/ml/machine-learning-databases/madelon/MADELON/madelon_valid.data'
train_response = 'https://archive.ics.uci.edu/ml/machine-learning-databases/madelon/MADELON/madelon_train.labels'
validation_response = 'https://archive.ics.uci.edu/ml/machine-learning-databases/madelon/madelon_valid.labels'
try:
    Xt = np.loadtxt(urllib2.urlopen(train_data))
    yt = np.loadtxt(urllib2.urlopen(train_response))
    Xv = np.loadtxt(urllib2.urlopen(validation_data))
    yv = np.loadtxt(urllib2.urlopen(validation_response))
except:
    # In case downloading the data doesn't works,
    # just manually download the files into the working directory
    Xt = np.loadtxt('madelon_train.data')
    yt = np.loadtxt('madelon_train.labels')
    Xv = np.loadtxt('madelon_valid.data')
    yv = np.loadtxt('madelon_valid.labels')

print ('Training set: %i observations %i feature' % (Xt.shape))
print ('Validation set: %i observations %i feature' % (Xv.shape))

from scipy.stats import describe
print (describe(Xt))

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline

def visualize_correlation_matrix(data, hurdle = 0.0):
    R = np.corrcoef(data, rowvar=0)
    R[np.where(np.abs(R)<hurdle)] = 0.0
    heatmap = plt.pcolor(R, cmap=mpl.cm.coolwarm, alpha=0.8)
    heatmap.axes.set_frame_on(False)
    plt.xticks(rotation=90)
    plt.tick_params(axis='both', which='both', bottom='off', top='off', left = 'off',
    right = 'off')
    plt.colorbar()
    plt.show()

visualize_correlation_matrix(Xt[:,100:150], hurdle=0.0)

from sklearn.cross_validation import cross_val_score
from sklearn.linear_model import LogisticRegression

logit = LogisticRegression()

logit.fit(Xt,yt)

from sklearn.metrics import roc_auc_score
print ('Training area under the curve: %0.3f' %...


I am working on the file "RegressionAnalysisWithPython/Chap_6 - Achieving Generalization.ipynb".
The first task is:
Random Search
Can you write Python code for it?

from sklearn.grid_search import RandomizedSearchCV
from scipy.stats import expon
np.random.seed(101)
search_func=RandomizedSearchCV(estimator=ridge, param_distributions={'alpha':np.logspace(-4,2,100)}, n_iter=10,
                               scoring='mean_squared_error', n_jobs=1, iid=False, refit=True, cv=stratified_cv_iterator)
search_func.fit(second_order.fit_transform(X), y)
print ('Best alpha: %0.5f' % search_func.best_params_['alpha'])
print ('Best CV mean squared error: %0.3f' % np.abs(search_func.best_score_))


I am working on the file "RegressionAnalysisWithPython/Chap_6 - Achieving Generalization.ipynb".
The first task is:
Regularization
Ridge
Can you write Python code for it?

from sklearn.linear_model import Ridge
ridge = Ridge(normalize=True)
# The following commented line is to show a logistic regression with L2 regularization
# lr_l2 = LogisticRegression(C=1.0, penalty='l2', tol=0.01)
ridge.fit(second_order.fit_transform(X), y)

lm.fit(second_order.fit_transform(X), y)

print ('Average coefficient: Non regularized = %0.3f Ridge = %0.3f' % (np.mean(lm.coef_), np.mean(ridge.coef_)))
print ('Min coefficient: Non regularized = %0.3f Ridge = %0.3f' % (np.min(lm.coef_), np.min(ridge.coef_)))
print ('Max coefficient: Non regularized = %0.3f Ridge = %0.3f' % (np.max(lm.coef_), np.max(ridge.coef_)))


I am working on the file "RegressionAnalysisWithPython/Chap_6 - Achieving Generalization.ipynb".
The first task is:
Stability selection
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score
from sklearn.linear_model import RandomizedLogisticRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

threshold = 0.03
stability_selection = RandomizedLogisticRegression(n_resampling=300, n_jobs=1, random_state=101, scaling=0.15,
                                                   sample_fraction=0.50, selection_threshold=threshold)
interactions = PolynomialFeatures(degree=4, interaction_only=True)
model = make_pipeline(stability_selection, interactions, logit)
model.fit(Xt,yt)

print ('Number of features picked by stability selection: %i' % np.sum(model.steps[0][1].all_scores_ >= threshold))

from sklearn.metrics import roc_auc_score
print ('Area Under the Curve: %0.3f' % roc_auc_score(yv,model.predict_proba(Xv)[:,1]))


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Bootstrap
Bootstrap some bitcoins so we can spend them.  In regtest, you use generate and mine 101 blocks rewarding you with 50 bitcoins in a coinbase transaction.  On testnet, you use a faucet to get some bitcoins.  On mainnet, you need actual bitcoins.  We will stick with regtest for now.
Can you write Python code for it?

## N.B. our balance is zero in the genesis block
print 'Initial balance, before any mining ' + str(Bob.getbalance())

Bob.generate(101)

print 'Balance after mining 101 blocks ' + str(Bob.getbalance())


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Connect nodes to bitcoin-rpc module
We hard-code the username and passwords here for pedagogical ease.  We display the blockchain information using the widely used, but deprecated getinfo() RPC.
getinfo() returns:
version - The version number of this bitcoin-qt or bitcoind program itself. Both of are equivalent. -qt is simply the graphical user interface version
protocolversion: The version of the bitcoin network protocol supported by this client (user agent software).
walletversion: The version of the wallet.dat file. Wallet.dat contains bitcoin addresses and public & private key pairs for these addresses. There is additional data on the wallet. Care must be taken to not restore from an old wallet backup. New addresses generated in the wallet since the old backup was made will not exist in the old backup! Source
balance: The total number of bitcoins held in the wallet.dat file.
blocks: The total number of blocks which constitute the shared block chain.
timeoffset: Seconds of difference between this node's "wall clock time" and the median reported by our network peers.
connections: the number of peers on the bitcoin P2P network that this node is connected to.
proxy: If using a proxy to connect to the network, listed here, otherwise blank.
difficulty: the current mining difficulty factor. Difficulty is increased as more miners and more hash compute power compete to be the next one to have a block of transactions added to the blockchain.
testnet: Boolean value (true OR false). There is a parallel bitcoin network, the testnet, where trials and experiments may be carried out without impacting the official, live bitcoin P2P network
keypoololdest: timestamp (UNIX epoch) of the oldest key in the keypool
keypoolsize: A number of addresses are kept in reserve by the client. This is the size of that reserve.
paytxfee: Specifies what fee the client is willing to pay to expedite transactions. Miners may choose to ignore transactions that do not pay a fee, and these fee-less transactions will have low priority on queue of...
Can you write Python code for it?

bitcoin.SelectParams('regtest')

Bob  = bitcoin.rpc.RawProxy("http://bitcoinrpc:P0@127.0.0.1:16591")
Mary = bitcoin.rpc.RawProxy("http://bitcoinrpc:P0@127.0.0.1:16592")

info = Bob.getinfo()
for key in info.keys():
    print key + ' : ' + str(info[key])


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Import the python bitcoin module created by Peter Todd.
Can you write Python code for it?

import bitcoin
import bitcoin.rpc

import bitcoin.core
import bitcoin.core.script


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Let's create an address and send some bitcoins there.  Some things to notice:

We set up a new proxy, using a different RPC port on regtest
Address is NOT on blockchain, it is just a 32 byte hash that is almost certainly unique and never before used.
We send some coins to new address, note the transaction structure
The mempool shows the pending transaction
We need to mine to write the transaction to the blockchain
Can you write Python code for it?

print 'Mary\'s balance = ' + str(Mary.getbalance())
#print 'Mary\'s peers: '
#print Mary.getpeerinfo()

getnewaddress = Mary.getnewaddress()
print '\nNew address ' + str(getnewaddress)
print '\nMary\'s address has received how many BTC? ' +\
str(Mary.getreceivedbyaddress(getnewaddress,0))

##have Bob (proxy) send 25 bitcoins to Mary
txid = Bob.sendtoaddress(getnewaddress,25)

getmempoolinfo = Bob.getmempoolinfo()
getrawmempool = Bob.getrawmempool(True)
print '\ngetmempoolinfo ' + str(getmempoolinfo)
print '\ngetrawmempool'
print getrawmempool
print '\n'
for key in getrawmempool.keys():
    for field in getrawmempool[key].keys():
        print str(field) + ' : ' + str(getrawmempool[key][field])
#print '\ntxid from sendtoaddress output ' + str(txid)
print '\nIs the send txid the same as memory pool txid? ****' +\
    str(txid == getrawmempool.keys()[0]) + '****'

print '\nMary\'s balance before mining = ' + str(Mary.getbalance())
print 'Bob\'s balance before mining = ' + str(Bob.getbalance())

##how can I see transaction details before mining?
print '\nMemory Pool Raw Transaction Data\n'
import pprint
pprint.pprint(Bob.getrawtransaction(txid,1))

##N.B. no transaction on the blockchain yet!!!


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Raw Transaction RPCs

CreateRawTransaction: creates an unsigned serialized transaction that spends a previous output to a new output with a P2PKH or P2SH address. The transaction is not stored in the wallet or transmitted to the network.
DecodeRawTransaction: decodes a serialized transaction hex string into a JSON object describing the transaction.
DecodeScript: decodes a hex-encoded P2SH redeem script.
GetRawTransaction: gets a hex-encoded serialized transaction or a JSON object describing the transaction. By default, Bitcoin Core only stores complete transaction data for UTXOs and your own transactions, so the RPC may fail on historic transactions unless you use the non-default txindex=1 in your Bitcoin Core startup settings.
SendRawTransaction: validates a transaction and broadcasts it to the peer-to-peer network.
SignRawTransaction: signs a transaction in the serialized transaction format using private keys stored in the wallet or provided in the call.

Generate some bitcoins with a mining start
Now let's look at the block structure, rerunning code from above.  Note that the block height is now 101.  There is one transaction for 50BTC.  Note that this is a coinbase transaction with just a vout and no vin.
How to make serialized transaction?
Can you write Python code for it?

getblockcount = Bob.getblockcount()
print '\ngetblockcount = ' + str(getblockcount)

getblockhash = Bob.getblockhash(getblockcount)
print '\ngetblockhash = ' + str(getblockhash)

print '\ngetblock\n'
getblock = Bob.getblock(getblockhash)
print getblock

tx = getblock['tx']
print '\n' + str(len(tx)) + ' Transactions\n'
print tx

for i in range(len(tx)):
    print '\nSerialized Transaction #' + str(i) +'\n'
    serializedTX = Bob.getrawtransaction(tx[i],0)
    print serializedTX
    print '\nRaw Transaction\n'
    rawTX = Bob.getrawtransaction(tx[i],1)
    print rawTX
    print '\nDecoded Transaction\n'
    decodedTX    = Bob.decoderawtransaction(serializedTX)
    print decodedTX


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Wallet Programs
Permitting receiving and spending of satoshis is the only essential feature of wallet software—but a particular wallet program doesn’t need to do both things. Two wallet programs can work together, one program distributing public keys in order to receive satoshis and another program signing transactions spending those satoshis.
Wallet programs also need to interact with the peer-to-peer network to get information from the block chain and to broadcast new transactions. However, the programs which distribute public keys or sign transactions don’t need to interact with the peer-to-peer network themselves.
This leaves us with three necessary, but separable, parts of a wallet system: a public key distribution program, a signing program, and a networked program. In the subsections below, we will describe common combinations of these parts.
Full-Service Wallets
The simplest wallet is a program which performs all three functions: it generates private keys, derives the corresponding public keys, helps distribute those public keys as necessary, monitors for outputs spent to those public keys, creates and signs transactions spending those outputs, and broadcasts the signed transactions.

Full-Service Wallets
As of this writing, almost all popular wallets can be used as full-service wallets.
The main advantage of full-service wallets is that they are easy to use. A single program does everything the user needs to receive and spend satoshis.
The main disadvantage of full-service wallets is that they store the private keys on a device connected to the Internet. The compromise of such devices is a common occurrence, and an Internet connection makes it easy to transmit private keys from a compromised device to an attacker.
Utility RPCs

CreateMultiSig: creates a P2SH multi-signature address.
EstimateFee: estimates the transaction fee per kilobyte that needs to be paid for a transaction to be included within a certain number of blocks. New in 0.10.0
EstimatePriority: estimates the priority that a transaction...
Can you write Python code for it?

# Eloipool - Python Bitcoin pool server
# Copyright (C) 2011-2012  Luke Dashjr <luke-jr+eloipool@utopios.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# original code https://github.com/Crypto-Expert/stratum-mining/blob/master/lib/merkletree.py
# http://code.runnable.com/U3jqtyYUmAUxtsSS/bitcoin-block-merkle-root-python

from hashlib import sha256
from util import doublesha

class MerkleTree:
    def __init__(self, data, detailed=False):
        self.data = data
        self.recalculate(detailed)
        self._hash_steps = None

    def recalculate(self, detailed=False):
        L = self.data
        steps = []
        if detailed:
            detail = []
            PreL = []
            StartL = 0
        else:
            detail = None
            PreL = [None]
            StartL = 2
        Ll = len(L)
        if detailed or Ll > 1:
            while True:
                if detailed:
                    detail += L
                if Ll == 1:
                    break
                steps.append(L[1])
                if Ll % 2:
                    L += [L[-1]]
                L = PreL + [doublesha(L[i] + L[i + 1]) for i in range(StartL, Ll, 2)]
                Ll = len(L)
        self._steps = steps
        self.detail = detail

    def hash_steps(self):
        if self._hash_steps == None:
            self._hash_steps = doublesha(''.join(self._steps))
        return self._hash_steps
      ...


I am working on the file "Regtest_RPC.ipynb".
The first task is:
Wallet RPCs
Note: the wallet RPCs are only available if Bitcoin Core was built with wallet support, which is the default.

AddMultiSigAddress: adds a P2SH multisig address to the wallet.
BackupWallet: safely copies wallet.dat to the specified file, which can be a directory or a path with filename.
DumpPrivKey: returns the wallet-import-format (WIP) private key corresponding to an address. (But does not remove it from the wallet.)
DumpWallet: creates or overwrites a file with all wallet keys in a human-readable format.
EncryptWallet: encrypts the wallet with a passphrase. This is only to enable encryption for the first time. After encryption is enabled, you will need to enter the passphrase to use private keys.
GetAccountAddress: returns the current Bitcoin address for receiving payments to this account. If the account doesn’t exist, it creates both the account and a new address for receiving payment. Once a payment has been received to an address, future calls to this RPC for the same account will return a different address.
GetAccount: returns the name of the account associated with the given address.
GetAddressesByAccount: returns a list of every address assigned to a particular account.
GetBalance: gets the balance in decimal bitcoins across all accounts or for a particular account.
GetNewAddress: returns a new Bitcoin address for receiving payments. If an account is specified, payments received with the address will be credited to that account.
GetRawChangeAddress: returns a new Bitcoin address for receiving change. This is for use with raw transactions, not normal use.
GetReceivedByAccount: returns the total amount received by addresses in a particular account from transactions with the specified number of confirmations. It does not count coinbase transactions.
GetReceivedByAddress: returns the total amount received by the specified address in transactions with the specified number of confirmations. It does not count coinbase transactions.
GetTransaction: gets detailed information about an in-wallet...
Can you write Python code for it?

print '\nBob\'s Wallet\n'
wallet= Bob.getwalletinfo()
for key in wallet.keys():
    print key + '\t' + str(wallet[key])
print '\nMary\'s Wallet\n'
wallet= Mary.getwalletinfo()
for key in wallet.keys():
    print key + '\t' + str(wallet[key])

import time
print '\nMary has ' + str(len(Mary.listtransactions())) + ' transactions from Bob\'s largesse'

print '\nMary\'s first address has received how many BTC? ' +\
str(Mary.getreceivedbyaddress( Mary.listtransactions()[0]['address'],0))
#str(Mary.getreceivedbyaddress('mmT3ER6w98jZAKwtTZrr3DSxrchS7fGxKW',0))

print '\nBob has ' + str(len(Bob.listtransactions())) + ' transactions from all that mining'
#print Bob.listtransactions()

##let's send Mary some more of Bob's bitcoins so we can see her unconfirmed balance
getnewaddress = Mary.getnewaddress()
print '\nNew address ' + str(getnewaddress)

##have Bob (proxy) send 0.5 bitcoins to Mary
txid = Bob.sendtoaddress(getnewaddress,0.5)

time.sleep(2)

print '\nMary\'s unconfirmed balance ' + str(Mary.getunconfirmedbalance())
print '\nMary\'s confirmed balance ' + str(Mary.getbalance())

##let's mine 6 blocks
Bob.generate(6)

time.sleep(2)

print 'After Bob\'s mining'
print '\nMary\'s unconfirmed balance ' + str(Mary.getunconfirmedbalance())
print '\nMary\'s confirmed balance ' + str(Mary.getbalance())


I am working on the file "Regular Expressions.ipynb".
The first task is:
Character Sets
Character sets are used when you wish to match any one of a group of characters at a point in the input. Brackets are used to construct character set inputs. For example: the input [ab] searches for occurrences of either a or b.
Let's see some examples:
Can you write Python code for it?

test_phrase = 'sdsd..sssddd...sdddsddd...dsds...dsssss...sdddd'

test_patterns = [ '[sd]',    # either s or d
            's[sd]+']   # s followed by one or more s or d

multi_re_find(test_patterns,test_phrase)


I am working on the file "Regular Expressions.ipynb".
The first task is:
It makes sense that the first [sd] returns every instance. Also the second input will just return any thing starting with an s in this particular case of the test phrase input.
Exclusion
We can use ^ to exclude terms by incorporating it into the bracket syntax notation. For example: [^...] will match any single character not in the brackets. Let's see some examples:
Can you write Python code for it?

test_phrase = 'This is a string! But it has punctuation. How can we remove it?'


I am working on the file "Regular Expressions.ipynb".
The first task is:
Note how re.split() returns a list with the term to spit on removed and the terms in the list are a split up version of the string. Create a couple of more examples for yourself to make sure you understand!
Finding all instances of a pattern
You can use re.findall() to find all the instances of a pattern in a string. For example:
Can you write Python code for it?

# Returns a list of all matches
re.findall('match','test phrase match is in middle')


I am working on the file "Regular Expressions.ipynb".
The first task is:
Now we've seen that re.search() will take the pattern, scan the text, and then returns a Match object. If no pattern is found, a None is returned. To give a clearer picture of this match object, check out the cell below:
Can you write Python code for it?

# List of patterns to search for
pattern = 'term1'

# Text to parse
text = 'This is a string with term1, but it does not have the other term.'

match = re.search(pattern,  text)

type(match)


I am working on the file "Regular Expressions.ipynb".
The first task is:
Pattern re Syntax
This will be the bulk of this lecture on using re with Python. Regular expressions supports a huge variety of patterns the just simply finding where a single string occurred.
We can use metacharacters along with re to find specific types of patterns.
Since we will be testing multiple re syntax forms, let's create a function that will print out results given a list of various regular expressions and a phrase to parse:
Can you write Python code for it?

def multi_re_find(patterns,phrase):
    '''
    Takes in a list of regex patterns
    Prints a list of all matches
    '''
    for pattern in patterns:
        print 'Searching the phrase using the re check: %r' %pattern
        print re.findall(pattern,phrase)
        print '\n'


I am working on the file "Regular Expressions.ipynb".
The first task is:
Repetition Syntax
There are five ways to express repetition in a pattern:
1.) A pattern followed by the meta-character * is repeated zero or more times.
2.) Replace the * with + and the pattern must appear at least once.
3.) Using ? means the pattern appears zero or one time.
4.) For a specific number of occurrences, use {m} after the pattern, where m is replaced with the number of times         the pattern should repeat.
5.) Use {m,n} where m is the minimum number of repetitions and n is the maximum. Leaving out n ({m,}) means the           value appears at least m times, with no maximum.

Now we will see an example of each of these using our multi_re_find function:
Can you write Python code for it?

test_phrase = 'sdsd..sssddd...sdddsddd...dsds...dsssss...sdddd'

test_patterns = [ 'sd*',     # s followed by zero or more d's
                'sd+',          # s followed by one or more d's
                'sd?',          # s followed by zero or one d's
                'sd{3}',        # s followed by three d's
                'sd{2,3}',      # s followed by two to three d's
                ]

multi_re_find(test_patterns,test_phrase)


I am working on the file "Regular Expressions.ipynb".
The first task is:
Split with regular expressions
Let's see how we can split with the re syntax. This should look similar to how you used the split() method with strings.
Can you write Python code for it?

# Term to split on
split_term = '@'

phrase = 'What is the domain name of someone with the email: hello@gmail.com'

# Split the phrase
re.split(split_term,phrase)


I am working on the file "Regular Expressions.ipynb".
The first task is:
Use [^!.? ] to check for matches that are not a !,.,?, or space. Add the + to check that the match appears at least once, this basically translate into finding the words.
Can you write Python code for it?


I am working on the file "RegularExpressions.ipynb".
The first task is:
Character Class Size

The number of characters in a class can be specified by metacharacters

 *  matches zero or more characters:
[0-9]*  (matches zero or more digits)

+ matches one or more characters
? matches exactly one character
{m,n}  match at least m characters and at most n characters

QUESTION: How would you express *,+, and ? with m,n syntax?

QUESTION: What would the following regular expressions match?
Can you write Python code for it?

r8 = re.compile(r"""a[bcd]*b""",re.I)

r9 = re.compile(r"""a{1,3}b""")
print (r8.findall(test))
print (r9.findall(test))
print(r9.findall("""aabaaabab"""))


I am working on the file "RegularExpressions.ipynb".
The first task is:
Find all strings that end in "ython"
Find all instances of "Brian"
Find all jython or python instances
Find all python or perl instances
Find all names that start with B
Can you write Python code for it?

rEx1 = re.compile(r"""[a-zA-Z]*ython""")
print rEx1.findall(testString)

rEx2 = re.compile(r"""Brian""",re.I)
print rEx2.findall(testString)

rEx3 = re.compile(r"""jython|python|ziggy|zoom""",re.I) # this illustrates an OR
print rEx3.findall(testString)

rEx4 = re.compile(r"""python|perl""") # this illustrates an OR
rEx4b = re.compile(r"""python|perl""",re.IGNORECASE) # this illustrates an OR

print rEx4.findall(testString)
print rEx4b.findall(testString)

rEx5 = re.compile(r"""B[a-z]*""") # this illustrates an AND
print rEx5.findall(testString)


I am working on the file "RegularExpressions.ipynb".
The first task is:
How about finding names?
Let's look at two formats
Brian Chapman
Chapman, Brian
Can you write Python code for it?

findName1 = re.compile(r"""[A-Z][a-z]+\s+[A-Z][a-z]+""")
#print findName1.findall("Brian Chapman, Wendy Chapman, Jeremiah Chapman")
names1 = findName1.findall(txt)
print (names1[:80])

findName2 = re.compile(r"""[A-Z][a-z]*,\s+[A-Z][a-z]*""")
names2 = findName2.findall(txt)
print (names2[:20])

testString = """Brian has a nephew named Ben. Br. Chapman died yesterday. Brian Chapman Brian E. Chapman Brian Earl Chapman Wendy Webber Chapman Clare 1234 4321.1234
python python.org http://python.org www.python.org jython zython Brad Bob cpython brian http://www.python.org perl Perl PERL"""


I am working on the file "RegularExpressions.ipynb".
The first task is:
How about finding telephones?
Can you write Python code for it?

txt = open("../Resources/contact.html","r").read()
findPhone = re.compile(r"""[0-9]{3,3}-\d{4,4}|[0-9]{3,3}-[0-9]{3,3}-\d{4,4}""")
print (findPhone.findall(txt))


I am working on the file "RegularExpressions.ipynb".
The first task is:
If I want, I can make the regular expression case insensitive with a COMPLIATION FLAG.
Can you write Python code for it?

r1 = re.compile(r"""Brian""")
print (r1.findall(nameString))
r2 = re.compile(r"""Brian""", re.IGNORECASE)
print (r2.findall(nameString))
r3 = re.compile(r"""Brian""",re.I) # alias for IGNORECASE
print (r3.findall(nameString))


I am working on the file "RegularExpressions.ipynb".
The first task is:
Metacharacters (\)

The backslash character is used to escape all the metacharacters, so to search for a * you would type \*.
Try the regular expression with out the backslash
Can you write Python code for it?

r11 = re.compile(r"\*")
print (r11.findall("""The range of the function f1* is [0,12]
                  and the domain of f1* is [0,144]"""))


I am working on the file "RegularExpressions.ipynb".
The first task is:
Metacharacters in Character Classes

Within character classes, metacharacters do not have special meaning.
That is, they are treated just like any other character to be matched.
One exception: ^{} as the first character in the class denotes the compliment.
That is, match all characters except those denoted in the class.
Can you write Python code for it?

r10 = re.compile(r"""[^A-Z]""")
print (r10.findall(test))


I am working on the file "RegularExpressions.ipynb".
The first task is:
Putting Things Together

How could we use regular expressions to recognize zip codes?
Can you write Python code for it?

address = "My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108."

findZipcode = re.compile(r"""\d{5,5}""")
findZipcode2 = re.compile(r"""[0-9]{5,5}""")
print (findZipcode.findall(address))
print (findZipcode2.findall(address))


I am working on the file "RegularExpressions.ipynb".
The first task is:
The Real Power of Regular Expressions...

Comes from metacharacters
Here is a list of all the metacharacters
    . ^ $ * + ? { [ ] \ | ( )

Metacharacters: []

Square brackets are used to specify a "character class, which is a set of characters that you wish to match."
[abc]
[a-c]
[a-z]
[a-zA-Z]
[a-zA-Z0-9]
Can you write Python code for it?

test = """abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMN
                       OPQRSTUVWXYZ0123456789"""

r4 = re.compile(r"""[abc]""")

print (r4.findall(test))

r5 = re.compile(r"""[a-c]""")
print (r5.findall(test))

r6 = re.compile(r"""[a-z]""")
print (r6.findall(test))

r7 = re.compile(r"""[a-zA-Z0-9]""")
print (r7.findall(test))


I am working on the file "RegularExpressions.ipynb".
The first task is:
\D : Matches any non-digit character;
this is equivalent to the class [^0-9].
Can you write Python code for it?

r13 = re.compile(r"\D")
print (r13.findall("My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108.")
)


I am working on the file "RegularExpressions.ipynb".
The first task is:
\d : Matches any decimal digit;
this is equivalent to the class [0-9].
Can you write Python code for it?

r12 = re.compile(r"\d")
print (r12.findall("My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108."))


I am working on the file "RegularExpressions.ipynb".
The first task is:
\s : Matches any whitespace character;
this is equivalent to the [ \t\n\r\f\v].
Can you write Python code for it?

r14 = re.compile(r"\s")
result14= r14.findall("My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108.")
print (result14)
r15 = re.compile(r"[ \t\n\r\f\v]")
result15 = r15.findall("My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108.")
print (result15)
print( result14 == result15)


I am working on the file "RegularExpressions.ipynb".
The first task is:
\w: Matches any alphanumeric character;
this is equivalent to the class [a-zA-Z0-9_].
Can you write Python code for it?

r18 = re.compile(r"\w")
r19 = re.compile(r"[a-zA-Z0-9_]")
result18 = r18.findall("My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108.")
result19 = r19.findall("My work address is 729 Arapeen Drive, Salt Lake City, UT, 84108.")
print (result18)
print (result19)
print (result18 == result19)


I am working on the file "Regularized Regression.ipynb".
The first task is:
0) Dependencies
Can you write Python code for it?

################################################################################
######################### CHANGE BQ PROJECT NAME BELOW #########################
################################################################################

project_name = '' #add proj name and dataset

# Google credentials authentication libraries
from google.colab import auth
auth.authenticate_user()

# data processing libraries
import numpy as np
from numpy.core.numeric import NaN
import pandas as pd
import pandas_gbq
import datetime

# modeling and metrics
from scipy.optimize import least_squares
from statsmodels.tools.tools import add_constant
import statsmodels.api as sm
from sklearn.model_selection import train_test_split, cross_val_score, LeaveOneOut, KFold, LeavePOut
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
from statsmodels.stats.stattools import durbin_watson

# Visualization
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

# Calculating Relative Importance
!pip install relativeImp
from relativeImp import relativeImp

# BigQuery Magics
'''
BigQuery magics are used to run BigQuery SQL queries in a python environment.
These queries can also be run in the BigQuery UI
'''

from google.cloud import bigquery
from google.cloud.bigquery import magics

magics.context.project = project_name  #update your project name

client = bigquery.Client(project=magics.context.project)
%load_ext google.cloud.bigquery
bigquery.USE_LEGACY_SQL = False


I am working on the file "Regularized Regression.ipynb".
The first task is:
1.1) Define KPI column and feature set
Can you write Python code for it?

KPI_COL = "y1"
y = df[KPI_COL]
X = df[df.columns[df.columns != KPI_COL]].values


I am working on the file "Regularized Regression.ipynb".
The first task is:
2.2) Print the model evaluation metrics
Can you write Python code for it?

#R-squared
reg.score(X,y)

# Generate predictions to calculate MAE, MSE, RMSE
Y_prediction = reg.predict(X)

mean_absolute_error(y,Y_prediction)

mean_squared_error(y,Y_prediction)

rmse = np.sqrt(mean_squared_error(y,Y_prediction))
rmse


I am working on the file "Regularized Regression.ipynb".
The first task is:
3) Calculate contribution of each digitl
Use the relativeImp package
to conduct key driver analysis and generate relative importance values by feature in the model.
The relativeImp function produces a raw relative importance and a normalized relative importance value.
- Raw relative importance sums to the r-squared of the linear model.
- Normalized relative importance is scaled to sum to 1
Can you write Python code for it?

yName = 'y1'
xNames = df[df.columns[df.columns != KPI_COL]].columns.to_list()
df_results = relativeImp(df, outcomeName = yName, driverNames = xNames)

df_results


I am working on the file "Regularized Regression.ipynb".
The first task is:
4.3) Normality of Errors
Can you write Python code for it?

'''
Visually inspect the residuals to confirm normality
'''

fig = sm.qqplot(residuals)

sns.kdeplot(residuals, label = '', shade = True)
plt.xlabel('Model Residuals'); plt.ylabel('Density'); plt.title('Distribution of Residuals');


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
3. Load Monte Carlo algorithms
3a. Load Plotting Utilities
Load a custom Python class of plotting utilities, PlotUtils(), and start a new instance.
Can you write Python code for it?

%run ../PlotUtils.py

plotutls = PlotUtils()


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
4. Monte Carlo Prediction algorithms (estimating $V \approx v_{\pi}$)
Now that we learned how to generate an episode, we will see how to perform First Vist MC Prediction
4a. Approximate state-value functions after 100.000 episodes (first-visit MC)
Can you write Python code for it?

s_values = mc.first_visit_mc_prediction(env, sample_policy, n_episodes=1e+4)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
4b. Approximate state-value functions after 500.000 episodes (first-visit MC)
Can you write Python code for it?

s_values = mc.first_visit_mc_prediction(env, sample_policy, n_episodes=5e+5)

title = 'Approximate State-value function after 500.000 episodes\n(first-visit MC)'
plotutls.plot_blackjack_1(s_values, title=title)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
5. Monte-Carlo Control algorithms
5a. Monte Carlo Exploring Starts (ES) [estimating $\pi \approx \pi_{*}$]
Can you write Python code for it?

# Generalization of the np.argmax to break ties randomly when exist
def argmax(q_values):
    """
    Takes in a list of q_values and returns the index of the item
    with the highest value. Breaks ties randomly.
    returns: int - the index of the highest value in q_values

    Parameters
    ----------
    q_values : array_like
        Input array.

    Returns
    -------
    index_array : ndarray of ints
        The index of the highest value in q_values. Breaks ties randomly.
    """
    idxmax = [ix_ for ix_ in range(len(q_values))
              if q_values[ix_] == np.max(q_values)]
    return np.random.choice(idxmax)

# Behavior policy is greedy or epsilon-greedy (if epsilon > 0, but small)
def behavior_policy(obs, q_values, q_values_count, epsilon=0):
    '''
    Define greedy behavior policy using numpy array to keep q_values table.
    '''
    # Reset state indices to appropriately refer state in q_values numpy array
    player_sum, dealer_card, usable_ace = obs
    player_sum -= 12; dealer_card -= 1; usable_ace = int(usable_ace)

    # Get average returns for the possible state-action pairs and current state
    q_values_ = (q_values[player_sum, dealer_card, usable_ace,:] /
                 q_values_count[player_sum, dealer_card, usable_ace,:])

    # Return the argmax of the action values list for current state (break ties randomly)
    a_star = argmax(q_values_)
    greedy_prob = 1

    if epsilon > 0:
        assert(epsilon < 1)
        non_greedy_prob = epsilon / q_values.size
        greedy_prob = 1 - epsilon + non_greedy_prob
        a_star = np.random.binomial(1, greedy_prob)

    return a_star, greedy_prob

def target_default_policy(obs):
    '''
    Define the target policy we would like to achieve
    '''
    # Define ideal player policy
    player_sum, dealer_card, usable_ace = obs
    return (0 if player_sum >= 20 else 1), 1


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
5b. On-policy first-visit MC Control for $\varepsilon-\text{soft}$ policies [estimating $\pi\approx\pi_{*}$]
Can you write Python code for it?

n_episodes = 5e+5; epsilon=0.35
q_values = mc.monte_carlo_on_policy(env, behavior_policy,
                                    target_default_policy, n_episodes=n_episodes, epsilon=epsilon)

title = 'Black-Jack Optimal Policies after {0:,d} episodes\n[first-visit MC, ε-soft policies (ε={1:.2f})]'.format(int(n_episodes), epsilon)
plotutls.plot_blackjack_2(q_values, suptitle=title)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
5c. Off-policy every-visit MC Prediction [Policy Evaluation (estimating $Q\approx q_{\pi}$)]
Using Weighted Importance Sampling
Can you write Python code for it?

# Define a random behavior policy
def behavior_random_policy(obs):
    return np.random.binomial(1, 0.5), 0.5

n_episodes = 5e+5
q_values = mc.monte_carlo_off_policy_prediction(env, behavior_random_policy,
                                                target_default_policy, n_episodes=n_episodes)

title = 'Black-Jack Policy Evaluation after {0:,d} episodes\n[Off-policy every-visit MC Prediction]'.format(int(n_episodes))
plotutls.plot_blackjack_2(q_values, suptitle=title)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
5d. Off-policy every-visit MC Control [estimating $\pi\approx \pi_{*}$]
Using Weighted Importance Sampling
Can you write Python code for it?

n_episodes = 5e+6; epsilon=0.75
q_values, pi_values = mc.monte_carlo_off_policy_control(env, behavior_policy,
                                                        target_default_policy, n_episodes=n_episodes, epsilon=epsilon)

title=('Black-Jack Optimal Policies after {0:,d} episodes\n[Off-Policy every-visit MC Control, with ε-soft behavior policy (ε={1:.2f})]'
       .format(int(n_episodes), epsilon))
plotutls.plot_blackjack_3(q_values, pi_values, suptitle=title)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
Let us see first few elements in the value table
Can you write Python code for it?

for i in range(10):
  print(s_values.popitem())


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
Monte Carlo Exploring Starts (ES) [estimating $\pi \approx \pi_{*}$, (first-visit MC after 10.000 episodes)]
Can you write Python code for it?

n_episodes = 1e+4
q_values = mc.monte_carlo_es(env, behavior_policy,
                             target_default_policy, n_episodes=n_episodes)

title = 'Black-Jack Optimal Policies after {0:,} episodes\n(first-visit MC with Exploring Starts)'.format(int(n_episodes), format_spec='d')
plotutls.plot_blackjack_2(q_values, suptitle=title)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
Monte Carlo with Exploring Starts (ES) [estimating $\pi \approx \pi_{*}$, (first-visit MC after 500.000 episodes)]
Can you write Python code for it?

n_episodes = 5e+5
q_values = mc.monte_carlo_es(env, behavior_policy,
                             target_default_policy, n_episodes=n_episodes)

title = 'Black-Jack Optimal Policies after {0:,} episodes\n(first-visit MC with Exploring Starts)'.format(int(n_episodes), format_spec='d')
plotutls.plot_blackjack_2(q_values, suptitle=title)


I am working on the file "Reinforcement-Learning/MC-simulations/01.BlackJack_MC.ipynb".
The first task is:
Note: There is only one "Blackjack" environment defined in OpenAI Gym
Can you write Python code for it?

[env for env in list(gym.envs.registry.env_specs.keys()) if env.startswith('Blackjack')]