I am working on the file "Linear Programming with OR-Tools.ipynb".
The first task is:
Where there are 10 hours a day dedicated to production. Process times are given in minutes per unit while profit is given in USD.
The optimal mix of the two products would be characterized by the following model:
$\begin{aligned}
x_1 = \textrm{Units of product 1} \newline
x_2 = \textrm{Units of product 2}
\end{aligned}$
$$\textrm{Maximize  } z = 20x_1 + 30x_2$$
subject to
$$\begin{array}{rcl}
10x_1 + 5x_2 \le 600 \newline
6x_1 + 20x_2 \le 600 \newline
8x_1 + 10x_2 \le 600 \newline
x_1, x_2 \ge 0
\end{array}$$
(we will assume that continuous solution values are acceptable for this problem)
Can you write Python code for it?

simpleprod = pywraplp.Solver('Simple_Production', pywraplp.Solver.GLOP_LINEAR_PROGRAMMING)

x1 = simpleprod.NumVar(0, simpleprod.infinity(), 'x1')
x2 = simpleprod.NumVar(0, simpleprod.infinity(), 'x2')

for i in problemdata.columns[:-1]:
    simpleprod.Add(problemdata.loc[problemdata.index[0], i]*x1 + problemdata.loc[problemdata.index[1], i]*x2 <= 600)

profit = simpleprod.Objective()
profit.SetCoefficient(x1, 20)
profit.SetCoefficient(x2, 30)
profit.SetMaximization()

status = simpleprod.Solve()

if status not in [simpleprod.OPTIMAL, simpleprod.FEASIBLE]:
    raise Exception('No feasible solution found')

print("The company should produce",round(x1.solution_value(),2),"units of product 1")
print("The company should produce",round(x2.solution_value(),2),"units of product 2")
print("The optimal profit is", round(profit.Value(),2), 'USD')


I am working on the file "Linear Regression.ipynb".
The first task is:
Bootstrap to find parameter confidence intervals
Can you write Python code for it?

from sklearn.utils import resample

bootstrap_resamples = 5000
intercepts = []
coefs = []
for k in range(bootstrap_resamples):
    #resample population with replacement
    samples_resampled = resample(X,Y,replace=True,n_samples=len(X))

    ## Fit model to resampled data
    # Create linear regression object
    regr = linear_model.LinearRegression()

    # Train the model using the training sets
    regr.fit(samples_resampled[0], samples_resampled[1])

    coefs.append(regr.coef_[0][0])
    intercepts.append(regr.intercept_[0])


I am working on the file "Linear Regression.ipynb".
The first task is:
Calculate confidence interval
Can you write Python code for it?

alpha = 0.95
p_lower = ((1-alpha)/2.0) * 100
p_upper = (alpha + ((1-alpha)/2.0)) * 100
coefs_lower = np.percentile(coefs,p_lower)
coefs_upper = np.percentile(coefs,p_upper)
intercepts_lower = np.percentile(intercepts,p_lower)
intercepts_upper = np.percentile(intercepts,p_upper)
print('Coefs %.0f%% CI = %.5f - %.5f' % (alpha*100,coefs_lower,coefs_upper))
print('Intercepts %.0f%% CI = %.5f - %.5f' % (alpha*100,intercepts_lower,intercepts_upper))


I am working on the file "Linear Regression.ipynb".
The first task is:
It looks like our mean square error between our training and testing was pretty close.
Step 9 : Residual Plots
In regression analysis, the difference between the observed value of the dependent variable (y) and the predicted value (ŷ) is called the residual (e). Each data point has one residual, so that:
Residual=Observedvalue−Predictedvalue
You can think of these residuals in the same way as the D value we discussed earlier, in this case however, there were multiple data points considered.
A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis. If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data; otherwise, a non-linear model is more appropriate.
Residual plots are a good way to visualize the errors in your data. If you have done a good job then your data should be randomly scattered around line zero. If there is some strucutre or pattern, that means your model is not capturing some thing. There could be an interaction between 2 variables that you're not considering, or may be you are measuring time dependent data. If this is the case go back to your model and check your data set closely.
So now let's go ahead and create the residual plot. For more info on the residual plots check out this great link.
Can you write Python code for it?

# Scater plot the training data
train = plt.scatter(pred_train, (pred_train - Y_train), c='b', alpha=0.8)

# Scatter plot the testing data
test = plt.scatter(pred_test, (pred_test - Y_test), c='r', alpha=0.6)

# Horizontal line
plt.hlines(y=0, xmin=-10, xmax=50)

#Labels
plt.legend((train,test),('Training','Test'),loc='lower left')
plt.title('Residual Plots')

plt.savefig('residualplot.png')


I am working on the file "Linear Regression.ipynb".
The first task is:
Let us write a function to fit the model such that it automatically stops once the improvement in the value of the cost function is below a certain threshold.
Can you write Python code for it?

def fit(X, y):
    theta = np.zeros([X.shape[1]])
    theta, cost = descend(theta, X, y)

    for i in range(10000):
        cost_ = cost
        theta, cost = descend(theta, X, y)
        if cost_ - cost < 1e-7:
            break

        if i % 10 == 0:
            print("epoch: {0}, cost: {1}".format(i, cost))

    print("epoch: {0}, cost: {1}".format(i, cost))
    print("theta: {0}".format(theta))


I am working on the file "Linear Regression.ipynb".
The first task is:
Let's transform our data into the right matrix format. Note that we add a column of one's to the $X$ matrix, to be multiplied with $\theta_0$.
Can you write Python code for it?

X = houses[['sqft', 'rooms']].as_matrix()
X = np.column_stack([np.ones([X.shape[0]]), X])
y = houses[['price']].as_matrix().ravel()


I am working on the file "Linear Regression.ipynb".
The first task is:
Let's try to fit the model again with our scaled input and output matrices.
Can you write Python code for it?


I am working on the file "Linear Regression.ipynb".
The first task is:
Next we implement the hypothesis and cost functions and the parameter update using gradient descent.
Can you write Python code for it?

# Hypothesis function
def h(theta, X):
    return np.matmul(X, theta)

# Cost function
def J(theta, X, y):
    d = h(theta, X) - y
    return 0.5 * np.dot(d, d.T)

# One step of gradient descent
def descend(theta, X, y, alpha=0.01):
    error = h(theta, X) - y
    t = theta - alpha * np.matmul(X.T, error)
    return t, np.dot(error, error.T)


I am working on the file "Linear Regression.ipynb".
The first task is:
Now we can make out a slight trend that price increases along with the number of rooms in that house, which intuitively makes sense! Now let's use scikit learn to see if we can fit the data linearly.
Let's try to do the following:
1.) Use pandas to transform the boston dataset into a DataFrame:
2.) Then use seaborn to perform an lmplot on that DataFrame to reproduce the scatter plot with a linear fit line.
Can you write Python code for it?

# converting into dataFrame

boston_df = DataFrame(boston.data)
boston_df.columns=  boston.feature_names

boston_df.head()

# Creating a price column in dataFrame

boston_df['PRICE'] = boston.target

boston_df.head()


I am working on the file "Linear Regression.ipynb".
The first task is:
Now, you might be reminded of the seaborn lmplot function we used during the visualization lectures. You could use it here to do a linear fit automatically!
Can you write Python code for it?

# linear regression plot

sns.lmplot('RM', 'PRICE', data=boston_df)


I am working on the file "Linear Regression.ipynb".
The first task is:
Plot the data
Can you write Python code for it?

fig = plt.figure(1, figsize=(10, 4))
plt.scatter([X], [Y], color='blue', edgecolor='k')
plt.plot(X, Y_pred, color='red', linewidth=1)

plt.xticks(())
plt.yticks(())

print('Coefficients: ', regr.coef_)

plt.show()

# The mean squared error
print("Mean squared error: %.2f"
      % mean_squared_error(Y, Y_pred))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % r2_score(Y, Y_pred))


I am working on the file "Linear Regression.ipynb".
The first task is:
Prepare Dataset
Can you write Python code for it?

with open('data_w1w4.csv', 'r') as f:
      reader = csv.reader(f)
      data = list(reader)

matrix = obtain_data_matrix(data)
samples = len(matrix)
print("Number of samples: " + str(samples))

Y = matrix[:,[8]]
X = matrix[:,[9]]
S = matrix[:,[11]]


I am working on the file "Linear Regression.ipynb".
The first task is:
Step 2: Visualizing current data
You should always try to do a quick visualization fo the data you have. Let's go ahead an make a histogram of the prices.
Can you write Python code for it?

# histogram of prices

plt.hist(boston.target, bins=50)

plt.xlabel('Prices in 1000$')
plt.ylabel('Number of houses')
plt.title('Prices Vs Houses')

plt.savefig('house_vs_price.png')

plt.scatter(boston.data[:,5], boston.target)

#label
plt.ylabel('Price in $1000s')
plt.xlabel('Number of rooms')


I am working on the file "Linear Regression.ipynb".
The first task is:
Step 3: The mathematics behind the Least Squares Method.
In this we'll use the least squares method as the way to estimate the coefficients. Here's a quick breakdown of how this method works mathematically:
Take a quick look at the plot we created above using seaborn. Now consider each point, and know that they each have a coordinate in the form (X,Y). Now draw an imaginary line between each point and our current "best-fit" line. We'll call the distanace between each point and our current best-fit line, D. To get a quick image of what we're currently trying to visualize, take a look at the picture below:
Can you write Python code for it?

# Quick display of image form wikipedia
from IPython.display import Image
url = 'http://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Linear_least_squares_example2.svg/220px-Linear_least_squares_example2.svg.png'
Image(url)


I am working on the file "Linear Regression.ipynb".
The first task is:
Step 5: Getting the error
We've just completed a single variable regression using the least squares method with Python! Let's see if we can find the error in our fitted check the link. Checking out the documentation here, we see that the resulting array has the total squared error. For each element, it checks the the difference between the line and the true value (our original D value), squares it, and returns the sum of all these. This was the summed D^2 value we discussed earlier.
It's probably easier to understand the root mean squared error, which is similar to the standard deviation. In this case, to find the root mean square error we divide by the number of elements and then take the square root. There is also an issue of bias and an unbiased regression, but we'll delve into those topics later.
For now let's see how we can get the root mean squared error of the line we just fitted.
Can you write Python code for it?

"""
Dependent variable always on y axis and independent variable on x axis while plotting.
"""

result = np.linalg.lstsq(x,y)[1]

# Total error
total_error = np.sqrt(result/len(x))

print "The root mean square error is: {}" .format(float(total_error))


I am working on the file "Linear Regression.ipynb".
The first task is:
Step 8: Predicting Prices
Now that we have our training and testing sets, let's go ahead and try to use them to predict house prices. We'll use our training set for the prediction and then use our testing set for validation.
Can you write Python code for it?

legr = LinearRegression()

legr.fit(X_train, Y_train)

pred_train = legr.predict(X_train)
pred_test = legr.predict(X_test)

print "Fit a model X_train, and calculate MSE with Y_train: {}" .format(np.mean((Y_train - pred_train)**2))
print "Fit a model X_train, and calculate MSE with X_test and Y_test: {}" .format(np.mean((Y_test - pred_test)**2))


I am working on the file "Linear Regression.ipynb".
The first task is:
Success!
The gradient descent converges in just 38 steps.
We can verify that our solution is correct by fitting the same data using the library functions for linear regression in sklearn.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

l = LinearRegression(fit_intercept=False)
l.fit(X_scaled, y_scaled)
l.coef_


I am working on the file "Linear Regression.ipynb".
The first task is:
That doesn't look good. We expected the cost to steadily decrease as gradient descent progressed. Instead, the cost function diverged so much it exceeded our ability to represent it as a floating-point number. What happened?
The answer is that our training data is not on the right scale. The area in square feet is in the thousands, the number of rooms is 1-10 and the price is in the hundreds of thousands. With such widely varying numbers the gradient descent algorithm can overshoot the minimum and instead diverge to infinity.
The solution is to scale all of the variables to have mean of 0 and variance of 1. We can use the scale function from sklearn to do this.
Can you write Python code for it?


I am working on the file "Linear Regression.ipynb".
The first task is:
Use the model (LinearRegression)
Can you write Python code for it?

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(X, Y)

# Make predictions using the testing set
Y_pred = regr.predict(X)


I am working on the file "Linear Regression.ipynb".
The first task is:
Visualize frequency distributions of bootstrapped parameters
Can you write Python code for it?

plt.hist(coefs)
plt.xlabel('Coefficient X0')
plt.title('Frquency Distribution of Coefficient X0')
plt.show()

plt.hist(intercepts)
plt.xlabel('Intercept')
plt.title('Frquency Distribution of Intercepts')
plt.show()


I am working on the file "Linear Regression.ipynb".
The first task is:
We are now ready to fit the model using gradient descent. Let's initialize our parameters to 0 and run 50 iterations of gradient descent to see how it behaves.
Can you write Python code for it?

theta = np.zeros([X.shape[1]])
for i in range(50):
    theta, cost = descend(theta, X, y)

    if i % 10 == 0:
        print("epoch: {0}, cost: {1}".format(i, cost))

print("epoch: {0}, cost: {1}".format(i, cost))
print("theta: {0}".format(theta))


I am working on the file "Linear Regression.ipynb".
The first task is:
We can plot the data again to visualize the effect of the scaling operation.
Can you write Python code for it?

plt.figure(1)
plt.subplot(211)
plt.xlabel('sq. feet')
plt.ylabel('price')
plt.scatter(X_scaled[:, 1], y_scaled)
plt.subplot(212)
plt.xlabel('no. of rooms')
plt.ylabel('price')
plt.scatter(X_scaled[:, 2], y_scaled)
plt.tight_layout()


I am working on the file "Linear Regression.ipynb".
The first task is:
下面以数学形式定义梯度下降，随机选择某个初始 $\theta$，之后重复以下更新：
$$ \theta_j = \theta_j - \alpha\frac{\partial}{\partial\theta_j}J(\theta) $$
（同时对所有值进行更新，$j = 0,...,n$）
这里，$\alpha$ 称为学习速率learning rate。这个算法非常自然地每次往J下降幅度最为陡峭的方向前进一小步。当 $\alpha$ 过小时，算法的收敛速度会比较慢；而 $\alpha$ 过大时，会出现算法不收敛的情况。另外，梯度下降过程中，梯度 $\frac{\partial}{\partial\theta_j}J(\theta)$ 本身会不断变小，因此更新的幅度也会越来越小：
$$\begin{equation}
\begin{split}
\frac{\partial}{\partial\theta_j}J(\theta) & = \sum_{i=1}^m\frac{\partial}{\partial\theta_j}\frac{1}{2}(h_\theta(x^{(i)}) - y^{(i)})^2 \
 & = \sum_{i=1}^m2 \cdot \frac{1}{2}(h_\theta(x^{(i)}) - y^{(i)}) \cdot \frac{\partial}{\partial\theta_j}(h_\theta(x^{(i)}) - y^{(i)}) \
 & = \sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)}) \cdot \frac{\partial}{\partial\theta_j}(\sum_{k=0}^{n}\theta_kx_k^{(i)}-y^{(i)}) \
 & = \sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)}
\end{split}
\end{equation}$$
我们根据以下规则进行更新：
$$ \theta_j = \theta_j - \alpha\sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} $$
重复直至 $\theta$ 收敛即可。
这个规则称为最小二乘 LMS least mean square，也被称为Widrow-Hoff学习规则。
每次更新过程中都检查整个训练集，我们称这种算法为批量梯度下降 batch gradient descent。对于线性回归来说，成本函数只有唯一一个全局最小值，不存在局域最小值。也就是说，$J$ 是一个凸二次函数。梯度下降对于线性回归总能收敛（唯一的前提，学习速率 $\alpha$ 没有选得过大）。
下图展现了动画形式的梯度下降训练过程：
Can you write Python code for it?

fig, ax = plt.subplots()
ax.scatter(X.reshape(-1), y, alpha=0.5)
plt.xlabel("Number of rooms")
plt.ylabel("Price in $1000")
x = np.arange(3, 10, 1)
line, = ax.plot([], [], lw=2)

def animate(i):
    line.set_data(x, np.column_stack((np.ones((x.size, 1)), x)) @ sampled_path[i])
    return line,

def init():
    line.set_data([], [])
    return line,

anim = animation.FuncAnimation(fig, animate, frames=sampled_path.shape[0], init_func=init, interval=800,
                               repeat_delay=3000, blit=True)

HTML(anim.to_html5_video())

# 使用批量梯度下降训练出的theta
print(gradient_descent_result.theta)

# 使用sklearn的LinearRegression模型训练出的theta
print(minima)


I am working on the file "Linear Regression.ipynb".
The first task is:
为了方便描述，我们这里引入一些符号。$x^{(i)}$ 用来表示输入变量，比如这里的平均房间数，它也被称为特征features。$y^{(i)}$ 用来表示输出变量，或我们希望预测的目标target变量。一对 $(x^{(i)}, y^{(i)})$ 称为一个训练样本training example，而我们将使用的 $m$ 个训练样本 ${(x^{(i)}, y^{(i)});i=1,...,m}$ 的集合称为训练集training set。注意这里的上标 $(i)$ 是对训练集的一个索引编号，不是幂运算。我们用 $\chi$ 表示输入空间，$\gamma$ 表示输出空间。在这个例子中 $\chi=\gamma=\mathbb{R}$
正式地定义监督学习：给定一个训练集，我们希望学习到一个函数 $h: \chi \rightarrow \gamma$，使得 $h(x)$ 可以很好地预测 $y$。由于历史原因，函数 $h$ 被称为假设hypothesis。
当我们希望预测的目标变量是连续型变量时，比如这里的房价，我们称这是一个回归Regression问题。当 $y$ 仅仅只取少数离散值时，我们称这是一个分类Classification问题。
线性回归 Linear Regression
包括以下内容：
    1. 最小二乘法 LMS algorithm
    2. 正规方程 The normal equations
    3. 概率诠释 Probabilistic interpretation
    4. 局部加权线性回归 Locally weighted linear regression
Can you write Python code for it?


I am working on the file "Linear Regression.ipynb".
The first task is:
对于这个数据集，$x$ 是一个$\mathbb{R}$上的一维向量。$x_1^i$是该区域的平均房间数。
要进行监督学习，我们需要首先决定如何来表示假设/函数 $h$。较为简单的情况下，我们选取 $x$ 的线性函数来预测 $y$ ：
$$ h_\theta(x) = \theta_0 + \theta_1x_1 $$
这里，$\theta_i$称为参数parameters（有时也称为权重weights），通过这组参数，我们将 $\chi$ 线性映射到 $\gamma$。在不引发误解的情况下，我们将 $h_\theta(x)$简写为 $h(x)$。习惯上，我们设 $x_0=1$ （称为截距项intercept term），这样
$$ h(x) = \sum_{i=0}^n \theta_ix_i = \theta^Tx $$
对于右式，我们将 $\theta$ 和 $x$ 看做向量，$n$ 是特征数量（不包括 $x_0$ )
这样，给定训练集，我们需要挑选，或者说学习出参数 $\theta$ 的值。一个直观的方法是使 $h(x)$ 尽量接近 $y$，至少对于训练集来说，尽量地接近。为了更正式地定义“接近”，我们定义成本函数cost function：
$$ J(\theta) = \frac{1}{2}\sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$
1. 最小二乘法 Linear Regression
我们希望挑选出使 $J(\theta)$ 最小化的值 $\theta$。一种搜索算法的思路是，我们随机猜测一个 $\theta$，然后按某种规则重复地改变 $\theta$ 使得 $J(\theta)$ 不断变小，直到 $\theta$ 收敛到某个值使得 $J(\theta)$ 取得最小值。
特别地，我们将考虑梯度下降Gradient Descent算法。如下图所示，$J(\theta)$ 是关于 $\theta_0$ 和 $\theta_1$ 的函数。首先我们随机选择一个 $ \theta = [-10, 17.5]$，想象在图形中下坡，我们需要选择一个下降最快的方向。不断地前进，直到来到某个局域最小值local minima。在我们的图中，只有一个全局最小值global minima（以红星标记）。但梯度下降的一个特性是，如果存在局域最小值时，根据随机选择初始化 $\theta$ 的不同，我们可能会到达不同的局域最小值。局部最小值的问题，在今后的模型中会遇到。
Can you write Python code for it?

# 读取训练集和目标变量
X, y = df[["RM"]].values, df["Price"].values
m, n = X.shape
# 增加截距列
X_intercept = np.column_stack((np.ones((m, 1)), X))

@np.vectorize
def cost_function(*args):
    """成本函数"""
    theta = np.array(args)
    return 0.5 * ((X_intercept @ theta - y) ** 2).sum()

def gradient_descent_optimize(cost_func, init_theta=None):
    """使用梯度下降寻找最优的theta"""
    alpha = 0.00003

    def good_enough(old_cost, new_cost):
        return np.abs(new_cost - old_cost) / old_cost < 0.00000001

    def improve_theta(old_theta):
        return old_theta - alpha * ((X_intercept @ old_theta - y) @ X_intercept)

    if init_theta is None:
        init_theta = np.array([0, 0])
    path_ = [init_theta]
    o_theta, o_cost = init_theta, cost_function(*init_theta)
    while True:
        n_theta = improve_theta(o_theta)
        n_cost = cost_func(*n_theta)
        path_.append(n_theta)
        logging.debug((o_theta, n_theta, o_cost, n_cost))
        if good_enough(o_cost, n_cost):
            break
        o_theta, o_cost = n_theta, n_cost
    Result = namedtuple('Result', ['theta', 'path'])
    return Result(theta=n_theta, path=np.array(path_))

# 使用梯度下降获取优化结果
init_guess = np.array([-10, 17.5])
gradient_descent_result = gradient_descent_optimize(cost_function, init_theta=init_guess)
path = gradient_descent_result.path
sampled_path = np.concatenate((np.array([path[0, ]]),
                               path[(2 ** np.array(range(int(np.ceil(np.log2(path.shape[0])))))).astype(np.int32), ]))
sampled_path_mat = sampled_path.T

# 使用sklearn的LinearRegression模型，直接获取最终的theta值
regr = LinearRegression()
regr.fit(X, y)
theta0_optimize, theta1_optimize = regr.intercept_, regr.coef_[0]
minima = np.array([theta0_optimize, theta1_optimize])
minima_ = minima.reshape(-1, 1)

# 根据最优化的theta，生成以此为中心的网格数据
step = 0.2
theta0_min, theta0_max = theta0_optimize - np.abs(theta0_optimize), theta0_optimize + np.abs(theta0_optimize)
theta1_min, theta1_max = theta1_optimize - np.abs(theta1_optimize), theta1_optimize + np.abs(theta1_optimize)
theta0_range,...


I am working on the file "Linear Regression.ipynb".
The first task is:
我们仅使用RM（平均房间数）和Price（按1000美元计的房价中位数）两个指标。
绘制平均房间数和房价中位数的散点图
Can you write Python code for it?

df = DataFrame(data=boston.data, columns=boston.feature_names)[["RM"]].join(DataFrame(data=boston.target, columns=["Price"]))
plt.scatter(df["RM"], df["Price"])
plt.xlabel("Number of rooms")
plt.ylabel("Price in $1000")
plt.show()


I am working on the file "Linear Regression/Lecture 1.ipynb".
The first task is:
Visualise data
Can you write Python code for it?


I am working on the file "Linear Regression/Lecture 1.ipynb".
The first task is:
We want create a function $h_{\theta}(x) = \theta_0 + \theta_1 x$ which will predict y for new unseen x.
Example $h_{\theta}(x) = 1 + 1.7x$
Can you write Python code for it?

def h(theta, x):
    return theta[0] + theta[1]*x

theta = [-10, 1.7]
df.plot(kind='scatter', x='x', y='y')
x = np.linspace(5, 25, 100)
plt.plot(
    x,
    h(theta, x),
    c='r'
)


I am working on the file "Linear Regression/Lecture 2.ipynb".
The first task is:
For many variables we will use vectorized implementation
$$X=\left[\begin{array}{cc}
1 & (\vec x^{(1)})^T \
1 & (\vec x^{(2)})^T \
\vdots & \vdots\
1 & (\vec x^{(m)})^T \
\end{array}\right]
= \left[\begin{array}{cccc}
1 & x_1^{(1)} & \cdots & x_n^{(1)} \
1 & x_1^{(2)} & \cdots & x_n^{(2)} \
\vdots & \vdots & \ddots & \vdots\
1 & x_1^{(m)} & \cdots & x_n^{(m)} \
\end{array}\right] $$
$$\vec{y} =
\left[\begin{array}{c}
y^{(1)}\
y^{(2)}\
\vdots\
y^{(m)}\
\end{array}\right]
\quad
\theta = \left[\begin{array}{c}
\theta_0\
\theta_1\
\vdots\
\theta_n\
\end{array}\right]$$
Vectorized implementation is much faster than that one from previous Lecture.
Can you write Python code for it?

df = pd.read_csv("ex1data1.txt", header=None)
df.columns = columns=['x', 'y']
X = np.matrix(df.x.values[:, np.newaxis])
# adding theta_0
m = len(X)
X = np.concatenate((np.ones((1,m)).T, X), axis=1)
y = np.matrix(df.y.values[:, np.newaxis])
theta = np.matrix([-5, 1.3]).reshape(2, 1)

print 'X', X[:10]
print 'y', y[:10]
print 'theta', theta


I am working on the file "Linear Regression/Lecture 2.ipynb".
The first task is:
How we count derivatives?
Let's count gradinet:
$$\nabla J(\theta) = \frac{1}{|\vec y|} X^T\left(X\theta-\vec y\right)$$
Gradinet Descent (vectorized)
$$ \theta = \theta - \alpha \nabla J(\theta) $$
Assigment 1.
Implement vectorized GD Algorithm
Normal matrix method
We can count $\hat\theta$ using this equation:
$$\theta = (X^TX)^{-1}X^T \vec y$$
Assigment 2.
Implement normal matrix method and check if $\theta$ vector is the same in GD method and normal Matrix method
Use pinv for computing inverse matrix. It's more numerical stable
|Gradient Method | Normal matrix|
|----------------|--------------|
|need to choose $\alpha$| no need to choose $\alpha$|
|needs many iterations | no iterations |
|it works for large amount of features (x)| slow for large amount of features (x)
| | we need to count inverse matrix |
Assigment 3.
Use library scikit-learn (normal matrix and gradient method) to fit model & predict y for x = 1, 10, 100
Can you write Python code for it?

from sklearn.linear_model import (
    LinearRegression,
    SGDRegressor
)


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
A NumPy array of floats, for instance:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
Convex optimization and semidefinite programmes
This section is especially for the fans of semidefinite programming relaxations. It requires further packages: PICOS, which is the Python equivalent of Yalmip, and CVXOPT, which is similar to the MATLAB toolbox CVX. If you are interested in the convex relaxations of polynomial optimization problems, take a look at Ncpol2sdpa, which is similar to Gloptipoly and SparsePOP in MATLAB, except that it also works for noncommutative variables.
The example below is the relaxation of a commutative polynomial optimization problem consisting of a moment matrix and a localizing matrix. It uses PICOS, which only works with Python 2. If you want to work with Python 3, either use CVXOPT or Ncpol2sdpa. The problem is written as
$$ \min_{y}2y_{12}$$
such that
$$\left[ \begin{array}{c|cc|cc}1 & y_{1} & y_{2} & y_{12} & y_{22}\\hline{}y_{1} & y_{1} & y_{12} & y_{12} & y_{122}\y_{2} & y_{12} & y_{22} & y_{122} & y_{222}\\hline{}y_{12} & y_{12} & y_{122} & y_{122} & y_{1222}\y_{22} & y_{122} & y_{222} & y_{1222} & y_{2222}\end{array} \right] \succeq{}0$$
$$\left[ \begin{array}{c|cc}-y_{22}+y_{2}+0.5 & -y_{122}+y_{12}+0.5y_{1} & -y_{222}+y_{22}+0.5y_{2}\\hline{}-y_{122}+y_{12}+0.5y_{1} & -y_{122}+y_{12}+0.5y_{1} & -y_{1222}+y_{122}+0.5y_{12}\-y_{222}+y_{22}+0.5y_{2} & -y_{1222}+y_{122}+0.5y_{12} & -y_{2222}+y_{222}+0.5y_{22}\end{array}\right]\succeq{}0.$$
Can you write Python code for it?

import picos
prob = picos.Problem()
M = prob.add_variable('M',(5,5),vtype='symmetric')
prob.add_constraint(M >> 0)
prob.add_constraint(M[0,0]==1)
prob.add_constraint(-M[1,1]+M[0,1]==0)
#y_12
prob.add_constraint(-M[0,3]+M[1,2]==0)
prob.add_constraint(-M[0,3]+M[1,3]==0)
#y_22
prob.add_constraint(-M[0,4]+M[2,2]==0)
#y_122
prob.add_constraint(-M[1,4]+M[2,3]==0)
prob.add_constraint(-M[1,4]+M[3,3]==0)

prob.set_objective('min', 2*M[1,2])

Mineq = prob.add_variable('Mineq',(3,3),vtype='symmetric')
prob.add_constraint(Mineq >> 0)

prob.add_constraint(Mineq[0,0]==-M[2,2]+M[0,2]+0.5)
prob.add_constraint(Mineq[0,1]==-M[1,4]+M[0,3]+0.5*M[0,1])
prob.add_constraint(Mineq[0,2]==-M[2,4]+M[0,4]+0.5*M[0,2])
prob.add_constraint(-Mineq[1,1]+Mineq[0,1]==0)
prob.add_constraint(Mineq[1,2]==-M[3,4]+M[2,3]+0.5*M[0,3])
prob.add_constraint(Mineq[2,2]==-M[4,4]+M[2,4]+0.5*M[0,4])

sol = prob.solve(solver='cvxopt', verbose = 0)
print(sol)


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
For matrices, you can ask for column-first or row-first representation. This makes sense if you know that some part of your code is written in FORTRAN or C, so you can have a zero-overhead interfacing with the lower-level code.
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
If you extend the size of a NumPy array, it actually does a deep copy of the existing elements. This comes from how the data is represented in memory: NumPy allocates continuous regions for efficient handling.
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
Lists and NumPy arrays are different. The latter is very similar to matrices and vectors in MATLAB. A list can be heterogeneous and vary in size, whereas a NumPy array has a fixed type and size. Here is a valid list of a float, a string, and a function:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
NumPy arrays support slicing on multidimensional arrays, which works identically to MATLAB. Note, however, that indexing starts with 0 and the last index is non-inclusive:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
QuTiP
QuTiP is the Quantum Toolbox in Python that builds on Numpy, SciPy, MatPlotLib and Cython. It relies on sparse data structures to calculate the dynamics of open systems, but it has hundreds of other uses. For instance, plotting Bloch spheres.
Can you write Python code for it?

from qutip import *
%matplotlib inline

b = Bloch()
b.make_sphere()


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
Reshaping is also allowed:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
SciPy
Sparse matrices, optimization, basic signal processing, interpolation, Fourier transforms, statistics, and even more linear algebra reside in this package. There are usually more featureful packages for most of the functionality, but SciPy is a decent common denominator. SciPy builds on NumPy, and this is true for most other scientific packages: NumPy is indispensable, the rest depends on what you want to do and how much control you want over the algorithms. For instance, using SciPy, the transfer-matrix method is about a hundred lines of straightforward code.
As an example, we look at optimization by a quasi-Newton method, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, which is the same as MATLAB uses in fminunc. The following example is from the SciPy tutorial. We import the minimize function and define a function with an initial point:
Can you write Python code for it?

from scipy.optimize import minimize
def rosen(x):
    """The Rosenbrock function"""
    return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1])**2.0)
x0 = array([1.3, 0.7, 0.8, 1.9, 1.2])

res = minimize(rosen, x0)
print(res.x)


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
The default solution for doing numerical linear algebra in Python is NumPy, which provides a low-level infrastructure for the calculations, and SciPy, which has many high-level functions. These are efficient, fast, and quality-checked packages that evolve fast. There are, of course, other options. For instance, Theano allows a 140x faster calculations of certain tensor product operations using GPUs. We focus on NumPy and SciPy for the rest of this tutorial, but do explore other options. We start with importing all functions:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
The last one is worth extra attention: by default, multiplication is element-wise. For a dot product, we must call the dot function:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
There are a few other convenience functions built in the arrays:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
This is a modicum more inconvenient than the MATLAB-style input of vectors. The designers of NumPy could have opted for MATLAB-style input, but this casting of arrays from list is more consistent with the language. We can change the size of Python lists, but not of arrays:
Can you write Python code for it?


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
What is even more important in SciPy is the sparse matrix package. They require some understanding of how different types of sparse matrices are stored, but in exchange, you can tune your algorithm better than in MATLAB or Mathematica.
Can you write Python code for it?

import scipy.sparse as sparse
S = sparse.rand(10, 20, density=0.01)
S

u, s, vt = sparse.linalg.svds(S, 2)
s


I am working on the file "Linear algebra and numerical calculations.ipynb".
The first task is:
With arrays, we can do straightforward numerical operations. These are vectorized, which means they execute extremely fast:
Can you write Python code for it?

print(a+1)
print(2*a)
print(sqrt(a))
print(a*a)


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
$P(Y=k|X=x) \propto \frac{-(x-\mu_k)^2}{2\sigma_k^2} - log(\sigma_k\sqrt{2\pi})$
$-\sum_{x \in X} \frac{-(x-\mu_k)^2}{2\sigma_k^2} - log(\sigma_k\sqrt{2\pi})$
$\bigg|\bigg(\frac{-(x-\mu_k)^2}{2\sigma_k^2} - log(\sigma_k\sqrt{2\pi})\bigg) - \bigg(\frac{-(x-\mu_{k'})^2}
{2\sigma_{k'}^2} - log(\sigma_{k'}\sqrt{2\pi})\bigg)\bigg|$
Can you write Python code for it?

qpredictions = np.array([qmodel.predict(x) for x in iris_test['Sepal.Length']])

plt.figure(figsize=(15, 5))
X_ = np.linspace(0, 20, 200)
iris_training.groupby('Species')['Sepal.Length'].hist()
# iris_test.groupby('Species')['Sepal.Length'].hist()

ax = plt.gca()
ax2 = ax.twinx()
for k in qmodel.K_labels:
    i = qmodel.K_labels.index(k)
    ax2.plot(X_, stats.norm.pdf(X_, loc=qmodel.mu[i], scale=qmodel.sigma[i]),
             label='{0}, $\mu={1}$, $\sigma={2}$'.format(k, qmodel.mu[i], qmodel.sigma[i]), lw=4)

plt.legend(loc=2)
plt.xlim(2, 9)
plt.show()

results = pd.DataFrame(np.array([qpredictions, truth]).T,
                       columns=['Prediction', 'Truth'])
vcounts = results.groupby('Prediction').Truth.value_counts()
vcounts_dense = np.zeros((3,3))
for i in xrange(qmodel.K):
    k_i = qmodel.K_labels[i]
    for j in xrange(qmodel.K):
        k_j = qmodel.K_labels[j]
        try:
            vcounts_dense[i,j] = vcounts[k_i][k_j]
        except KeyError:
            pass
comparison = pd.DataFrame(vcounts_dense, columns=qmodel.K_labels)
comparison['Truth'] = qmodel.K_labels
comparison

c = np.array(zip(qpredictions, truth)).T
float((c[0] == c[1]).sum())/c.shape[1]

Hemocrit = pd.read_csv('data/Hemocrit.csv')

model = lda(Hemocrit.status, Hemocrit.hemocrit)


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
Confusion matrix:
Can you write Python code for it?

predictions = [model.predict(h) for h in Hemocrit.hemocrit]
truth = Hemocrit.status.values
confusion = pd.DataFrame(np.array([predictions, truth]).T, columns=('Prediction', 'Truth'))

confusion.groupby('Prediction').Truth.value_counts()


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
LDA assumes that the variance in each group is the same, and that the predictor(s) are normally distributed for each group. In other words, different $\mu_k$, one shared $\sigma$.
Can you write Python code for it?

X = [np.linspace(0, 7, 50),
     np.linspace(2, 10, 50),
     np.linspace(7, 16, 50)]

plt.figure(figsize=(10, 4))
k = 1
for x in X:
    mu_k = x.mean()
    plt.plot(x, stats.norm.pdf(x, loc=mu_k))
    plt.plot([mu_k, mu_k], [0, 0.5], c='k')
    plt.text(mu_k + 0.2, 0.5, "$\mu_%i$" % k, size=18)
    k += 1
plt.ylim(0, 0.75)

plt.xlabel('Predictor', size=18)
plt.ylabel('Probability', size=18)
plt.show()


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
LDA is like inverted ANOVA: ANOVA looks for differences in a continuous response among categories, whereas LDA infers categories using a continuous predictor.
Can you write Python code for it?

bycategory = [ [Y[i] for i in xrange(len(Y)) if K[i] == k] for k in categories ]

plt.figure(figsize=(10, 5))
plt.subplot(121)
plt.boxplot(bycategory)
plt.ylim(0, 8)
plt.title('ANOVA')

plt.subplot(122)
plt.boxplot(bycategory, 0, 'rs', 0)
plt.title('LDA')
plt.xlim(0, 8)

plt.show()


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
Linear Discriminant Analysis
Predict groupings in continuous data.
Can you write Python code for it?

X = np.linspace(0, 20, 100)
def f(x):
    if x < 7:
        return 'a', 2. + np.random.random()
    elif x < 14:
        return 'b', 4 + np.random.random()
    else:
        return 'c', 6 + np.random.random()
K, Y = zip(*[f(x) for x in X])
colors = plt.get_cmap('Set1')
categories = ['a', 'b', 'c']
plt.scatter(X, Y, c=[colors(categories.index(k)*20) for k in K])
plt.show()


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
Receiver Operator Characteristic (ROC) curve
Provides a visual summary of the confusion matrix over a range of criteria. Given a confusion matrix, $N=TN+FP$, $P=TP+FN$.
Can you write Python code for it?

plt.figure(figsize=(5, 5))
plt.text(0.25, 0.75, 'TN', size=18)
plt.text(0.75, 0.75, 'FP', size=18)
plt.text(0.25, 0.25, 'FN', size=18)
plt.text(0.75, 0.25, 'TN', size=18)

plt.xticks([0.25, 0.75], ['Neg', 'Pos'], size=20)
plt.yticks([0.25, 0.75], ['Pos', 'Neg'], size=20)
plt.ylabel('Truth', size=24)
plt.xlabel('Prediction', size=24)
plt.title('Confusion Matrix', size=26)
plt.show()


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
The default approach was to predict 'Cheat' when $P(Cheater\big|X) > 0.5$.
Can you write Python code for it?

# Histogram of hemocrit values for cheaters and non-cheaters.
Hemocrit[Hemocrit.status == 'Cheat'].hemocrit.hist(histtype='step')
Hemocrit[Hemocrit.status == 'Clean'].hemocrit.hist(histtype='step')
plt.ylim(0, 40)
plt.ylabel('N')

# Probability of being a cheater (or not) as a function of hemocrit.
ax = plt.gca()
ax2 = ax.twinx()

R = np.linspace(0, 100, 500)
post = np.array([model.posterior(r) for r in R])
ax2.plot(R, post[:, 0], label=model.K_labels[0])
ax2.plot(R, post[:, 1], label=model.K_labels[1])
plt.ylabel('P(Y=k)')

plt.xlabel('Hemocrit')
plt.legend()
plt.xlim(40, 60)
plt.title('Criterion: P > 0.5')
plt.show()


I am working on the file "Linear and Quadratic Discriminant Analysis.ipynb".
The first task is:
Trying the same thing, but with QDA:
Can you write Python code for it?

qmodel = qda(Hemocrit.status, Hemocrit.hemocrit)
qpredictions = np.array([qmodel.predict(h) for h in Hemocrit.hemocrit])
truth = Hemocrit.status.values
qconfusion = pd.DataFrame(np.array([qpredictions, truth]).T, columns=('Prediction', 'Truth'))


I am working on the file "Linear models.ipynb".
The first task is:
Exercises

Compare  Logistic regression with l1 penalty and l2 penalty by plotting the coefficients as above for the digits dataset. Classify odd vs even digits to make it a binary task.
Can you write Python code for it?

y % 2

# %load solutions/linear_models.py


I am working on the file "Linear models.ipynb".
The first task is:
Lasso (L1 penalty)
$$ \text{min}_{w, b} \sum_i || w^\mathsf{T}x_i + b  - y_i||^2  + \alpha ||w||_1$$
Can you write Python code for it?

from sklearn.linear_model import Lasso

lasso_models = {}
training_scores = []
test_scores = []

for alpha in [30, 10, 1, .01]:
    lasso = Lasso(alpha=alpha).fit(X_train, y_train)
    training_scores.append(lasso.score(X_train, y_train))
    test_scores.append(lasso.score(X_test, y_test))
    lasso_models[alpha] = lasso
plt.figure()
plt.plot(training_scores, label="training scores")
plt.plot(test_scores, label="test scores")
plt.xticks(range(4), [30, 10, 1, .01])
plt.legend(loc="best")

plt.figure(figsize=(10, 5))
plt.plot(true_coefficient[coefficient_sorting], "o", label="true", c='b')

for i, alpha in enumerate([30, 10, 1, .01]):
    plt.plot(lasso_models[alpha].coef_[coefficient_sorting], "o", label="alpha = %.2f" % alpha, c=plt.cm.summer(i / 3.))

plt.legend(loc="best")


I am working on the file "Linear models.ipynb".
The first task is:
Linear models for regression
y_pred = x_test[0] * coef_[0] + ... + x_test[n_features-1] * coef_[n_features-1] + intercept_
Can you write Python code for it?

from sklearn.datasets import make_regression
from sklearn.cross_validation import train_test_split

X, y, true_coefficient = make_regression(n_samples=80, n_features=30, n_informative=10, noise=100, coef=True, random_state=5)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=5)
print(X_train.shape)
print(y_train.shape)


I am working on the file "Linear models.ipynb".
The first task is:
Multi-Class linear classification
Can you write Python code for it?

from sklearn.datasets import make_blobs
plt.figure()
X, y = make_blobs(random_state=42)
plt.scatter(X[:, 0], X[:, 1], c=y)

from sklearn.svm import LinearSVC
linear_svm = LinearSVC().fit(X, y)
print(linear_svm.coef_.shape)
print(linear_svm.intercept_.shape)

plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=y)
line = np.linspace(-15, 15)
for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):
    plt.plot(line, -(line * coef[0] + intercept) / coef[1])
plt.ylim(-10, 15)
plt.xlim(-10, 8)


I am working on the file "Linear vs Nonlinear Least Squares.ipynb".
The first task is:
NOW, let's assume that instead of fitting a line we wanted to fit a parabola. This is still a linear least squares problem. That's because linear least squares only requires that the function being fit is linear in its parameters. We will look more at what that means below. Let's take a general quadratic equation.
$$\alpha x_i^2 + \beta x_i + \gamma = y_i$$
Now we have three degrees of freedom and must fit all 3 parameters. We pose this problem the same way as above. We want the quadratic equation to satisfy all of the points $(x_i,y_i)$ simultaneously. We want $\alpha$, $\beta$, and $\gamma$ such that all of the below equations are true.
$$\begin{array}{lcl} \alpha (1)^2+\beta(1)+\gamma & = & 2 \ \alpha(2)^2+ \beta(2) + \gamma & = & 1 \ \alpha(2.5)^2+ \beta(2.5) + \gamma & = & 2 \end{array}$$
In matrix form...
$$\begin{bmatrix} 1 & 1 & 1 \ 4 & 2 & 1 \ 6.25 & 2.5 & 1 \end{bmatrix} \begin{bmatrix} \alpha \ \beta \ \gamma \end{bmatrix} = \begin{bmatrix} 2 \ 1 \ 2 \end{bmatrix}$$
This time, there does exist a unique solution. A quadratic equation has 3 degrees of freedom and there are 3 constraints posed. Our good friend Gauss proved that $n$ distinct points uniquely define a polynomial of degree $n-1$. So we will find the "best" solution using the above technique and show that the error is zero.
Can you write Python code for it?

A = np.array(((1,1,1),(4,2,1),(6.25,2.5,1))) # The matrix for our new 3x3 system of equations.
x = np.dot(np.dot(np.linalg.inv(np.dot(A.T, A)), A.T), b) # Project b onto Col(A)
error = np.linalg.norm( np.dot(A,x) - b )
print "Error = ", error


I am working on the file "Linear vs Nonlinear Least Squares.ipynb".
The first task is:
Now we look at the resulting parabola and see that it passes through all 3 points.
Can you write Python code for it?

yvals = x[0]*xvals*xvals + x[1]*xvals + x[2] # All y values for the equation of the line
plt.scatter((1,2,2.5), (2,1,2)); plt.plot(xvals,yvals); plt.xlim((0,3)); plt.ylim((0,3));


I am working on the file "Linear-SVM.ipynb".
The first task is:
Analytic Expressions
Let $X \in R^{n \times d+1}$ and $y = (y_1,...,y_n)^T \in R^{n+1}$ and $\texttt{loss}(...) \ge 0$
Objective function:
$$F(w) = \|w\|^2 + \frac Cn \|\texttt{loss}(y,Xw)\|_1$$
Clearly,
$$(\vec\nabla F(w))j = 2w_j + \frac Cn \sum^{n}{i=1}\frac{d}{d(Xw)i}\texttt{loss}(y_i,(Xw)_i) \cdot X{i,j} \quad \texttt{for} ~j = 1,2,...,d+1$$
For Hinge Loss:
$$l_{hinge}(y,t) := \max(0, 1 - yt)$$
Then
$$\frac{d}{dt}l_{hinge}(y,t) := \begin{cases} 0, & \mbox{if } 1-yt \lt 0\
-y, & \mbox{if } 1-yt \gt 0  \end{cases}$$
And
$$F(w) = \|w\|^2 + \frac Cn \sum^n\max(0, 1 - y(Xw))$$
And $\texttt{for j = 1,2,...,d+1}$
$$(\vec\nabla F(w))_j = \begin{cases} 2w_j, & \mbox{if } 1-y(Xw) \lt 0\
2w_j + \frac Cn \sum^{n}{i=1} -y \cdot X{i,j}, & \mbox{if } 1-y(Xw) \gt 0  \end{cases}$$
Where $y(Xw) \in R^{n}$

For Square Hinge Loss:
$$l_{square-hinge}(y,t) := \max(0, 1 - yt)^2$$
Then
$$\frac{d}{dt}l_{square-hinge}(y,t) := \begin{cases} 0, & \mbox{if } 1-yt \lt 0\
2(1 - yt)(-y), & \mbox{if } 1-yt \gt 0  \end{cases}$$
And
$$F(w) = \|w\|^2 + \frac Cn \sum^n\max(0, 1 - y(Xw))^2$$
And $\texttt{for j = 1,2,...,d+1}$
$$(\vec\nabla F(w))_j = \begin{cases} 2w_j, & \mbox{if } 1-y(Xw) \le 0\
2w_j + \frac Cn \sum^{n}_{i=1} 2(1 - y(Xw))(-y) \cdot X_{i,j}, & \mbox{if } 1-y(Xw) \gt 0  \end{cases}$$
Where $y*(Xw) \in R^{n}$

For Huberized Hinge Loss:
$$l_{huber-hinge}(y,t) := \begin{cases} 0, & \mbox{if } yt \gt 1+h\
\frac{(1+h-yt)^2}{4h}, & \mbox{if } |1-yt| \le h \
1-yt, & \mbox{if } yt \lt 1-h \end{cases}$$
$$\frac{d}{dt}l_{huber-hinge}(y,t) := \begin{cases} 0, & \mbox{if } yt \gt 1+h\
\frac{2(1+h-yt)(-y)}{4h}, & \mbox{if } |1-yt| \le h \
-y, & \mbox{if } yt \lt 1-h \end{cases}$$
We have continuity in $\frac{d}{dt}l_{huber-hinge}(y,t)$ since the derivatives limits' agree at the critical points;
$$ \lim_{t^+ \rightarrow \frac{1+h}{y} } \frac{d}{dt}l_{huber-hinge}(y,t) = 0$$
$$ \lim_{t^- \rightarrow \frac{1+h}{y} } \frac{d}{dt}l_{huber-hinge}(y,t) = \lim_{t^- \rightarrow \frac{1+h}{y} } \frac{2(1+h-yt)(-y)}{4h} = \lim_{t^-...
Can you write Python code for it?

from sklearn.datasets import make_regression
from sklearn.cross_validation import train_test_split
from sklearn.kernel_ridge import KernelRidge
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.svm import LinearSVC
from sklearn.linear_model import SGDClassifier
import matplotlib.pyplot as plt
import sys
import numpy as np
from numpy.linalg import norm
from numpy.random import randint
import random
import math
import time

def loss_hinge(y,t): # const
    return max(0,1-(y*t))

def loss_hinge_der(y,t): # const
    if 1-(y*t) <= 0:
        return 0
    return -y

def loss_huber_hinge(y,t,h): # const
    if y*t > 1+h:
        return 0
    if abs(1-(y*t)) <= h:
        return pow((1+h-(y*t)),2)/(4*h)
    if y*t < 1-h:
        return 1 - (y*t)

def loss_huber_hinge_der(y,t,h): # const
    if y*t > 1+h:
        return 0
    if abs(1-(y*t)) <= h:
        return 2*(1+h-(y*t))*(-y)/(4*h)
    if y*t < 1-h:
        return -y

def loss(y,t,loss_type,h): # const
    if loss_type=='hinge':
        return loss_hinge(y,t)
    if loss_type=='modified_huber':
        return loss_huber_hinge(y,t,h)

def loss_der(y,t,loss_type,h): # const
    if loss_type=='hinge':
        return loss_hinge_der(y,t)
    if loss_type=='modified_huber':
        return loss_huber_hinge_der(y,t,h)

def compute_obj(w,C,X,y,loss_type,h): # const
    ret = 0.0
    assert len(X)==len(y)
    assert len(X[0])==len(w)
    for i in range(len(X)):
        ret += loss(y[i],np.dot(X[i],w),loss_type,h)
    return norm(w)**2 + C*ret/len(X)

def compute_grad(w,C,X,y,loss_type,h): # const
    if len(X)==len(w):
        grad = 2*w.copy()
        for i in range(len(w)):
            grad[i] += C*(loss_der(y,np.dot(X,w),loss_type,h)*X[i])
        return grad
    if len(X)==len(y) and len(X[0])==len(w):
        n=len(X)
        X[n-1,len(w)-1]
        grad = 2*w.copy()
        for i in range(len(w)):
            loss_sum = 0.0
            for j in range(n):
                loss_sum += \
...


I am working on the file "Linear-SVM.ipynb".
The first task is:
Here we notice that non-stochastic descent walks straight to the solution, but stochastic descent randomly walks around, eventually making it.
Can you write Python code for it?

%matplotlib nbagg
plt.clf()
plt.cla()

ax = plt.subplot(1,1,1)

obj_array = np.asarray(obj_array)
ax.scatter(range(1,len(obj_array)+1),obj_array,marker='^',label='Non-Stochastic')

obj_stoch_array = np.asarray(obj_stoch_array)
ax.scatter(range(1,len(obj_stoch_array)+1),obj_stoch_array,marker='*',label='Stochastic')

handles, labels = ax.get_legend_handles_labels()
plt.legend(handles, labels)
plt.title('Objective Function over iterations')
plt.ylabel('F (w)')
plt.xlabel('Iteration')


I am working on the file "Linear-SVM.ipynb".
The first task is:
Here we notice that training error is lower than test error, that is to be expected. And stochastic error is higher than non-stochastic error, but they both converge.
Can you write Python code for it?

%matplotlib nbagg
plt.clf()
plt.cla()

ax = plt.subplot(1,1,1)

# print w
x_plot=[w[0], 0]
y_plot=[w[1], 0]
ax.plot(x_plot,y_plot)

# print clf.coef_[0]
x_plot=[clf.coef_[0][0], 0]
y_plot=[clf.coef_[0][1], 0]
ax.plot(x_plot,y_plot)

ax.scatter((X[y==0])[:,0],(X[y==0])[:,1],marker='*')
ax.scatter((X[y==1])[:,0],(X[y==1])[:,1],marker='^')

handles, labels = ax.get_legend_handles_labels()
plt.title('Data, Scikit-learn Hyperplane, and Our Own Hyperplane')
plt.ylabel('y')
plt.xlabel('x')


I am working on the file "Linear-SVM.ipynb".
The first task is:
SVM Gradient Descent Usage Example
Can you write Python code for it?

# Generate data
'''Generate 2 Gaussians samples with the same covariance matrix'''
n, dim = 500, 2
np.random.seed(0)
C = np.array([[0., -0.23], [0.83, .23]])
gap = 1
X = np.r_[np.dot(np.random.randn(n, dim)+gap, C),
          np.dot(np.random.randn(n, dim)-gap, C)]

# append constant dimension
X = np.column_stack((X, np.ones(X.shape[0])))

y = np.hstack((-1*np.ones(n), np.ones(n)))

assert len(X[y==-1])==len(y[y==-1]);assert len(X[y==1])==len(y[y==1]);assert len(X)==len(y)

X_train, X_test, y_train, y_test = train_test_split(X, y,
                            test_size=0.5, random_state=20140210)

assert len(X_train)>1;assert len(X_test)>1;assert len(X_train)==len(y_train);assert len(X_test)==len(y_test)

max_iter=1000
C=1.0

for loss_type in ['modified_huber','hinge']:
    for h_index in range(1,10):
        h=(.1*(1.1**h_index))
        print 'parameters: loss_type',loss_type,'h',h

        w, w_array, obj_array, training_error_array, testing_error_array = my_svm(X_train, y_train,
                                                                                loss_type=loss_type,
                                                                                max_iter=max_iter,h=h,C=C,
                                                                                X_test=X_test, y_test=y_test,
                                                                                stochastic=False)
        print 'Custom w =',w,' test score = ',score(X_test, y_test, w=w)

        clf = SGDClassifier(loss=loss_type, penalty="l2",alpha=1/C, fit_intercept=False)
        clf.fit(X_train, y_train);        assert clf.intercept_ == 0
        print 'SGDClassifier w = ',clf.coef_[0],' test score = ',score(X_test, y_test,
                                                                       w=clf.coef_[0])
        clf = LinearSVC( penalty="l2",C=C, fit_intercept=False); clf.fit(X_train, y_train); assert clf.intercept_ == 0
        print 'LinearSVC w = ',clf.coef_[0],' test score = ',score(X_test, y_test,...


I am working on the file "Linear-SVM.ipynb".
The first task is:
SVM Stochastic Gradient Descent Usage Example
Can you write Python code for it?

# Generate data
'''Generate 2 Gaussians samples with the same covariance matrix'''
n, dim = 2*(10**7), 2
n, dim = 500, 2
np.random.seed(0)
C = np.array([[0., -0.23], [0.83, .23]])
gap = 1
X = np.r_[np.dot(np.random.randn(n, dim)+gap, C),
          np.dot(np.random.randn(n, dim)-gap, C)]

# append constant dimension
X = np.column_stack((X, np.ones(X.shape[0])))

y = np.hstack((-1*np.ones(n), np.ones(n)))

assert len(X[y==-1])==len(y[y==-1]);assert len(X[y==1])==len(y[y==1]);assert len(X)==len(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=20140210)

assert len(X_train)>1;assert len(X_test)>1;assert len(X_train)==len(y_train);assert len(X_test)==len(y_test)

max_iter=1000
C=1.0

for loss_type in ['modified_huber','hinge']:
    for h_index in range(1,10):
        h=(.1*(1.1**h_index))
        print 'parameters: loss_type',loss_type,'h',h
        w, w_stoch_array, obj_stoch_array, training_error_stoch_array, testing_error_stoch_array = \
        my_svm(X_train,
            y_train, loss_type=loss_type, max_iter=max_iter,h=h,C=C, X_test=X_test, y_test=y_test,stochastic=True)

        print 'Custom w =',w,' test score = ',score(X_test, y_test, w=w)

        clf = SGDClassifier(loss=loss_type, penalty="l2",alpha=1/C, fit_intercept=False);clf.fit(X_train, y_train)
        assert clf.intercept_ == 0
        print 'SGDClassifier w = ',clf.coef_[0],' test score = ',score(X_test, y_test, w=clf.coef_[0])

        clf = LinearSVC( penalty="l2",C=C, fit_intercept=False)
        clf.fit(X_train, y_train)
        assert clf.intercept_ == 0
        print 'LinearSVC w = ',clf.coef_[0],' test score = ',score(X_test, y_test, w=clf.coef_[0])
        print
        print
#         break
#     break

%matplotlib nbagg
plt.clf()
plt.cla()

ax = plt.subplot(1,1,1)

w_array = np.asarray(w_array)
ax.scatter(w_array[:,0],w_array[:,1],marker='^',label='Non-Stochastic')

w_stoch_array =...


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Choice of two projections
It turns out our implementation has a nasty deficiency.
Can you write Python code for it?

qr_test(qr_householder1, numpy.eye(1))

qr_test(qr_householder1, numpy.eye(3,2))


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Classical Gram-Schmidt is highly parallel, but unstable, as evidenced by the lack of orthogonality in $Q$.
Right-looking algorithms
The implementations above have been "left-looking"; when working on column $i$, we compare it only to columns to the left (i.e., $j < i$).  We can reorder the algorithm to look to the right by projecting $q_i$ out of all columns $j > i$.  This algorithm is stable while being just as parallel as gram_schmidt_classical.
Can you write Python code for it?

def gram_schmidt_modified(X):
    Q = X.copy()
    R = numpy.zeros((len(X.T), len(X.T)))
    for i in range(len(Q.T)):
        R[i,i] = numpy.linalg.norm(Q[:,i])
        Q[:,i] /= R[i,i]
        R[i,i+1:] = Q[:,i+1:].T.dot(Q[:,i])
        Q[:,i+1:] -= numpy.outer(Q[:,i], R[i,i+1:])
    return Q, R

qr_test(gram_schmidt_modified, V)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Cost of Householder factorization
The dominant cost comes from the line
Python
    R[i:,i:] -= 2 * numpy.outer(v, v.dot(R[i:,i:]))
were R[i:,i:] is an $(m-i)\times(n-i)$ matrix.
This line performs $2(m-i)(n-i)$ operations in v.dot(R[i:,i:]), another $(m-i)(n-i)$ in the "outer" product and again in subtraction.  As written, multiplication by 2 would be another $(m-i)(n-i)$ operations, but is only $m-i$ operations if we rewrite as
Python
    w = 2*v
    R[i:,i:] -= numpy.outer(w, v.dot(R[i:,i:]))
in which case the leading order cost is $4(m-i)(n-i)$.  To compute the total cost, we need to sum over all columns $i$,
$$\begin{split} \sum_{i=1}^n 4(m-i)(n-i) &= 4 \Big[ \sum_{i=1}^n (m-n)(n-i) + \sum_{i=1}^n (n-i)^2 \Big] \
&= 4 (m-n) \sum_{i=1}^n i + 4 \sum_{i=1}^n i^2 \
&\approx 2 (m-n) n^2 + 4 n^3/3 \
&= 2 m n^2 - \frac 2 3 n^3 .
\end{split}$$
Recall that Gram-Schmidt QR cost $2 m n^2$, so Householder costs about the same when $m \gg n$ and is markedly less expensive when $m \approx n$.
Least squares and the normal equations
A least squares problem takes the form: given an $m\times n$ matrix $A$ ($m \ge n$), find $x$ such that
$$ \lVert Ax - b \rVert $$
is minimized.  If $A$ is square and full rank, then this minimizer will satisfy $A x - b = 0$, but that is not the case in general because $b$ is not in the range of $A$.
The residual $A x - b$ must be orthogonal to the range of $A$.

Is this the same as saying $A^T (A x - b) = 0$?
If $QR = A$, is it the same as $Q^T (A x - b) = 0$?

In HW2, we showed that $QQ^T$ is an orthogonal projector onto the range of $Q$.  If $QR = A$,
$$ QQ^T (A x - b) = QQ^T(Q R x - b) = Q (Q^T Q) R x - QQ^T b = QR x - QQ^T b = A x - QQ^T b . $$
So if $b$ is in the range of $A$, we can solve $A x = b$.  If not, we need only orthogonally project $b$ into the range of $A$.
Solution by QR (Householder)
Solve $R x = Q^T b$.

QR factorization costs $2 m n^2 - \frac 2 3 n^3$ operations and is done once per matrix $A$.
Computing $Q^T b$ costs $4 (m-n)n + 2 n^2 = 4 mn - 2n^2$ (using the...
Can you write Python code for it?

# Test accuracy of solver for an ill-conditioned square matrix

x = numpy.linspace(-1,1,19)
A = numpy.vander(x)
print('cond(A) = ',numpy.linalg.cond(A))
Q, R = numpy.linalg.qr(A)
print('cond(R^{-1} Q^T A) =', numpy.linalg.cond(numpy.linalg.solve(R, Q.T.dot(A))))
L = numpy.linalg.cholesky(A.T.dot(A))
print('cond(L^{-T} L^{-1} A^T A) =', numpy.linalg.cond(numpy.linalg.solve(L.T, numpy.linalg.solve(L, A.T.dot(A)))))


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Evidently $p(x) = 12.983 x^3 - 1.748 x^2 - 9.476 x + 3.352$ is the unique cubic polynomial that interpolates those points.
Applying $B^{-1}$ converted from the values at the marked points to the polynomial coefficients.
Inner products and orthogonality
The inner product
$$ x^T y = \sum_i x_i y_i $$
of vectors (or columns of a matrix) tell us about their magnitude and about the angle.
The norm is induced by the inner product,
$$ \lVert x \rVert = \sqrt{x^T x} $$
and the angle $\theta$ is defined by
$$ \cos \theta = \frac{x^T y}{\lVert x \rVert \, \lVert y \rVert} . $$
Inner products are bilinear, which means that they satisfy some convenient algebraic properties
$$ \begin{split}
(x + y)^T z &= x^T z + y^T z \
x^T (y + z) &= x^T y + x^T z \
(\alpha x)^T (\beta y) &= \alpha \beta x^T y \
\end{split} . $$
The pairwise inner products between two sets of vectors can be expressed by collecting the sets as columns in matrices and writing $A = X^T Y$ where $A_{i,j} = x_i^T y_j$.
It follows from this definition that
$$ (X^T Y)^T = Y^T X .$$
Orthogonal matrices
If $x^T y = 0$ then we say $x$ and $y$ are orthogonal (or "$x$ is orthogonal to $y$").
A vector is said to be normalized if $\lVert x \rVert = 1$.
If $x$ is orthogonal to $y$ and $\lVert x \rVert = \lVert y \rVert = 1$ then we say $x$ and $y$ are orthonormal.
A matrix with orthonormal columns is said to be an orthogonal matrix.
We typically use $Q$ or $U$ and $V$ for matrices that are known/constructed to be orthogonal.
Orthogonal matrices are always full rank -- the columns are linearly independent.
The inverse of a square orthogonal matrix is its transpose:
$$ Q^T Q = Q Q^T = I . $$
Orthogonal matrices are a powerful building block for robust numerical algorithms.
Can you write Python code for it?

# Make some polynomials
q0 = A.dot(numpy.array([0,0,0,.5]))  # .5
q1 = A.dot(numpy.array([0,0,1,0]))  # x
q2 = A.dot(numpy.array([0,1,0,0]))  # x^2

pyplot.figure()
pyplot.plot(x, numpy.array([q0, q1, q2]).T)

# Inner products of even and odd functions

q0 = q0 / numpy.linalg.norm(q0)
q1.dot(q0), q2.dot(q0), q2.dot(q1)

# What is the constant component of q2?

pyplot.figure()
pyplot.plot(x, q2.dot(q0)*q0)

# Let's project that away so that q2 is orthogonal to q0

q2 = q2 - q2.dot(q0)*q0

Q = numpy.array([q0, q1, q2]).T
print(Q.T.dot(Q))
pyplot.figure()
pyplot.plot(x, Q)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Gram-Schmidt Orthogonalization
Given a collection of vectors (columns of a matrix), we can find an orthogonal basis by applying the above procedure one column at a time.
Can you write Python code for it?

def gram_schmidt_naive(X):
    Q = numpy.zeros_like(X)
    R = numpy.zeros((len(X.T),len(X.T)))
    for i in range(len(Q.T)):
        v = X[:,i].copy()
        for j in range(i):
            r = v.dot(Q[:,j])
            R[j,i] = r
            v -= r * Q[:,j]    # "modified Gram-Schmidt" - remove each component before next dot product
        R[i,i] = numpy.linalg.norm(v)
        Q[:,i] = v / R[i,i]
    return Q, R

Q, R = gram_schmidt_naive(A)
print(Q.T.dot(Q))
print(numpy.linalg.norm(Q.dot(R)-A))
pyplot.figure()
pyplot.plot(x, Q)

Q, R = gram_schmidt_naive(numpy.vander(x, 4, increasing=True))
pyplot.figure()
pyplot.plot(x, Q)


def gram_schmidt_naive(X):
    Q = numpy.zeros_like(X)
    R = numpy.zeros((len(X.T),len(X.T)))
    for i in range(len(Q.T)):
        v = X[:,i].copy()
        for j in range(i):
            r = v.dot(Q[:,j])
            R[j,i] = r
            v -= r * Q[:,j]    # "modified Gram-Schmidt" - remove each component before next dot product
        R[i,i] = numpy.linalg.norm(v)
        Q[:,i] = v / R[i,i]
    return Q, R

x = numpy.linspace(-1,1,50)
k = 6
A = numpy.vander(x, k, increasing=True)
Q, R = gram_schmidt_naive(A)
print(numpy.linalg.norm(Q.T.dot(Q) - numpy.eye(k)))
print(numpy.linalg.norm(Q.dot(R)-A))
pyplot.figure()
pyplot.plot(x, Q)
A.shape

Q, R = gram_schmidt_naive(numpy.vander(x, 4, increasing=True))
pyplot.figure()
pyplot.plot(x, Q)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Householder triangularization
Gram-Schmidt methods perform triangular transformations to build an orthogonal matrix.  As we have seen, $X = QR$ is satisfied accurately, but $Q$ may not be orthogonal when $X$ is ill-conditioned.  Householder triangularization instead applies a sequence of orthogonal transformations to build a triangular matrix.
$$ \underbrace{Q_{n-1} \dotsb Q_0}_{Q^T} A = R $$
The structure of the algorithm is
$$ \underbrace{\begin{bmatrix} * & * & * \ * & * & * \ * & * & * \ * & * & * \ * & * & * \ \end{bmatrix}}{A} \to
\underbrace{\begin{bmatrix} * & * & * \ 0 & * & * \ 0 & * & * \ 0 & * & * \ 0 & * & * \ \end{bmatrix}}{Q_0 A} \to
\underbrace{\begin{bmatrix} * & * & * \ 0 & * & * \ 0 & 0 & * \ 0 & 0 & * \ 0 & 0 & * \ \end{bmatrix}}{Q_1 Q_0 A} \to
\underbrace{\begin{bmatrix} * & * & * \ 0 & * & * \ 0 & 0 & * \ 0 & 0 & 0 \ 0 & 0 & 0 \ \end{bmatrix}}{Q_2 Q_1 Q_0 A}
$$
where the elementary orthogonal matrices $Q_i$ chosen to introduce zeros below the diagonal in the $i$th column of $R$.
Each of these transformations will have the form
$$Q_i = \begin{bmatrix} I_i & 0 \ 0 & F \end{bmatrix}$$
where $F$ is a "reflection" that achieves
$$ F x = \begin{bmatrix} \lVert x \rVert \ 0 \ 0 \ \vdots \end{bmatrix} $$
where $x$ is the column of $R$ from the diagonal down.
This transformation is a reflection across a plane with normal $v = Fx - x = \lVert x \rVert e_1 - x$.

The reflection, as depected above by Trefethen and Bau (1999) can be written $F = I - 2 \frac{v v^T}{v^T v}$.
Can you write Python code for it?

def householder_Q_times(V, x):
    """Apply orthogonal matrix represented as list of Householder reflectors"""
    y = x.copy()
    for i in reversed(range(len(V))):
        y[i:] -= 2 * V[i] * V[i].dot(y[i:])
    return y

def qr_householder1(A):
    "Compute QR factorization using naive Householder reflection"
    m, n = A.shape
    R = A.copy()
    V = []
    for i in range(n):
        x = R[i:,i]
        v = -x
        v[0] += numpy.linalg.norm(x)
        v = v/numpy.linalg.norm(v)     # Normalized reflector plane
        R[i:,i:] -= 2 * numpy.outer(v, v.dot(R[i:,i:]))
        V.append(v)                    # Storing reflectors is equivalent to storing orthogonal matrix
    Q = numpy.eye(m, n)
    for i in range(n):
        Q[:,i] = householder_Q_times(V, Q[:,i])
    return Q, numpy.triu(R[:n,:])

qr_test(qr_householder1, numpy.array([[1.,2],[3,4],[5,6]]))

qr_test(qr_householder1, V)
qr_test(numpy.linalg.qr, V)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Inside qr_householder1, we have the lines
x = R[i:,i]
        v = -x
        v[0] += numpy.linalg.norm(x)
        v = v/numpy.linalg.norm(v)     # Normalized reflector plane
What happens when $$x = \begin{bmatrix}1 \ 0 \end{bmatrix}$$
(i.e., the column of $R$ is already upper triangular)?
We are trying to define a reflector plane (via its normal vector) from the zero vector,
$$v = \lVert x \rVert e_0 - x .$$
When we try to normalize this vector, we divide zero by zero and the algorithm breaks down (nan).  Maybe we just need to test for this special case and "skip ahead" when no reflection is needed?  And if so, how would we define $Q$?
Can you write Python code for it?

qr_test(qr_householder1, numpy.array([[1.,1], [2e-8,1]]))
print(qr_householder1(numpy.array([[1.,1], [2e-8,1]])))


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Now suppose we know the value of a polynomial at a few points.
We can use the Vandermonde matrix to find a polynomial through those points.
Can you write Python code for it?

x1 = numpy.array([-0.9, 0.1, 0.5, 0.8])  # points where we know values
y = numpy.array([1, 2.4, -0.2, 1.3])     # values at those points
pyplot.figure()
pyplot.plot(x1, y, '*')
B = numpy.vander(x1)                     # Vandermonde matrix at the known points
p = numpy.linalg.solve(B, y)             # Compute the polynomial coefficients
print(p)
pyplot.plot(x, A.dot(p))                 # Plot the polynomial evaluated at all points
print('B =', B, '\np =', p)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
QR blocking
Applying the orthogonal factor $Q$, both during factorization and on each iteration, is sequential.  For matrix-matrix multiplication, we found that blocking was extremely important for performance.  The same is true here, but we will need a transform.  We would like to represent a sequence of elementary reflectors as a single block reflector:
$$\begin{gather} (I - v_0 \tau_0 v_0^T) (I - v_1 \tau_1 v_1^T) &= I - v_0 \tau_0 v_0^T + v_0 \tau_0 v_0^T v_1 \tau_1 v_1^T - v_1 \tau_1 v_1^T \
&= I - V \begin{bmatrix} \tau_0 & \tau_{01} \ 0 & \tau_1 \end{bmatrix} V^T \end{gather} $$
where $V = [ v_0 | v_1 ]$ is an $m\times 2$ matrix and $\tau_{01} = -\tau_0 v_0^T v_1 \tau_1$. This representation can be created recursively and is called the "compact WY" representation of $Q$ and can be computed using LAPACK's dgeqrt2.
Can you write Python code for it?

class QRT:
    def __init__(self, F, T):
        self.F = F
        self.T = T
        self.shape = self.F.shape
    def R(self):
        n = len(self.T)
        return numpy.triu(self.F[:n,:])
    def Qdot(self, x):
        m, n = self.shape
        k = x.shape[1] if len(x.shape) == 2 else 1
        if x.shape[0] not in {n, m}:
            raise ValueError("operands could not be multiplied with shapes", self.shape, x.shape)
        y = numpy.zeros((m, k))
        y[:x.shape[0]] = x
        D = numpy.tril(self.F[:n])
        numpy.fill_diagonal(D, 1)
        tmp = D.T.dot(x[:n])
        if x.shape[0] > n:
            tmp += self.F[n:].T.dot(x[n:])
        tmp = self.T.dot(tmp)
        y[:n] -= D.dot(tmp)
        y[n:] -= self.F[n:].dot(tmp)
        return y

def qr_householder_wy(A):
    "Compute QR factorization using blocked Householder reflection, result in compact WY"
    m, n = A.shape
    F = A.copy()
    T = numpy.zeros((n, n))
    for i in range(n):
        v = F[i:,i] # in-place
        d = v[1:].dot(v[1:])
        norm = numpy.sqrt(v[0]**2 + d)
        Rii = -numpy.copysign(norm, v[0]) # Choose the further of the two reflections
        v[0] -= Rii
        tau = 2 * v[0]**2 / (v[0]**2 + d)
        v /= v[0]
        # Update the remaining panel
        F[i:,i+1:] -= tau * numpy.outer(v, v.dot(F[i:,i+1:]))
        T[i,i] = tau
        if i > 0: # Add this column to T
            T[:i,i] = -tau * T[:i,:i].dot(F[i:,:i].T.dot(v))
        F[i,i] = Rii
    return QRT(F, T)

m, n = 5, 4
A = vander_cheb(cosspace(-1,1,m), n)
qrt = qr_householder_wy(A)
qr = qr_householder_inplace(A)
I = numpy.eye(n,n)
print(numpy.linalg.norm(qrt.R() - qr.R()), numpy.linalg.norm(qrt.Qdot(I) - qr.Qdot(I)))


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Relative condition number
Given the relative nature of floating point arithmetic, it is more useful to discuss relative condition number,
$$ \kappa = \max_{\delta x} \frac{|\delta f|/|f|}{|\delta x|/|x|}
= \max_{\delta x} \Big[ \frac{|\delta f|/|\delta x|}{|f| / |x|} \Big] $$
or, if $f$ is differentiable,
$$ \kappa = \max_{\delta x} |f'(x)| \frac{|x|}{|f|} . $$
How does a condition number get big?
Take-home message
The relative accuracy of the best-case algorithm will not be reliably better than $\epsilon_{\text{machine}}$ times the condition number.
$$ \max_{\delta x} \frac{|\delta f|}{|f|} \ge \kappa \cdot \epsilon_{\text{machine}} $$
Can you write Python code for it?

numpy.log(1 + 1e-10)  - numpy.log1p(1e-10)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Some common terminology

The range of $A$ is the space spanned by its columns.  This definition coincides with the range of a function $f(x)$ when $f(x) = A x$.
The nullspace of $A$ is the space of vectors $x$ such that $A x = 0$.
The rank of $A$ is the dimension of its range.
A matrix has full rank if the nullspace of either $A$ or $A^T$ is empty (only the 0 vector).  Equivalently, if all the columns of $A$ (or $A^T$) are linearly independent.
A nonsingular (or invertible) matrix is a square matrix of full rank.  We call the inverse $A^{-1}$ and it satisfies $A^{-1} A = A A^{-1} = I$.

$\DeclareMathOperator{\rank}{rank} \DeclareMathOperator{\null}{null} $
If $A \in \mathbb{R}^{m\times m}$, which of these doesn't belong?
1. $A$ has an inverse $A^{-1}$
2. $\rank (A) = m$
3. $\null(A) = {0}$
4. $A A^T = A^T A$
5. $\det(A) \ne 0$
6. $A x = 0$ implies that $x = 0$
When we write $x = A^{-1} y$, we mean that $x$ is the unique vector such that $A x = y$.
(It is rare that we explicitly compute a matrix $A^{-1}$, though it's not as "bad" as people may have told you.)
A vector $y$ is equivalent to $\sum_i e_i y_i$ where $e_i$ are columns of the identity.
Meanwhile, $x = A^{-1} y$ means that we are expressing that same vector $y$ in the basis of the columns of $A$, i.e., $\sum_i A_{:,i} x_i$.
Can you write Python code for it?

B = numpy.array([[2, 3],[0, 4]])
print(B)
print(B.dot(B.T), B.T.dot(B))
Binv = numpy.linalg.inv(B)
Binv.dot(B), B.dot(Binv)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
The error $QR - A$ is still $10^{-8}$ for this very well-conditioned matrix so something else must be at play here.
Can you write Python code for it?

def qr_householder2(A):
    "Compute QR factorization using Householder reflection"
    m, n = A.shape
    R = A.copy()
    V = []
    for i in range(n):
        v = R[i:,i].copy()
        v[0] += numpy.sign(v[0])*numpy.linalg.norm(v) # Choose the further of the two reflections
        v = v/numpy.linalg.norm(v)     # Normalized reflector plane
        R[i:,i:] -= 2 * numpy.outer(v, v.dot(R[i:,i:]))
        V.append(v)                    # Storing reflectors is equivalent to storing orthogonal matrix
    Q = numpy.eye(m, n)
    for i in range(n):
        Q[:,i] = householder_Q_times(V, Q[:,i])
    return Q, numpy.triu(R[:n,:])

qr_test(qr_householder2, numpy.eye(3,2))
qr_test(qr_householder2, numpy.array([[1.,1], [1e-8,1]]))
print(qr_householder2(numpy.array([[1.,1], [1e-8,1]])))

qr_test(qr_householder2, V)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Theorem: all full-rank $m\times n$ matrices ($m \ge n$) have a unique $Q R$ factorization with $R_{j,j} > 0$.
Absolute condition number
Consider a function $f: X \to Y$ and define the absolute condition number
$$ \hat\kappa = \lim_{\delta \to 0} \max_{|\delta x| < \delta} \frac{|f(x + \delta x) - f(x)|}{|\delta x|} = \max_{\delta x} \frac{|\delta f|}{|\delta x|}. $$
If $f$ is differentiable, then $\hat\kappa = |f'(x)|$.
Floating point arithmetic
Floating point arithmetic $x \circledast y := \text{float}(x * y)$ is exact within a relative accuracy $\epsilon_{\text{machine}}$.  Formally,
$$ x \circledast y = (x * y) (1 + \epsilon) $$
for some $|\epsilon| \le \epsilon_{\text{machine}}$.
Can you write Python code for it?

eps = numpy.float32(1)
while numpy.float32(1) + eps > 1:
    eps /= numpy.float64(2)
eps_machine = 2*eps  # We call this "machine epsilon"
print(eps_machine)

format((.2 - 1/3) + 2/15, '.20f')

format(.1, '.20f')


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Theorem: all full-rank $m\times n$ matrices ($m \ge n$) have a unique $Q R$ factorization with $R_{j,j} > 0$.
Orthogonal polynomials
We used x = numpy.linspace(-1,1) which uses $m=50$ points by default.  The number 50 is arbitrary and as we use more points, our columns become better approximations of continuous functions and the vector inner product becomes an integral (up to scaling):
$$ \frac 2 m \sum_{i=1}^m p_i q_i \approx \int_{-1}^1 p(x) q(x) . $$
When we orthogonalize the monomials using this inner product, we get the Legendre Polynomials (up to scaling).  These polynomials have important applications in physics and engineering, as well as playing an important role in approximation (which we will go into in more detail).
Solving equations using QR
To solve
$$ A x = b $$
we can compute $A = QR$ and then
$$ x = R^{-1} Q^T b . $$
This also works for non-square systems!
Can you write Python code for it?

x1 = numpy.array([-0.9, 0.1, 0.5, 0.8])  # points where we know values
y = numpy.array([1, 2.4, -0.2, 1.3])     # values at those points
pyplot.figure()
pyplot.plot(x1, y, '*')
B = numpy.vander(x1, 2)                     # Vandermonde matrix at the known points
Q, R = gram_schmidt_naive(B)
p = numpy.linalg.solve(R, Q.T.dot(y))             # Compute the polynomial coefficients
print(p)
pyplot.plot(x, numpy.vander(x,2).dot(p))                 # Plot the polynomial evaluated at all points
print('B =', B, '\np =', p)

m = 20
V = numpy.vander(numpy.linspace(-1,1,m), increasing=False)
Q, R = gram_schmidt_naive(V)

def qr_test(qr, V):
    Q, R = qr(V)
    m = len(Q.T)
    print(qr.__name__, numpy.linalg.norm(Q.dot(R) - V), numpy.linalg.norm(Q.T.dot(Q) - numpy.eye(m)))

qr_test(gram_schmidt_naive, V)
qr_test(numpy.linalg.qr, V)

def gram_schmidt_classical(X):
    Q = numpy.zeros_like(X)
    R = numpy.zeros((len(X.T),len(X.T)))
    for i in range(len(Q.T)):
        v = X[:,i].copy()
        R[:i,i] = Q[:,:i].T.dot(v)
        v -= Q[:,:i].dot(R[:i,i])
        R[i,i] = numpy.linalg.norm(v)
        Q[:,i] = v / R[i,i]
    return Q, R

qr_test(gram_schmidt_classical, V)


I am working on the file "LinearAlgebra.ipynb".
The first task is:
This type of matrix is very common; we can also create it with numpy.vander.

We can evaluate polynomials using matrix-vector multiplication.
For example,
$$ 5x^3 - 3x = \Bigg[ x^3 \Bigg|\, x^2 \Bigg|\, x \,\Bigg|\, 1 \Bigg] \begin{bmatrix}5 \ 0 \ -3 \ 0 \end{bmatrix} . $$
Can you write Python code for it?

pyplot.figure()
p = numpy.array([5,0,-3,0])
pyplot.plot(x, A.dot(p))


I am working on the file "LinearAlgebra.ipynb".
The first task is:
Vandermonde matrices
A Vandermonde matrix is one whose columns are functions evaluated at discrete points.
Can you write Python code for it?

x = numpy.linspace(-1,1)
A = numpy.array([x**3, x**2, x, 1+0*x]).T
print('shape =', A.shape)  # This is a tall matrix with 4 columns
pyplot.style.use('ggplot')
pyplot.figure()
pyplot.plot(x, A)
pyplot.ylim((-1.1,1.1))
pyplot.show()


I am working on the file "LinearAlgebra.ipynb".
The first task is:
We now have a usable implementation of Householder QR.  There are some further concerns for factoring rank-deficient matrices.  We will visit the concept of pivoting later, in the context of LU and Cholesky factorization.
Condition number of a matrix
We may have informally referred to a matrix as "ill-conditioned" when the columns are nearly linearly dependent, but let's make this concept for precise.  Recall the definition of (relative) condition number from the Rootfinding notes,
$$ \kappa = \max_{\delta x} \frac{|\delta f|/|f|}{|\delta x|/|x|} . $$
We understood this definition for scalar problems, but it also makes sense when the inputs and/or outputs are vectors (or matrices, etc.) and absolute value is replaced by vector (or matrix) norms.  Let's consider the case of matrix-vector multiplication, for which $f(x) = A x$.
$$ \kappa(A) = \max_{\delta x} \frac{\lVert A (x+\delta x) - A x \rVert/\lVert A x \rVert}{\lVert \delta x\rVert/\lVert x \rVert}
= \max_{\delta x} \frac{\lVert A \delta x \rVert}{\lVert \delta x \rVert} \, \frac{\lVert x \rVert}{\lVert A x \rVert} = \lVert A \rVert \frac{\lVert x \rVert}{\lVert A x \rVert} . $$
There are two problems here:

I wrote $\kappa(A)$ but my formula depends on $x$.
What is that $\lVert A \rVert$ beastie?

Stack push: Matrix norms
Vector norms are built into the linear space (and defined in term of the inner product).  Matrix norms are induced by vector norms, according to
$$ \lVert A \rVert = \max_{x \ne 0} \frac{\lVert A x \rVert}{\lVert x \rVert} . $$

This equation makes sense for non-square matrices -- the vector norms of the input and output spaces may differ.
Due to linearity, all that matters is direction of $x$, so it could equivalently be written

$$ \lVert A \rVert = \max_{\lVert x \rVert = 1} \lVert A x \rVert . $$
Stack pop
Now we understand the formula for condition number, but it depends on $x$.  Consider the matrix
$$ A = \begin{bmatrix} 1 & 0 \ 0 & 0 \end{bmatrix} . $$

What is the norm of this matrix?
What is the condition number when $x =...
Can you write Python code for it?

def R_solve(R, b):
    """Solve Rx = b using back substitution."""
    x = b.copy()
    m = len(b)
    for i in reversed(range(m)):
        x[i] -= R[i,i+1:].dot(x[i+1:])
        x[i] /= R[i,i]
    return x

x = numpy.linspace(-1,1,15)
A = numpy.vander(x, 4)
print(A.shape)
Q, R = numpy.linalg.qr(A)
b = Q.T.dot(A.dot(numpy.array([1,2,3,4])))
numpy.linalg.norm(R_solve(R, b) - numpy.linalg.solve(R, b))
R_solve(R, b)


def R_solve(R, b):
    """Solve Rx = b using back substitution."""
    x = b.copy()
    m = len(b)
    for i in reversed(range(m)):
        x[i] -= R[i,i+1:].dot(x[i+1:])
        x[i] /= R[i,i]
    return x

Q, R = numpy.linalg.qr(A)
b = Q.T.dot(A.dot(numpy.array([1,2,3,4])))
numpy.linalg.norm(R_solve(R, b) - numpy.linalg.solve(R, b))
R_solve(R, b)


I am working on the file "LinearRegression/Cost Function.ipynb".
The first task is:
Gradient Descent
Initialize vector w with random numbers taken from a standardized normal curve (0 mean and 0 variance).
Update the values of w until the cost, J(w), is small enough to let us know we have reached the optimum minimum.
Update the coefficients one-by-one by subtracting each from a portion alpha (learning rate) of the partial derivative of the cost function:

Basic Implementation of Gradient Descent
Can you write Python code for it?

observations = len(dataset)
X = dataset['RM'].values.reshape((observations, 1)) # X should always be a matrix
X = np.column_stack((X, np.ones(observations))) # Add the bias
y = dataset['target'].values # y is a vector

import random

def random_w(p):
    return np.array([np.random.normal() for j in range(p)])

def hypothesis(X, w):
    return np.dot(X, w)

def loss(X, w, y):
    return hypothesis(X, w) - y

def squared_loss(X, w, y):
    return loss(X, w, y)**2

def gradient(X, w, y):
    gradients = list()
    n = float(len(y))
    for j in range(len(w)):
        gradients.append(np.sum(loss(X, w, y) * X[:, j]) / n)
    return gradients

def update(X, w, y, alpha=0.01):
    return [t - alpha * g for t, g in zip(w, gradient(X, w, y))]

def optimize(X, y, alpha=0.01, eta=10**-12, iterations=1000):
    w = random_w(X.shape[1])
    path = list()
    for k in range(iterations):
        SSL = np.sum(squared_loss(X, w, y))
        new_w = update(X, w, y, alpha=alpha)
        new_SSL = np.sum(squared_loss(X, new_w, y))
        w = new_w
        if k >= 5 and (new_SSL - SSL <= eta and new_SSL - SSL >= -eta):
            path.append(new_SSL)
            return w, path
        if k % (iterations / 20) == 0:
            path.append(new_SSL)
    return w, path

alpha = 0.048
w, path = optimize(X, y, alpha, eta=10**-12, iterations=25000)
print('These are our final coefficients: {0}'.format(w))
print('Obtained walking on this path of squared loss {0}'.format(path))


I am working on the file "LinearRegression/Cost Function.ipynb".
The first task is:
Minimizing the Cost Function
Can you write Python code for it?

import numpy as np

x = np.array([9.5, 8.5, 8.0, 7.0, 6.0])

def squared_cost(v, e):
    return np.sum((v - e)**2)

from scipy.optimize import fmin

xopt = fmin(squared_cost, x0=0, xtol=1e-8, args=(x,))

print('The result of optimization is {0}'.format(xopt[0]))
print('The mean is {0}'.format(np.mean(x)))

def absolute_cost(v, e):
    return np.sum(np.abs(v - e))

xopt = fmin(absolute_cost, x0=0, xtol=1e-8, args=(x,))

print('The result of optimization is {0}'.format(xopt[0]))
print('The median is {0}'.format(np.median(x)))


I am working on the file "LinearRegression/Cost Function.ipynb".
The first task is:
Pseudoinverse and Other Optimization Methods
Can you write Python code for it?

observations = len(dataset)
X = dataset['RM'].values.reshape((observations, 1)) # X should always be a matrix, never a vector
Xb = np.column_stack((X, np.ones(observations))) # We add the bias
y = dataset['target'].values # y can be a vector

def matrix_inverse(X, y, pseudo=False):
    if pseudo:
        return np.dot(np.linalg.pinv(np.dot(X.T, X)), np.dot(X.T, y))
    else:
        return np.dot(np.linalg.inv(np.dot(X.T, X)), np.dot(X.T, y))

def normal_equations(X, y):
    return np.linalg.solve(np.dot(X.T, X), np.dot(X.T, y))

print(matrix_inverse(Xb, y))
print(matrix_inverse(Xb, y, pseudo=True))
print(normal_equations(Xb, y))


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
Calculate the correlation by summing all of the squared differences and divide by the number of observations. This will yield a number between -1 and 1 with positive indicating proportionality (direct/positive--both grow together; indirect/negative--one grows, other shrinks).
Covariance:

Pearson's correlation:
Can you write Python code for it?

def covariance(var1, var2, bias=0):
    observations = float(len(var1))
    return np.sum((var1 - np.mean(var1)) * (var2 - np.mean(var2))) / (observations - min(bias, 1))

def standardize(var):
    return (var - np.mean(var)) / np.std(var)

def correlation(var1, var2, bias=0):
    return covariance(standardize(var1), standardize(var2), bias)

print('Our correlation estimate: {0}'.format(correlation(dataset['RM'], dataset['target'])))

from scipy.stats.stats import pearsonr

print('Correlation from Scipy pearsonr: {0}'.format(pearsonr(dataset['RM'], dataset['target'])[0]))


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
Convert it into a column vector:
Can you write Python code for it?


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
Load datasets for demonstration
Can you write Python code for it?

from sklearn.datasets import fetch_california_housing
from sklearn.datasets import load_boston

boston = load_boston()
california = fetch_california_housing()


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
Multiple Feature Matrices

Create a multiple feature matrix:
Can you write Python code for it?


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
The Mean
Can you write Python code for it?

dataset.head()

mean_expected_value = dataset['target'].mean()
mean_expected_value


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
The Probability Density Function (PDF)
Can you write Python code for it?

x = np.linspace(-4, 4, 100)
for mean, variance in [(0, 0.7), (0, 1), (1, 1.5), (-2, 0.5)]:
    plt.plot(x, mlab.normpdf(x, mean, variance))


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
The mean can also be calculated using a NumPy function:
Can you write Python code for it?


I am working on the file "LinearRegression/Linear Models.ipynb".
The first task is:
Vectors can be considered as a single feature matrix:
Can you write Python code for it?


I am working on the file "LinearRegression/Linear Regression.ipynb".
The first task is:
Coefficient of Determination
Can you write Python code for it?

mean_sum_squared_errors = np.sum((dataset['target'] - dataset['target'].mean())**2)
regr_sum_squared_errors = np.sum((dataset['target'] - fitted_values)**2)
(mean_sum_squared_errors - regr_sum_squared_errors) / mean_sum_squared_errors

from scipy.stats.stats import pearsonr

(pearsonr(dataset['RM'], dataset['target'])[0])**2


I am working on the file "LinearRegression/Linear Regression.ipynb".
The first task is:
Evaluating the Fitted Values
Can you write Python code for it?

residuals = dataset['target'] - fitted_values
normalized_residuals = standardize(residuals)
normalized_residuals.head()

residual_scatter_plot = plt.plot(dataset['RM'], normalized_residuals, 'bp')
mean_residual = plt.plot([int(x_range[0]), round(x_range[1], 0)], [0, 0], '-', color='red', linewidth=2)
upper_bound = plt.plot([int(x_range[0]), round(x_range[1], 0)], [3, 3], '--', color='red', linewidth=1)
lower_bound = plt.plot([int(x_range[0]), round(x_range[1], 0)], [-3, -3], '--', color='red', linewidth=1)
plt.grid()


I am working on the file "LinearRegression/Linear Regression.ipynb".
The first task is:
Method 1
Can you write Python code for it?

linear_regression = sm.OLS(y, X)
fitted_model = linear_regression.fit()


I am working on the file "LinearRegression/Linear Regression.ipynb".
The first task is:
Method 2
Can you write Python code for it?

linear_regression = smf.ols(formula='target ~ RM', data=dataset)
fitted_model = linear_regression.fit()

fitted_model.summary()

fitted_model.params

betas = np.array(fitted_model.params)
fitted_values = fitted_model.predict(X)
fitted_values.head()


I am working on the file "LinearRegression/Linear Regression.ipynb".
The first task is:
Predicting with a Regression Model
Can you write Python code for it?

# For this cell to work, ensure that fitted_model is set from sm.OLS instead of smf.ols
RM = 5
Xp = np.array([1, RM])
print('Out model predicts if RM = {0} the answer value is {1}'.format(RM, fitted_model.predict(Xp)))

x_range = [dataset['RM'].min(), dataset['RM'].max()]
y_range = [dataset['target'].min(), dataset['target'].max()]
scatter_plot = dataset.plot(kind='scatter', x='RM', y='target', xlim=x_range, ylim=y_range)
mean_y = scatter_plot.plot(x_range, [dataset['target'].mean(), dataset['target'].mean()], '--', color='red', linewidth=1)
mean_x = scatter_plot.plot([dataset['RM'].mean(), dataset['RM'].mean()], y_range, '--', color='red', linewidth=1)
regression_line = scatter_plot.plot(dataset['RM'], fitted_values, '-', color='orange', linewidth=1)

predictions_by_dot_product = np.dot(X, betas)
print('Using the prediction method: {0}'.format(fitted_values[:10]))
print('Using betas and a dot product: {0}'.format(predictions_by_dot_product))


I am working on the file "LinearRegression/Linear Regression.ipynb".
The first task is:
Regressing with Scikit-learn
Can you write Python code for it?

from sklearn import linear_model

linear_regression = linear_model.LinearRegression(normalize=False, fit_intercept=True)

observations = len(dataset)
dataset['RM'].values.reshape((observations, 1)) # X should always be a matrix, never a vector
y = dataset['target'].values # y can be a vector

linear_regression.fit(X, y)

print(linear_regression.coef_)
print(linear_regression.intercept_)

print(linear_regression.predict(X)[:10])


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Check out the Data
Can you write Python code for it?

housing = pd.read_csv('USA_Housing.csv')

housing.head()
#displays the head of the data i.e. first few columns of the data

housing.info()
#displays information about the data

housing.describe()
#gives brief description of the data

housing.columns


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Create graph
Can you write Python code for it?

with tf.name_scope("FNT"):
    with tf.name_scope("Input"):
        X=tf.placeholder(dtype=tf.float32,name="X")
        Y=tf.placeholder(dtype=tf.float32,name="Y")
    w=tf.Variable(0.0,name="w")
    b=tf.Variable(0.0,name="b")
    Y_pred=w*X+b
    with tf.name_scope("Loss"):
        #loss=tf.square(Y-Y_pred)#This works only when using stochstic gradient descent
        loss=tf.reduce_mean(tf.square(Y-Y_pred),name="loss")
    with tf.name_scope("Train"):
        optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.00001).minimize(loss)#How it minimizes a non-scalar variable?
    with tf.name_scope(name="Summary") as scope:
        tf.summary.scalar("loss",loss)
        summary_op=tf.summary.merge(tf.get_collection(tf.GraphKeys.SUMMARIES,scope))


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Creating and Training the Model
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

lm = LinearRegression()

lm.fit(X_train,y_train)


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Data Visualization
Creating simple plots to check out the data
Can you write Python code for it?

sns.pairplot(housing)

sns.distplot(housing['Price'])
#distribution plot of 'Price'

sns.heatmap(housing.corr())
#heatmap-the darker the more related two features are.


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Import MNIST dataset
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
MNIST=input_data.read_data_sets(r"..\DataSet\mnist",one_hot=True)


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Interpreting the coefficients:

Holding all other features fixed, a 1 unit increase in Avg. Area Income is associated with an increase of \$21.52 .
Holding all other features fixed, a 1 unit increase in Avg. Area House Age is associated with an increase of \$164883.28 .
Holding all other features fixed, a 1 unit increase in Avg. Area Number of Rooms is associated with an increase of \$122368.67 .
Holding all other features fixed, a 1 unit increase in Avg. Area Number of Bedrooms is associated with an increase of \$2233.80 .
Holding all other features fixed, a 1 unit increase in Area Population is associated with an increase of \$15.15 .

This obviuosly doesnt make sense as it was a fake dataset but we get an idea of how Linear Regression works!
Predictions from our Model
Let's grab predictions off our test set and see how well it did!
Can you write Python code for it?

predictions = lm.predict(X_test)

plt.scatter(y_test,predictions)
#scatter plot resembles a straight line which means that we had a good prediction


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Model Evaluation
Evaluate the model by checking out it's coefficients and how we can interpret them.
Can you write Python code for it?

# print the intercept
print(lm.intercept_)

coeff_df = pd.DataFrame(lm.coef_,X.columns,columns=['Coefficient'])
coeff_df


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Plot and evaluate fitting
Can you write Python code for it?

plt.figure(figsize=(8,5))
plt.scatter(x=data.T[0],y=data.T[1],label="data")
x_plot=np.arange(0,45,0.1)
y_plot=w_val*x_plot+b_val
plt.plot(x_plot,y_plot,label="predict",color="red")
plt.xlabel("fires per 1000 housing units")
plt.ylabel("thefts per 1000 population")
plt.legend()
plt.show()


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Plot and evaluate fitting.
Can you write Python code for it?

plt.figure(figsize=(8,5))
plt.scatter(x=data.T[0],y=data.T[1],label="data")
x_plot=np.arange(0,45,0.1)
y_plot=param[0]*(x_plot**3)+param[1]*x_plot**2+param[2]*x_plot+param[3]
plt.plot(x_plot,y_plot,label="predict",color="red")
plt.xlabel("fires per 1000 housing units")
plt.ylabel("thefts per 1000 population")
plt.legend()
plt.show()


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Regression Evaluation Metrics
Here are three common evaluation metrics for regression problems:
Mean Absolute Error (MAE) is the mean of the absolute value of the errors:
$$\frac 1n\sum_{i=1}^n|y_i-\hat{y}_i|$$
Mean Squared Error (MSE) is the mean of the squared errors:
$$\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2$$
Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors:
$$\sqrt{\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2}$$
Comparing these metrics:

MAE is the easiest to understand, because it's the average error.
MSE is more popular than MAE, because MSE "punishes" larger errors, which tends to be useful in the real world.
RMSE is even more popular than MSE, because RMSE is interpretable in the "y" units.

All of these are loss functions, because we want to minimize them.
Can you write Python code for it?

from sklearn import metrics

print('MAE:', metrics.mean_absolute_error(y_test, predictions))
print('MSE:', metrics.mean_squared_error(y_test, predictions))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, predictions)))


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Residual Histogram
Can you write Python code for it?


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Testing
Can you write Python code for it?

with tf.Session() as sess:
    n_batches=int(MNIST.test.num_examples/batch_size)
    total_correct_preds = 0
    #Load the model(load the trianed variables)
    tf.train.Saver().restore(sess,"./model_checkpoints/MNIST_LR"+"-"+str(n_epochs))
    for i in range (n_batches):
        X_batch,Y_batch=MNIST.test.next_batch(batch_size)
        preds=sess.run(outln,feed_dict={X:X_batch,Y:Y_batch})
        #preds=tf.nn.softmax(preds)
        #No need for softmax to make predictions, actually. Spare computation.
        correct_preds=tf.equal(tf.argmax(preds,1),tf.argmax(Y_batch,1))
        batch_correct_preds=tf.reduce_sum(tf.cast(correct_preds,tf.float32))
        total_correct_preds+=sess.run(batch_correct_preds)
#Print accuracy
print("Accuracy {0}".format(total_correct_preds/MNIST.test.num_examples))


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Train $w$, $b$ with batch gradient descent in another session
Can you write Python code for it?

#Need to use new learning rate for batch gradient descent.
optimizer=tf.train.GradientDescentOptimizer(learning_rate=0.00001).minimize(loss)#Use new optimizer (with smaller learning rate).
with tf.Session() as sess:
    writer=tf.summary.FileWriter(r"./graphs",sess.graph)
    sess.run(tf.global_variables_initializer())
    for i in range(1000):#Number of iterations
        _,summary=sess.run([optimizer,summary_op],feed_dict={X:data.T[0],Y:data.T[1]})
        writer.add_summary(summary=summary,global_step=i)
    w_val,b_val=sess.run([w,b])
    writer.close()
print("w:",w_val,"b:",b_val)


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Train $w$, $b$ with stochastic gradient descent in a session
Can you write Python code for it?

with tf.Session() as sess:
    writer=tf.summary.FileWriter(r"./graphs",sess.graph)
    sess.run(tf.global_variables_initializer())
    for i in range(500):#Number of iterations
        for x,y in data:
            _,summary=sess.run([optimizer,summary_op],feed_dict={X:x,Y:y})
            writer.add_summary(summary,global_step=i)
    w_val,b_val=sess.run([w,b])
    writer.close()
print("w:",w_val,"b:",b_val)


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Train $w_{3}$, $w_{2}$, $w_{1}$, $w_{0}$ with stochastic gradient descent in a session
Can you write Python code for it?

with tf.Session() as sess:
    writer=tf.summary.FileWriter(r"./graphs",sess.graph)
    sess.run(tf.global_variables_initializer())
    for i in range(750):#Number of iterations
        for x,y in data:
            sess.run(optimizer,feed_dict={X:x,Y:y})
    param=sess.run([w3,w2,w1,w0])
    writer.close()
for i in range(4):
    print("w{}: {}".format(3-i,param[i]))


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Train Test Split
Split the data into a training set and a testing set.
We will train out model on the training set and then use the test set to evaluate the model.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=101)


I am working on the file "LinearRegression/LinearRegression.ipynb".
The first task is:
Training a Linear Regression Model'
We will first need to split our data into an X array that contains the features to train on, and a y array with the target variable,in this cases the 'Price' column.
We will also toss out 'Address' olumn because it only has text that LinearRegression model cant use.
X and y arrays
Can you write Python code for it?

X = housing[['Avg. Area Income', 'Avg. Area House Age', 'Avg. Area Number of Rooms',
               'Avg. Area Number of Bedrooms', 'Area Population']]
y = housing['Price']


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Great! Now we can get the best fit values!
Can you write Python code for it?

# Now get out m and b values for our best fit line
m, b = np.linalg.lstsq(X, Y)[0]


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Great! Now we can make out a slight trend that price increases along with the number of rooms in that house, which intuitively makes sense! Now let's use scikit learn to see if we can fit the data linearly.
Let's try to do the following:
1.) Use pandas to transform the boston dataset into a DataFrame:

2.) Then use seaborn to perform an lmplot on that DataFrame to reproduce the scatter plot with a linear fit line.
Can you write Python code for it?

# reset data as pandas DataFrame
boston_df = DataFrame(boston.data)

# label columns
boston_df.columns = boston.feature_names

#show
boston_df.head()


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Great! So we have basically made an equation for a line, but instead of just oneo coefficient m and an intercept b, we now have 13 coefficients. To get an idea of what this looks like check out the documentation for this equation:
$$ y(w,x) = w_0 + w_1 x_1 + ... + w_p x_p $$
Where $$w = (w_1, ...w_p)$$ as the coefficients and $$ w_0 $$ as the intercept
What we'll do next is set up a DataFrame showing all the Features and their estimated coefficients obtained form the linear regression.
Can you write Python code for it?

# Set a DataFrame from the Features
coeff_df = DataFrame(boston_df.columns)
coeff_df.columns = ['Features']

# Set a new column lining up the coefficients from the linear regression
coeff_df["Coefficient Estimate"] = pd.Series(lreg.coef_)

# Show
coeff_df


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Imports for plotting
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')
%matplotlib inline


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Let's go ahead check the intercept and number of coefficients.
Can you write Python code for it?

print(' The estimated intercept coefficient is %.2f ' %lreg.intercept_)

print(' The number of coefficients used was %d ' % len(lreg.coef_))


I am working on the file "LinearRegression/Supervised+Learning+-++Linear+Regression.ipynb".
The first task is:
Next we'll download the data set
Can you write Python code for it?

# Load the housing dataset
boston = load_boston()