I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro-checkpoint.ipynb".
The first task is:
input variables.
3. Scatter plots
3.1. 2D scatter plots
When the instances of the dataset are multidimensional, they cannot be visualized directly, but we can get a first rough idea about the regression task if we plot the target variable versus one of the input variables. These representations are known as <i>scatter plots</i>
Python methods plot and scatter from the matplotlib package can be used for these graphical representations.
Can you write Python code for it?

# Select a dataset
nrows = 4
ncols = 1 + (X.shape[1]-1)/nrows

# Some adjustment for the subplot.
pylab.subplots_adjust(hspace=0.2)

# Plot all variables
for idx in range(X.shape[1]):
    ax = plt.subplot(nrows,ncols,idx+1)
    ax.scatter(X[:,idx], S)    # <-- This is the key command
    ax.get_xaxis().set_ticks([])
    ax.get_yaxis().set_ticks([])
    plt.ylabel('Target')


I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro-checkpoint.ipynb".
The first task is:
observations of the target variable and
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro_student-checkpoint.ipynb".
The first task is:
1. The regression problem
The goal of regression methods is to predict the value of some target variable $S$ from the observation of one or more input variables $X_1, X_2, \ldots, X_N$ (that we will collect in a single vector $\bf X$).
Regression problems arise in situations where the value of the target variable is not easily accessible, but we can measure other dependent variables, from which we can try to predict $S$.

The only information available to estimate the relation between the inputs and the target is a dataset $\mathcal D$ containing several observations of all variables.
$$\mathcal{D} = {{\bf x}^{(k)}, s^{(k)}}_{k=1}^K$$
The dataset $\mathcal{D}$ must be used to find a function $f$ that, for any observation vector ${\bf x}$, computes an output $\hat{s} = f({\bf x})$ that is a good predition of the true value of the target, $s$.

2. Examples of regression problems.
The scikit-learn package contains several  datasets related to regression problems.

 Boston dataset: the target variable contains housing values in different suburbs of Boston. The goal is to predict these values based on several social, economic and demographic variables taken frome theses suburbs (you can get more details in the  UCI repository ).

 Diabetes dataset.

We can load these datasets as follows:
Can you write Python code for it?

from sklearn import datasets

# Load the dataset. Select it by uncommenting the appropriate line
D_all = datasets.load_boston()
#D_all = datasets.load_diabetes()

# Extract data and data parameters.
X = D_all.data         # Complete data matrix (including input and target variables)
S = D_all.target      # Target variables
n_samples = X.shape[0] # Number of observations
n_vars = X.shape[1]    # Number of variables (including input and target)


I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro_student-checkpoint.ipynb".
The first task is:
3.2. 3D Plots
With the addition of a third coordinate, plot and scatter can be used for 3D plotting.
Exercise 1:
Select the diabetes dataset. Visualize the target versus components 2 and 4. (You can get more info about the scatter command and an example of use in the  matplotlib documentation)
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro_student-checkpoint.ipynb".
The first task is:
The overal prediction performance is computed as the average of the loss computed over a set of samples:
$${\bar R} = \frac{1}{K}\sum_{k=1}^K l\left(s^{(k)}, \hat{s}^{(k)}\right)$$
Exercise 2:
The dataset in file 'datasets/x01.csv', taken from here records the average weight of the brain and body for a number of mammal species.
* Represent a scatter plot of the targe variable versus the one-dimensional input.
* Plot, over the same plot, the prediction function given by $S = 1.2 X$
* Compute the square error rate for the given dataset.
Can you write Python code for it?

# Load dataset in arrays X and S
df = pd.read_csv('datasets/x01.csv', sep=',', header=None)
X = df.values[:,0]
S = df.values[:,1]

# <SOL>

# </SOL>

if sys.version_info.major==2:
    Test.assertTrue(np.isclose(R, 153781.943889), 'Incorrect value for the average square error')


I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro_student-checkpoint.ipynb".
The first task is:
This dataset contains
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/.ipynb_checkpoints/regression_intro_student-checkpoint.ipynb".
The first task is:
input variables.
3. Scatter plots
3.1. 2D scatter plots
When the instances of the dataset are multidimensional, they cannot be visualized directly, but we can get a first rough idea about the regression task if we plot the target variable versus one of the input variables. These representations are known as <i>scatter plots</i>
Python methods plot and scatter from the matplotlib package can be used for these graphical representations.
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/regression_intro_professor.ipynb".
The first task is:
1. The regression problem
The goal of regression methods is to predict the value of some target variable $S$ from the observation of one or more input variables $X_0, X_1, \ldots, X_{m-1}$ (that we will collect in a single vector $\bf X$).
Regression problems arise in situations where the value of the target variable is not easily accessible, but we can measure other dependent variables, from which we can try to predict $S$.

The only information available to estimate the relation between the inputs and the target is a dataset $\mathcal D$ containing several observations of all variables.
$$\mathcal{D} = {{\bf x}{k}, s{k}}_{k=0}^{K-1}$$
The dataset $\mathcal{D}$ must be used to find a function $f$ that, for any observation vector ${\bf x}$, computes an output $\hat{s} = f({\bf x})$ that is a good predition of the true value of the target, $s$.

2. Examples of regression problems.
The scikit-learn package contains several  datasets related to regression problems.

 Boston dataset: the target variable contains housing values in different suburbs of Boston. The goal is to predict these values based on several social, economic and demographic variables taken frome theses suburbs (you can get more details in the  UCI repository ).

 Diabetes dataset.

We can load these datasets as follows:
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/regression_intro_professor.ipynb".
The first task is:
The overal prediction performance is computed as the average of the loss computed over a set of samples:
$${\bar R} = \frac{1}{K}\sum_{k=0}^{K-1} l\left(s_k, \hat{s}_k\right)$$
Exercise 2:
The dataset in file 'datasets/x01.csv', taken from here records the average weight of the brain and body for a number of mammal species.
* Represent a scatter plot of the targe variable versus the one-dimensional input.
* Plot, over the same plot, the prediction function given by $S = 1.2 X$
* Compute the square error rate for the given dataset.
Can you write Python code for it?

# Load dataset in arrays X and S
df = pd.read_csv('datasets/x01.csv', sep=',', header=None)
X = df.values[:,0]
S = df.values[:,1]

# <SOL>
fig = plt.figure()
plt.scatter(X, S)
plt.xlabel('$x_2$')
plt.ylabel('$x_4$')
xgrid = np.array([-200.0, 6900.0])
plt.plot(xgrid, 1.2*xgrid)

R = np.mean((S-1.2*X)**2)
print('The average square error is {0}'.format(R))
# </SOL>

if sys.version_info.major==2:
    Test.assertTrue(np.isclose(R, 153781.943889), 'Incorrect value for the average square error')
else:
    np.testing.assert_almost_equal(R, 153781.943889, decimal=4)
    print("Test passed")


I am working on the file "R1.Intro_Regression/regression_intro_professor.ipynb".
The first task is:
This dataset contains
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/regression_intro_professor.ipynb".
The first task is:
observations of the target variable and
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/regression_intro_student.ipynb".
The first task is:
1. The regression problem
The goal of regression methods is to predict the value of some target variable $S$ from the observation of one or more input variables $X_0, X_1, \ldots, X_{m-1}$ (that we will collect in a single vector $\bf X$).
Regression problems arise in situations where the value of the target variable is not easily accessible, but we can measure other dependent variables, from which we can try to predict $S$.

The only information available to estimate the relation between the inputs and the target is a dataset $\mathcal D$ containing several observations of all variables.
$$\mathcal{D} = {{\bf x}{k}, s{k}}_{k=0}^{K-1}$$
The dataset $\mathcal{D}$ must be used to find a function $f$ that, for any observation vector ${\bf x}$, computes an output $\hat{s} = f({\bf x})$ that is a good predition of the true value of the target, $s$.

2. Examples of regression problems.
The scikit-learn package contains several  datasets related to regression problems.

 Boston dataset: the target variable contains housing values in different suburbs of Boston. The goal is to predict these values based on several social, economic and demographic variables taken frome theses suburbs (you can get more details in the  UCI repository ).

 Diabetes dataset.

We can load these datasets as follows:
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/regression_intro_student.ipynb".
The first task is:
3.2. 3D Plots
With the addition of a third coordinate, plot and scatter can be used for 3D plotting.
Exercise 1:
Select the diabetes dataset. Visualize the target versus components 2 and 4. (You can get more info about the scatter command and an example of use in the  matplotlib documentation)
Can you write Python code for it?


I am working on the file "R1.Intro_Regression/regression_intro_student.ipynb".
The first task is:
4. Evaluating a regression task
In order to evaluate the performance of a given predictor, we need to quantify the quality of predictions. This is usually done by means of a loss function $l(s,\hat{s})$. Two common losses are

Square error: $l(s, \hat{s}) = (s - \hat{s})^2$
Absolute error: $l(s, \hat{s}) = |s - \hat{s}|$

Note that both the square and absolute errors are functions of the estimation error $e = s-{\hat s}$. However, this is not necessarily the case. As an example, imagine a situation in which we would like to introduce a penalty which increases with the magnitude of the estimated variable. For such case, the following cost would better fit our needs: $l(s,{\hat s}) = s^2 \left(s-{\hat s}\right)^2$.
Can you write Python code for it?

# In this section we will plot together the square and absolute errors
grid = np.linspace(-3,3,num=100)
plt.plot(grid, grid**2, 'b-', label='Square error')
plt.plot(grid, np.absolute(grid), 'r--', label='Absolute error')
plt.xlabel('Error')
plt.ylabel('Cost')
plt.legend(loc='best')
plt.show()


I am working on the file "R1.Intro_Regression/regression_intro_student.ipynb".
The first task is:
This dataset contains
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
1.1. Scatter plots
We can get a first rough idea about the regression task representing the scatter plot of each of the one-dimensional variables against the target data.
Can you write Python code for it?

pylab.subplots_adjust(hspace=0.2)

for idx in range(X_tr.shape[1]):
    ax1 = plt.subplot(3,3,idx+1)
    ax1.plot(X_tr[:,idx],S_tr,'.')
    ax1.get_xaxis().set_ticks([])
    ax1.get_yaxis().set_ticks([])

plt.show()


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
2. Baseline estimation. Using the average of the training set labels
A first very simple method to build the regression model is to use the average of all the target values in the training set as the output of the model, discarding the value of the observation input vector.
This approach can be considered as a baseline, given that any other method making an effective use of the observation variables, statistically related to $s$, should improve the performance of this method.
The prediction is thus given by
Can you write Python code for it?

# Mean of all target values in the training set
s_hat = np.mean(S_tr)
print(s_hat)


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
4. Multidimensional regression with the $k$-nn method
In the previous subsection, we have studied the performance of the $k$-nn method when using only one variable. Doing so was convenient, because it allowed us to plot the regression curves in a 2-D plot, and to get some insight about the consequences of modifying the number of neighbors.
For completeness, we evaluate now the performance of the $k$-nn method in this dataset when using all variables together. In fact, when designing a regression model, we should proceed in this manner, using all available information to make as accurate an estimation as possible. In this way, we can also account for correlations that might be present among the different observation variables, and that may carry very relevant information for the regression task.
For instance, in the STOCK dataset, it may be that the combination of the stock values of two airplane companies is more informative about the price of the target company, while the value for a single company is not enough.
<small> Also, in the CONCRETE dataset, it may be that for the particular problem at hand the combination of a large proportion of water and a small proportion of coarse grain is a clear indication of certain compressive strength of the material, while the proportion of water or coarse grain alone are not enough to get to that result.</small>
Can you write Python code for it?

k_max = 20

MSE_tr = [square_error(S_tr, knn_regression(X_tr, S_tr, X_tr, k)) for k in range(1, k_max+1)]
MSE_tst = [square_error(S_tst, knn_regression(X_tr, S_tr, X_tst, k)) for k in range(1, k_max+1)]

plt.plot(np.arange(k_max)+1, MSE_tr,'bo',label='Training square error')
plt.plot(np.arange(k_max)+1, MSE_tst,'ro',label='Test square error')
plt.xlabel('k')
plt.ylabel('Square error')

plt.legend(loc='best')
plt.show()


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
As we can see, the error initially decreases achiving a minimum (in the test set) for some finite value of $k$ ($k\approx 10$ for the STOCK dataset). Increasing the value of $k$ beyond that value results in poorer performance.
Exercise 2
Analize the training MSE for $k=1$. Why is it smaller than for any other $k$? Under which conditions will it be exactly zero?
Exercise 3
Modify the code above to visualize the square error from $k=1$ up to $k$ equal to the number of training instances. Can you relate the square error of the $k$-NN method with that of the baseline method for certain value of $k$?
3.1. Influence of the input variable
Having a look at the scatter plots, we can observe that some observation variables seem to have a more clear relationship with the target value. Thus, we can expect that not all variables are equally useful for the regression task. In the following plot, we carry out a study of the performance that can be achieved with each variable.
Note that, in practice, the test labels are not available for the selection of hyperparameter
$k$, so we should be careful about the conclusions of this experiment. A more realistic approach will be studied later when we introduce the concept of model validation.
Can you write Python code for it?

k_max = 20

var_performance = []
k_values = []

for var in range(X_tr.shape[1]):

    MSE_tr = [square_error(S_tr, knn_regression(X_tr[:,var], S_tr, X_tr[:, var], k))
              for k in range(1, k_max+1)]
    MSE_tst = [square_error(S_tst, knn_regression(X_tr[:,var], S_tr, X_tst[:, var], k))
               for k in range(1, k_max+1)]
    MSE_tr = np.asarray(MSE_tr)
    MSE_tst = np.asarray(MSE_tst)

    # We select the variable associated to the value of k for which the training error is minimum
    pos = np.argmin(MSE_tr)
    k_values.append(pos + 1)
    var_performance.append(MSE_tst[pos])

plt.stem(range(X_tr.shape[1]), var_performance)
plt.title('Results of unidimensional regression ($k$NN)')
plt.xlabel('Variable')
plt.ylabel('Test MSE')

plt.figure(2)
plt.stem(range(X_tr.shape[1]), k_values)
plt.xlabel('Variable')
plt.ylabel('$k$')
plt.title('Selection of the hyperparameter')
plt.show()


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
In this case, we can check that the average test square error is much lower than the error that was achieved when using only one variable, and also far better than the baseline method. It is also interesting to note that in this particular case the best performance is achieved for a small value of $k$, with the error increasing for larger values of the hyperparameter.
Nevertheless, as we discussed previously, these results should be taken carefully. How would we select the value of $k$, if test labels are (obvioulsy) not available for model validation?
5. Hyperparameter selection via cross-validation
An inconvenient of the application of the $k$-nn method is that the selection of $k$ influences the final error of the algorithm. In the previous experiments, we kept the value of $k$ that minimized the square error on the training set. However, we also noticed that the location of the minimum is not necessarily the same from the perspective of the test data. Ideally, we would like that the designed regression model works as well as possible on future unlabeled patterns that are not available during the training phase. This property is known as generalization. Fitting the training data is only pursued in the hope that we are also indirectly obtaining a model that generalizes well. In order to achieve this goal, there are some strategies that try to guarantee a correct generalization of the model. One of such approaches is known as cross-validation
Since using the test labels during the training phase is not allowed (they should be kept aside to simultate the future application of the regression model on unseen patterns), we need to figure out some way to improve our estimation of the hyperparameter that requires only training data. Cross-validation allows us to do so by following the following steps:

Split the training data into several (generally non-overlapping) subsets. If we use $M$ subsets, the method is referred to as $M$-fold cross-validation. If we consider each pattern a different subset,...
Can you write Python code for it?

### This fragment of code runs k-nn with M-fold cross validation

# Parameters:
M = 5       # Number of folds for M-cv
k_max = 40  # Maximum value of the k-nn hyperparameter to explore

# First we compute the train error curve, that will be useful for comparative visualization.
MSE_tr = [square_error(S_tr, knn_regression(X_tr, S_tr, X_tr, k)) for k in range(1, k_max+1)]

## M-CV
# Obtain the indices for the different folds
n_tr = X_tr.shape[0]
permutation = np.random.permutation(n_tr)

# Split the indices in M subsets with (almost) the same size.
set_indices = {i: [] for i in range(M)}
i = 0
for pos in range(n_tr):
    set_indices[i].append(permutation[pos])
    i = (i+1) % M

# Obtain the validation errors
MSE_val = np.zeros((1,k_max))
for i in range(M):
    val_indices = set_indices[i]

    # Take out the val_indices from the set of indices.
    tr_indices = list(set(permutation) - set(val_indices))

    MSE_val_iter = [square_error(S_tr[val_indices],
                                 knn_regression(X_tr[tr_indices, :], S_tr[tr_indices],
                                                X_tr[val_indices, :], k))
                    for k in range(1, k_max+1)]

    MSE_val = MSE_val + np.asarray(MSE_val_iter).T

MSE_val = MSE_val/M

# Select the best k based on the validation error
k_best = np.argmin(MSE_val) + 1

# Compute the final test MSE for the selecte k
MSE_tst = square_error(S_tst, knn_regression(X_tr, S_tr, X_tst, k_best))

plt.plot(np.arange(k_max)+1, MSE_tr, 'bo', label='Training square error')
plt.plot(np.arange(k_max)+1, MSE_val.T, 'go', label='Validation square error')
plt.plot([k_best, k_best], [0, MSE_tst],'r-')
plt.plot(k_best, MSE_tst,'ro',label='Test error')
plt.legend(loc='best')
plt.show()


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
Note that in the previous piece of code, function 'square_error' can be used when the second argument is a number instead of a vector with the same length as the first argument. The value will be subtracted from each of the components of the vector provided as the first argument.
Can you write Python code for it?

if sys.version_info.major == 2:
    Test.assertTrue(np.isclose(MSE_tr, square_error(S_tr, s_hat)),'Incorrect value for MSE_tr')
    Test.assertTrue(np.isclose(MSE_tst, square_error(S_tst, s_hat)),'Incorrect value for MSE_tst')


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn-checkpoint.ipynb".
The first task is:
for any input ${\bf x}$.
Exercise 1
Compute the mean square error over training and test sets, for the baseline estimation  method.
Can you write Python code for it?

# We start by defining a function that calculates the average square error
def square_error(s, s_est):
    # Squeeze is used to make sure that s and s_est have the appropriate dimensions.
    y = np.mean(np.power((s - s_est), 2))
    # y = np.mean(np.power((np.squeeze(s) - np.squeeze(s_est)), 2))
    return y

# Mean square error of the baseline prediction over the training data
# MSE_tr = <FILL IN>
MSE_tr = square_error(S_tr, s_hat)

# Mean square error of the baseline prediction over the test data
# MSE_tst = <FILL IN>
MSE_tst = square_error(S_tst, s_hat)

print('Average square error in the training set (baseline method): {0}'.format(MSE_tr))
print('Average square error in the test set (baseline method): {0}'.format(MSE_tst))


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn_student-checkpoint.ipynb".
The first task is:
2. Baseline estimation. Using the average of the training set labels
A first very simple method to build the regression model is to use the average of all the target values in the training set as the output of the model, discarding the value of the observation input vector.
This approach can be considered as a baseline, given that any other method making an effective use of the observation variables, statistically related to $s$, should improve the performance of this method.
The prediction is thus given by
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn_student-checkpoint.ipynb".
The first task is:
4. Multidimensional regression with the $k$-nn method
In the previous subsection, we have studied the performance of the $k$-nn method when using only one variable. Doing so was convenient, because it allowed us to plot the regression curves in a 2-D plot, and to get some insight about the consequences of modifying the number of neighbors.
For completeness, we evaluate now the performance of the $k$-nn method in this dataset when using all variables together. In fact, when designing a regression model, we should proceed in this manner, using all available information to make as accurate an estimation as possible. In this way, we can also account for correlations that might be present among the different observation variables, and that may carry very relevant information for the regression task.
For instance, in the STOCK dataset, it may be that the combination of the stock values of two airplane companies is more informative about the price of the target company, while the value for a single company is not enough.
<small> Also, in the CONCRETE dataset, it may be that for the particular problem at hand the combination of a large proportion of water and a small proportion of coarse grain is a clear indication of certain compressive strength of the material, while the proportion of water or coarse grain alone are not enough to get to that result.</small>
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn_student-checkpoint.ipynb".
The first task is:
Exercise 4
Modify the previous code to use only one of the variables in the input dataset
  - Following a cross-validation approach, select the best value of $k$ for the $k$-nn based in variable 0 only.
  - Compute the test error for the selected valua of $k$.
6. Scikit-learn implementation
In practice, most well-known machine learning methods are implemented and available for python. Probably, the most complete module for machine learning tools is Scikit-learn. The following piece of code uses the method
KNeighborsRegressor

available in Scikit-learn. The example has been taken from here. As you can check, this routine allows us to build the estimation for a particular point using a weighted average of the targets of the neighbors:
To obtain the estimation at a point ${\bf x}$:

Find $k$ closest points to ${\bf x}$ in the training set
Average the corresponding targets, weighting each value according to the distance of each point to ${\bf x}$, so that closer points have a larger influence in the estimation.
Can you write Python code for it?

# Author: Alexandre Gramfort <alexandre.gramfort@inria.fr>
#         Fabian Pedregosa <fabian.pedregosa@inria.fr>
#
# License: BSD 3 clause (C) INRIA

###############################################################################
# Generate sample data
import numpy as np
import matplotlib.pyplot as plt
from sklearn import neighbors

np.random.seed(0)
X = np.sort(5 * np.random.rand(40, 1), axis=0)
T = np.linspace(0, 5, 500)[:, np.newaxis]
y = np.sin(X).ravel()

# Add noise to targets
y[::5] += 1 * (0.5 - np.random.rand(8))

###############################################################################
# Fit regression model
n_neighbors = 5

for i, weights in enumerate(['uniform', 'distance']):
    knn = neighbors.KNeighborsRegressor(n_neighbors, weights=weights)
    y_ = knn.fit(X, y).predict(T)

    plt.subplot(2, 1, i + 1)
    plt.scatter(X, y, c='k', label='data')
    plt.plot(T, y_, c='g', label='prediction')
    plt.axis('tight')
    plt.legend()
    plt.title("KNeighborsRegressor (k = %i, weights = '%s')" % (n_neighbors,
                                                                weights))

plt.show()


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn_student-checkpoint.ipynb".
The first task is:
Note that in the previous piece of code, function 'square_error' can be used when the second argument is a number instead of a vector with the same length as the first argument. The value will be subtracted from each of the components of the vector provided as the first argument.
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/.ipynb_checkpoints/regression_knn_student-checkpoint.ipynb".
The first task is:
for any input ${\bf x}$.
Exercise 1
Compute the mean square error over training and test sets, for the baseline estimation  method.
Can you write Python code for it?

# We start by defining a function that calculates the average square error
def square_error(s, s_est):
    # Squeeze is used to make sure that s and s_est have the appropriate dimensions.
    y = np.mean(np.power((s - s_est), 2))
    # y = np.mean(np.power((np.squeeze(s) - np.squeeze(s_est)), 2))
    return y

# Mean square error of the baseline prediction over the training data
# MSE_tr = <FILL IN>

# Mean square error of the baseline prediction over the test data
# MSE_tst = <FILL IN>

print('Average square error in the training set (baseline method): {0}'.format(MSE_tr))
print('Average square error in the test set (baseline method): {0}'.format(MSE_tst))


I am working on the file "R2.kNN_Regression/regression_knn_professor.ipynb".
The first task is:
1. The dataset
We describe next the regression task that we will use in the session. The dataset is an adaptation of the  STOCK dataset, taken originally from the  StatLib Repository. The goal of this problem is to predict the values of the stocks of a given airplane company, given the values of another 9 companies in the same day.
 If you are reading this text from the python notebook with its full functionality, you can explore the results of the regression experiments using two alternative datasets:

The
CONCRETE dataset, taken from the Machine Learning Repository at the University of California Irvine. The goal of the CONCRETE dataset tas is to predict the compressive strength of cement mixtures based on eight observed variables related to the composition of the mixture and the age of the material).

The Advertising dataset, taken from the book  An Introduction to Statistical Learning with applications in R, with permission from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani. The goal of this problem is to predict the sales of a given product, knowing the investment in different advertising sectors. More specifically, the input and output variables can be described as follows:

Input features:

TV: advertising dollars spent on TV for a single product in a given market (in thousands of dollars)
Radio: advertising dollars spent on Radio
Newspaper: advertising dollars spent on Newspaper

Response variable:

Sales: sales of a single product in a given market (in thousands of widgets)

To do so, just replace stock by concrete or advertising in the next cell. Remind that you must run the cells again to see the changes.

Can you write Python code for it?

# SELECT dataset
# Available options are 'stock', 'concrete' or 'advertising'
ds_name = 'stock'

# Let us start by loading the data into the workspace, and visualizing the dimensions of all matrices
if ds_name == 'stock':
    # STOCK DATASET
    data = scipy.io.loadmat('datasets/stock.mat')
    X_tr = data['xTrain']
    S_tr = data['sTrain']
    X_tst = data['xTest']
    S_tst = data['sTest']

elif ds_name == 'concrete':
    # CONCRETE DATASET.
    data = scipy.io.loadmat('datasets/concrete.mat')
    X_tr = data['X_tr']
    S_tr = data['S_tr']
    X_tst = data['X_tst']
    S_tst = data['S_tst']

elif ds_name == 'advertising':
    # ADVERTISING DATASET
    df = pd.read_csv('datasets/Advertising.csv', header=0)
    X_tr = df.values[:150, 1:4]
    S_tr = df.values[:150, [-1]]   # The brackets around -1 is to make sure S_tr is a column vector, as in the other datasets
    X_tst = df.values[150:, 1:4]
    S_tst = df.values[150:, [-1]]

else:
    print('Unknown dataset')

# Print the data dimension and the dataset sizes
print("SELECTED DATASET: " + ds_name)
print("---- The size of the training set is {0}, that is: {1} samples with dimension {2}.".format(
    X_tr.shape, X_tr.shape[0], X_tr.shape[1]))
print("---- The target variable of the training set contains {0} samples with dimension {1}".format(
    S_tr.shape[0], S_tr.shape[1]))
print("---- The size of the test set is {0}, that is: {1} samples with dimension {2}.".format(
    X_tst.shape, X_tst.shape[0], X_tst.shape[1]))
print("---- The target variable of the test set contains {0} samples with dimension {1}".format(
    S_tst.shape[0], S_tst.shape[1]))


I am working on the file "R2.kNN_Regression/regression_knn_professor.ipynb".
The first task is:
Exercise 4
Modify the previous code to use only one of the variables in the input dataset
  - Following a cross-validation approach, select the best value of $k$ for the $k$-nn based in variable 0 only.
  - Compute the test error for the selected valua of $k$.
6. Scikit-learn implementation
In practice, most well-known machine learning methods are implemented and available for python. Probably, the most complete module for machine learning tools is Scikit-learn. The following piece of code uses the method
KNeighborsRegressor

available in Scikit-learn. The example has been taken from here. As you can check, this routine allows us to build the estimation for a particular point using a weighted average of the targets of the neighbors:
To obtain the estimation at a point ${\bf x}$:

Find $k$ closest points to ${\bf x}$ in the training set
Average the corresponding targets, weighting each value according to the distance of each point to ${\bf x}$, so that closer points have a larger influence in the estimation.
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/regression_knn_professor.ipynb".
The first task is:
for any input ${\bf x}$.
Exercise 1
Compute the mean square error over training and test sets, for the baseline estimation  method.
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
1. The dataset
We describe next the regression task that we will use in the session. The dataset is an adaptation of the  STOCK dataset, taken originally from the  StatLib Repository. The goal of this problem is to predict the values of the stocks of a given airplane company, given the values of another 9 companies in the same day.
 If you are reading this text from the python notebook with its full functionality, you can explore the results of the regression experiments using two alternative datasets:

The
CONCRETE dataset, taken from the Machine Learning Repository at the University of California Irvine. The goal of the CONCRETE dataset tas is to predict the compressive strength of cement mixtures based on eight observed variables related to the composition of the mixture and the age of the material).

The Advertising dataset, taken from the book  An Introduction to Statistical Learning with applications in R, with permission from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani. The goal of this problem is to predict the sales of a given product, knowing the investment in different advertising sectors. More specifically, the input and output variables can be described as follows:

Input features:

TV: advertising dollars spent on TV for a single product in a given market (in thousands of dollars)
Radio: advertising dollars spent on Radio
Newspaper: advertising dollars spent on Newspaper

Response variable:

Sales: sales of a single product in a given market (in thousands of widgets)

To do so, just replace stock by concrete or advertising in the next cell. Remind that you must run the cells again to see the changes.

Can you write Python code for it?


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
1.1. Scatter plots
We can get a first rough idea about the regression task representing the scatter plot of each of the one-dimensional variables against the target data.
Can you write Python code for it?

pylab.subplots_adjust(hspace=0.2)
for idx in range(X_tr.shape[1]):
    ax1 = plt.subplot(3,3,idx+1)
    ax1.plot(X_tr[:,idx],S_tr,'.')
    ax1.get_xaxis().set_ticks([])
    ax1.get_yaxis().set_ticks([])
plt.show()


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
2. Baseline estimation. Using the average of the training set labels
A first very simple method to build the regression model is to use the average of all the target values in the training set as the output of the model, discarding the value of the observation input vector.
This approach can be considered as a baseline, given that any other method making an effective use of the observation variables, statistically related to $s$, should improve the performance of this method.
The prediction is thus given by
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
3.1. Evolution of the error with the number of neighbors ($k$)
We see that a small $k$ results in a regression curve that exhibits many and large oscillations.  The curve is capturing any noise that may be present in the training data, and <i>overfits</i> the training set. On the other hand, picking a too large $k$ (e.g., 200) the regression curve becomes too smooth, averaging out the values of the labels in the training set over large intervals of the observation variable.
The next code illustrates this effect by plotting the average training and test square errors as a function of $k$.
Can you write Python code for it?

var = 0
k_max = 60

k_max = np.minimum(k_max, X_tr.shape[0])  # k_max cannot be larger than the number of samples

#Be careful with the use of range, e.g., range(3) = [0,1,2] and range(1,3) = [1,2]
MSEk_tr = [square_error(S_tr, knn_regression(X_tr[:,var], S_tr, X_tr[:,var],k))
           for k in range(1, k_max+1)]
MSEk_tst = [square_error(S_tst,knn_regression(X_tr[:,var], S_tr, X_tst[:,var],k))
            for k in range(1, k_max+1)]

kgrid = np.arange(1, k_max+1)
plt.plot(kgrid, MSEk_tr,'bo', label='Training square error')
plt.plot(kgrid, MSEk_tst,'ro', label='Test square error')
plt.xlabel('$k$')
plt.ylabel('Square Error')
plt.axis('tight')

plt.legend(loc='best')
plt.show()


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
As we can see, the error initially decreases achiving a minimum (in the test set) for some finite value of $k$ ($k\approx 10$ for the STOCK dataset). Increasing the value of $k$ beyond that value results in poorer performance.
Exercise 2
Analize the training MSE for $k=1$. Why is it smaller than for any other $k$? Under which conditions will it be exactly zero?
Exercise 3
Modify the code above to visualize the square error from $k=1$ up to $k$ equal to the number of training instances. Can you relate the square error of the $k$-NN method with that of the baseline method for certain value of $k$?
3.1. Influence of the input variable
Having a look at the scatter plots, we can observe that some observation variables seem to have a more clear relationship with the target value. Thus, we can expect that not all variables are equally useful for the regression task. In the following plot, we carry out a study of the performance that can be achieved with each variable.
Note that, in practice, the test labels are not available for the selection of hyperparameter
$k$, so we should be careful about the conclusions of this experiment. A more realistic approach will be studied later when we introduce the concept of model validation.
Can you write Python code for it?

k_max = 20

var_performance = []
k_values = []

for var in range(X_tr.shape[1]):

    MSE_tr = [square_error(S_tr, knn_regression(X_tr[:,var], S_tr, X_tr[:, var], k))
              for k in range(1, k_max+1)]
    MSE_tst = [square_error(S_tst, knn_regression(X_tr[:,var], S_tr, X_tst[:, var], k))
               for k in range(1, k_max+1)]
    MSE_tr = np.asarray(MSE_tr)
    MSE_tst = np.asarray(MSE_tst)

    # We select the variable associated to the value of k for which the training error is minimum
    pos = np.argmin(MSE_tr)
    k_values.append(pos + 1)
    var_performance.append(MSE_tst[pos])

plt.stem(range(X_tr.shape[1]), var_performance, use_line_collection=True)
plt.title('Results of unidimensional regression ($k$NN)')
plt.xlabel('Variable')
plt.ylabel('Test MSE')

plt.figure(2)
plt.stem(range(X_tr.shape[1]), k_values, use_line_collection=True)
plt.xlabel('Variable')
plt.ylabel('$k$')
plt.title('Selection of the hyperparameter')
plt.show()


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
In this case, we can check that the average test square error is much lower than the error that was achieved when using only one variable, and also far better than the baseline method. It is also interesting to note that in this particular case the best performance is achieved for a small value of $k$, with the error increasing for larger values of the hyperparameter.
Nevertheless, as we discussed previously, these results should be taken carefully. How would we select the value of $k$, if test labels are (obvioulsy) not available for model validation?
5. Hyperparameter selection via cross-validation
5.1. Generalization
An inconvenient of the application of the $k$-nn method is that the selection of $k$ influences the final error of the algorithm. In the previous experiments, we kept the value of $k$ that minimized the square error on the training set. However, we also noticed that the location of the minimum is not necessarily the same from the perspective of the test data. Ideally, we would like that the designed regression model works as well as possible on future unlabeled patterns that are not available during the training phase. This property is known as generalization.
Fitting the training data is only pursued in the hope that we are also indirectly obtaining a model that generalizes well. In order to achieve this goal, there are some strategies that try to guarantee a correct generalization of the model. One of such approaches is known as cross-validation
5.2. Cross-validation
Since using the test labels during the training phase is not allowed (they should be kept aside to simultate the future application of the regression model on unseen patterns), we need to figure out some way to improve our estimation of the hyperparameter that requires only training data. Cross-validation allows us to do so by following the following steps:

Split the training data into several (generally non-overlapping) subsets. If we use $M$ subsets, the method is referred to as $M$-fold cross-validation. If...
Can you write Python code for it?


I am working on the file "R2.kNN_Regression/regression_knn_student.ipynb".
The first task is:
for any input ${\bf x}$.
Exercise 1
Compute the mean square error over training and test sets, for the baseline estimation  method.
Can you write Python code for it?


I am working on the file "R2Py/chaos.ipynb".
The first task is:
透视表
~ pivot_table
Can you write Python code for it?

df = pd.read_csv('data/_all_atlogger2.csv')
df.head()
#df.columns

df.plot(kind='area', stacked=False)

df.plot(kind='kde', stacked=False)

import pandas as pd
import matplotlib.pyplot as plt

ls data/pom_*

df = pd.read_csv('data/pom_1302.txt')
df.head()


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regresion_LS-checkpoint.ipynb".
The first task is:
1.1. Parametric model
Parametric regression models assume a parametric expression for the regression curve, adjusting the free parameters according to some criterion that measures the quality of the proposed model.

For a unidimensional case like the one in the previous figure, a convenient approach is to recur to polynomial expressions:

$${\hat s}(x) = f(x) = w_0 + w_1 x + w_2 x^2 + \dots + w_{m-1} x^{m-1}$$

For multidimensional regression, polynomial expressions can include cross-products of the variables. For instance, for a case with two input variables, the degree 2 polynomial would be given by

$${\hat s}({\bf x}) = f({\bf x}) = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_2^2 + w_5 x_1 x_2$$

A linear model for multidimensional regression can be expressed as

$${\hat s}({\bf x}) = f({\bf x}) = w_0 + {\bf w}^\top {\bf x}$$
When we postulate such models, the regression model is reduced to finding the most appropriate values of the parameters ${\bf w} = [w_i]$.
All the previous models have in common the fact that they are linear in the parameters, even though they can implement highly non-linear functions. All the derivations in this notebook are equally valid for other non-linear transformations of the input variables, as long as we keep linear-in-the-parameters models.
Can you write Python code for it?

## Next, we represent some random polynomial functions for degrees between 0 and 14

max_degree = 15
K = 200

#Values of X to evaluate the function
X_grid = np.linspace(-1.5, 1.5, K)

for idx in range(max_degree):
    x1 = plt.subplot(3,5, idx+1)
    x1.get_xaxis().set_ticks([])
    x1.get_yaxis().set_ticks([])

    for kk in range(5):
        #Random generation of coefficients for the model
        we = np.random.randn(idx+1, 1)

        #Evaluate the polynomial with previous coefficients at X_grid values
        fout = np.polyval(we, X_grid)
        x1.plot(X_grid,fout,'g-')
        x1.set_ylim([-5,5])


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regresion_LS-checkpoint.ipynb".
The first task is:
Should we choose a polynomial?

What degree should we use for the polynomial?

For a given degree, how do we choose the weights?

For now, we will find the single "best" polynomial. In a future session, we will see how we can design methods that take into account different polynomia simultaneously.
Next, we will explain how to choose optimal weights according to Least-Squares criterion.
2. Least squares regression
2.1. Problem definition

The goal is to learn a (possibly non-linear) regression model from a set of $K$ labeled points, ${{\bf x}k,s_k}{k=0}^{K-1}$.

We assume a parametric function of the form:

$${\hat s}({\bf x}) = f({\bf x}) = w_0 z_0({\bf x}) + w_1 z_1({\bf x}) + \dots w_{m-1} z_{m-1}({\bf x})$$
where $z_i({\bf x})$ are particular transformations of the input vector variables.
Some examples are:

If ${\bf z} = {\bf x}$, the model is just a linear combination of the input variables

If ${\bf z} = \left[\begin{array}{c}1\{\bf x}\end{array}\right]$, we have again a linear combination with the inclusion of a constant term.

For unidimensional input $x$, ${\bf z} = [1, x, x^2, \dots,x^{m-1}]^\top$ would implement a polynomia of degree $m-1$.

Note that the variables of ${\bf z}$ could also be computed combining different variables of ${\bf x}$. E.g., if ${\bf x} = [x_1,x_2]^\top$, a degree-two polynomia would be implemented with
      $${\bf z} = \left[\begin{array}{c}1\x_1\x_2\x_1^2\x_2^2\x_1 x_2\end{array}\right]$$

The above expression does not assume a polynomial model. For instance, we could consider ${\bf z} = [\log(x_1),\log(x_2)]$

Least squares (LS) regression finds the coefficients of the model with the aim of minimizing the square of the residuals. If we define ${\bf w} = [w_0,w_1,\dots,w_{m-1}]^\top$, the LS solution would be defined as
\begin{equation}
{\bf w}{LS} = \arg \min{\bf w} \sum_{k=0}^{K-1} [e_k]^2 = \arg \min_{\bf w} \sum_{k=0}^{K-1} \left[s_k - {\hat s}_k \right]^2
\end{equation}
2.2. Vector Notation
In order to solve the LS problem it is convenient to...
Can you write Python code for it?

n_points = 20
n_grid = 200
frec = 3
std_n = 0.2
max_degree = 20

colors = 'brgcmyk'

#Location of the training points
X_tr = (3 * np.random.random((n_points,1)) - 0.5)

#Labels are obtained from a sinusoidal function, and contaminated by noise
S_tr = np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)

#Equally spaced points in the X-axis
X_grid = np.linspace(np.min(X_tr),np.max(X_tr),n_grid)

#We start by building the Z matrix
Z = []
for el in X_tr.tolist():
    Z.append([el[0]**k for k in range(max_degree+1)])
Z = np.matrix(Z)

Z_grid = []
for el in X_grid.tolist():
    Z_grid.append([el**k for k in range(max_degree+1)])
Z_grid = np.matrix(Z_grid)

plt.plot(X_tr,S_tr,'b.')

for k in [1, 2, n_points]: # range(max_degree+1):
    Z_iter = Z[:,:k+1]

    # Least square solution
    #w_LS = (np.linalg.inv(Z_iter.T.dot(Z_iter))).dot(Z_iter.T).dot(S_tr)

    # Least squares solution, with leass numerical errors
    w_LS, resid, rank, s = np.linalg.lstsq(Z_iter, S_tr, rcond=None)
    #estimates at all grid points
    fout = Z_grid[:,:k+1].dot(w_LS)
    fout = np.array(fout).flatten()
    plt.plot(X_grid,fout,colors[k%len(colors)]+'-',label='Degree '+str(k))

plt.legend(loc='best')
plt.ylim(1.2*np.min(S_tr), 1.2*np.max(S_tr))
plt.show()


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regression_intro_knn-checkpoint.ipynb".
The first task is:
1. Introduction to Regression. The $k$ nearest neighbors method
1.1 Introduction to the Estimation and Regression problems
In this session we will introduce the problem of estimating the value of a target variable $s$ from the knowledge of some variables statistically related to it, $x_i,~i=1,\dots,N$. We will collect all these variables in a vector $\bf x$, to which we will refer as pattern or observation vector. The regression problem can be illustrated as follows:

The statistical relation between $s$ and $\bf x$ can be given by:

A training set, consisting of several observation vectors for which the true target variable is known ${{\bf x}^{(k)}, s^{(k)}}_{k=1}^K$
The joint distribution $p(s,{\bf x})$

In this session (and also in the course) we mostly focus on the first case above, to which we refer as a machine learning approach (i.e., learning from data)
Designing a classifier is equivalent to designing a function that takes the observation vector as the input argument and outputs the estimated value of the target variable
Cost functions
In practice, once we have build a regressor, it will be used in many experiments, each of them consisting of different values of the target variable and also different observations (even for a common $s$).
The real goal of the regression is not to adjust the training set, but to make accurate predictions on future data
This requirement for accurateness implies that we need some way of measuring and penalizing the discrepancy between the true and estimated values of $s$. This is done by defining a cost function, such as:

Square error: $c(e) = e^2 = (s - {\hat s})^2$
Absolute error: $c(e) = |e| = |s - {\hat s}|$

The above expressions apply to a particular application of the regression model. In practice, we would like to obtain an appropriate average performance, i.e., when averaging over the future applications of the design.
Since we only have labels for the training data, some idea about the performance of the...
Can you write Python code for it?

# In this section we will plot together the square and absolute errors
grid = np.linspace(-3,3,num=100)
plt.plot(grid,grid**2,'b-',label='Square error')
plt.plot(grid,np.absolute(grid),'r--',label='Absolute error')
plt.xlabel('Error')
plt.ylabel('Cost')
plt.legend(loc='best')


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regression_intro_knn-checkpoint.ipynb".
The first task is:
1.3 Baseline estimation. Using the average of the training set labels
A first very simple method to build the regression model is to use the average of all the target values in the training set as the output of the model, discarding the value of the observation input vector.
This approach can be considered as a baseline, given that any other method making an effective use of the observation variables, statistically related to $s$, should improve the performance of this method.
Can you write Python code for it?

#We start by defining a function that calculates the average square error
def square_error(s,s_est):
    y = np.mean((s - s_est)**2)
    return y

s_hat = np.mean(S_tr)
print 'Average square error in the training set (baseline method): ' \
    + str(square_error(S_tr,s_hat))
print 'Average square error in the test set (baseline method): ' \
    + str(square_error(S_tst,s_hat))


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regression_intro_knn-checkpoint.ipynb".
The first task is:
1.7 Scikit-learn implementation
In practice, most well-known machine learning methods are implemented and available for python. Probably, the most complete module for machine learning tools is Scikit-learn. The following piece of code uses the method
KNeighborsRegressor

available in Scikit-learn. The example has been taken from here. As you can check, this routine allows us to build the estimation for a particular point using a weighted average of the targets of the neighbors:
To obtain the estimation at a point ${\bf x}$:

Find $k$ closest points to ${\bf x}$ in the training set
Average the corresponding targets, weighting each value according to the distance of each point to ${\bf x}$, so that closer points have a larger influence in the estimation.
Can you write Python code for it?


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regression_intro_knn-checkpoint.ipynb".
The first task is:
Discussion topics and exercises:

Analize the training square error for $k=0$
Modify the code above to visualize the square error from $k=1$ up to $k$ equal to the number of training instances. Can you relate the square error of the $k$-NN method with that of the baseline method for certain value of $k$?
Modify the code above to print on the screen the exact value of $k$ minimizing the test square error.

Analysis of the performance for the different variables
Having a look at the scatter plots, we can observe that some observation variables seem to have a more clear relationship with the target value. For instance, variable 1 seems to have a linear correlation, in spite of the noise. Thus, we can expect that not all variables are equally useful for the regression task. In the following plot, we carry out a study of the performance that can be achieved with each variable. Since the test labels cannot be used to select the hyperparamenter $k$, we will pick the value that minimizes the training error (for each variable).
Can you write Python code for it?

k_max = 20

var_performance = []
k_values = []

for var in range(X_tr.shape[1]):
    sq_error_tr = [square_error(S_tr,knn_regression(X_tr[:,var],S_tr,X_tr[:,var],k+1)) for k in range(k_max)]
    sq_error_tst = [square_error(S_tst,knn_regression(X_tr[:,var],S_tr,X_tst[:,var],k+1)) for k in range(k_max)]
    sq_error_tr = np.asarray(sq_error_tr)
    sq_error_tst = np.asarray(sq_error_tst)

    # We select the variable associated to the value of k for which the training error is minimum
    pos = np.argmin(sq_error_tr)
    k_values.append(pos+1)
    var_performance.append(sq_error_tst[pos])

plt.stem(range(X_tr.shape[1]),var_performance)
plt.title('Results of unidimensional regression (kNN)')
plt.xlabel('Variable')
plt.ylabel('Quadratic error')

plt.figure(2)
plt.stem(range(X_tr.shape[1]),k_values)
plt.xlabel('Variable')
plt.ylabel('k')
plt.title('Selection of the hyperparameter')


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regression_intro_knn-checkpoint.ipynb".
The first task is:
Parametric and non-parametric regression models
Generally speaking, we can distinguish two approaches when designing a regression model:

Parametric approach: In this case, the estimation function is given a priori a parametric form, and the goal of the design is to find the most appropriate values of the parameters according to a certain goal

For instance, we could assume a linear expression
   $${\hat s}({\bf x}) = {\bf w}^\top {\bf x}$$
   And adjust the parameter vector in order to minimize the average of the quadratic error over the training data. This is known as least-squares regression, and we will study it in a future session.

Non-parametric approach: In this case, the analytical shape of the regression model is not assumed a priori.

Today, we will explain some important facts about the regression problem, recurring to a non-parametric method known as the k-nearest neighbors approach.
1.2 The dataset
We describe next the regression task that we will use in the session. The dataset is an adaptation of the  STOCK dataset, taken originally from the  StatLib Repository. The goal of this problem is to predict the values of the stocks of a given airplane company, given the values of another 9 companis in the same day.
 (Alternatively, if you are reading this text from the python notebook with its full functionality, you can explore the results of the regression experiments using the
CONCRETE dataset which is taken from the Machine Learning Repository at the University of California Irvine. To do so, just uncomment the block of code entitled CONCRETE and place comments in STOCK in the cell bellow. Remind that you must run the cells again to see the changes. The goal of the CONCRETE dataset tas is to predict the compressive...
Can you write Python code for it?

# Let us start by loading the data into the workspace, and visualizing the dimensions of all matrices

# CONCRETE DATASET.
# Use this code block to use the concrete dataset.
#data = scipy.io.loadmat('concrete.mat')
#X_tr = data['X_tr']
#X_tst = data['X_tst']
#S_tr = data['S_tr']
#S_tst = data['S_tst']

# STOCK
# Use this code block to use the airplanes dataset.
data = scipy.io.loadmat('stock.mat')
X_tr = data['xTrain']
X_tst = data['xTest']
S_tr = data['sTrain']
S_tst = data['sTest']

print X_tr.shape
print S_tr.shape
print X_tst.shape
print S_tst.shape


I am working on the file "R3.Least_Squares/.ipynb_checkpoints/regression_intro_knn-checkpoint.ipynb".
The first task is:
Questions
Please, reflect about the following issues:

What is the difference between a variable and a pattern? How many of each do we have in this case?
What is the legal use of the test labels S_tst ?
What is the legal use of the test observation vectors `X_tst`` ?

Scatter plots
We can get a first rough idea about the regression task if we plot each of the one-dimensional variables against the target data. These representations are known as <i>scatter plots</i>
Can you write Python code for it?

pylab.subplots_adjust(hspace=0.2)

for idx in range(X_tr.shape[1]):
    ax1 = plt.subplot(3,3,idx+1)
    ax1.plot(X_tr[:,idx],S_tr,'.')
    ax1.get_xaxis().set_ticks([])
    ax1.get_yaxis().set_ticks([])
    #plt.ylabel('Compressive strength')

plt.show()


I am working on the file "R3.Least_Squares/regresion_LS.ipynb".
The first task is:
1.1. Parametric model
Parametric regression models assume a parametric expression for the regression curve, adjusting the free parameters according to some criterion that measures the quality of the proposed model.

For a unidimensional case like the one in the previous figure, a convenient approach is to recur to polynomial expressions:

$${\hat s}(x) = f(x) = w_0 + w_1 x + w_2 x^2 + \dots + w_{m-1} x^{m-1}$$

For multidimensional regression, polynomial expressions can include cross-products of the variables. For instance, for a case with two input variables, the degree 2 polynomial would be given by

$${\hat s}({\bf x}) = f({\bf x}) = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_2^2 + w_5 x_1 x_2$$

A linear model for multidimensional regression can be expressed as

$${\hat s}({\bf x}) = f({\bf x}) = w_0 + {\bf w}^\top {\bf x}$$
When we postulate such models, the regression model is reduced to finding the most appropriate values of the parameters ${\bf w} = [w_i]$.
All the previous models have in common the fact that they are linear in the parameters, even though they can implement highly non-linear functions. All the derivations in this notebook are equally valid for other non-linear transformations of the input variables, as long as we keep linear-in-the-parameters models.
Can you write Python code for it?


I am working on the file "R3.Least_Squares/regresion_LS.ipynb".
The first task is:
2.4.1 Limitations of the LS approach. The need for assumptions
Another way to visualize the effect of overfiting is to analyze the effect of variations o a single sample. Consider a training dataset consisting of 15 points which are given, and depict the regression curves that would be obtained if adding an additional point at a fixed location, depending on the target value of that point:
(You can run this code fragment several times, to check also the changes in the regression curves between executions, and depending also on the location of the training points)
Can you write Python code for it?

n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
n_val_16 = 5
degree = 18

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-.5,2.5,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

X_16 = .3 * np.ones((n_val_16,))
S_16 = np.linspace(np.min(S_tr),np.max(S_tr),n_val_16)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)
ax.plot(X_16,S_16,'ro',markersize=6)
ax.plot(X_grid,S_grid,'r-',label='True model')

for el in zip(X_16,S_16):
    #Add point to the training set
    X_tr_iter = np.append(X_tr,el[0])
    S_tr_iter = np.append(S_tr,el[1])

    #Obtain LS regression coefficients and evaluate it at X_grid
    w_LS = np.polyfit(X_tr_iter, S_tr_iter, degree)
    S_grid_iter = np.polyval(w_LS,X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

ax.set_xlim(-.5,2.5)
ax.set_ylim(S_16[0]-2,S_16[-1]+2)
ax.legend(loc='best')
plt.show()


I am working on the file "R3.Least_Squares/regresion_LS.ipynb".
The first task is:
Should we choose a polynomial?

What degree should we use for the polynomial?

For a given degree, how do we choose the weights?

For now, we will find the single "best" polynomial. In a future session, we will see how we can design methods that take into account different polynomia simultaneously.
Next, we will explain how to choose optimal weights according to Least-Squares criterion.
2. Least squares regression
2.1. Problem definition

The goal is to learn a (possibly non-linear) regression model from a set of $K$ labeled points, ${{\bf x}k,s_k}{k=0}^{K-1}$.

We assume a parametric function of the form:

$${\hat s}({\bf x}) = f({\bf x}) = w_0 z_0({\bf x}) + w_1 z_1({\bf x}) + \dots w_{m-1} z_{m-1}({\bf x})$$
where $z_i({\bf x})$ are particular transformations of the input vector variables.
Some examples are:

If ${\bf z} = {\bf x}$, the model is just a linear combination of the input variables

If ${\bf z} = \left[\begin{array}{c}1\{\bf x}\end{array}\right]$, we have again a linear combination with the inclusion of a constant term.

For unidimensional input $x$, ${\bf z} = [1, x, x^2, \dots,x^{m-1}]^\top$ would implement a polynomia of degree $m-1$.

Note that the variables of ${\bf z}$ could also be computed combining different variables of ${\bf x}$. E.g., if ${\bf x} = [x_1,x_2]^\top$, a degree-two polynomia would be implemented with
      $${\bf z} = \left[\begin{array}{c}1\x_1\x_2\x_1^2\x_2^2\x_1 x_2\end{array}\right]$$

The above expression does not assume a polynomial model. For instance, we could consider ${\bf z} = [\log(x_1),\log(x_2)]$

Least squares (LS) regression finds the coefficients of the model with the aim of minimizing the square of the residuals. If we define ${\bf w} = [w_0,w_1,\dots,w_{m-1}]^\top$, the LS solution would be defined as
\begin{equation}
{\bf w}{LS} = \arg \min{\bf w} \sum_{k=0}^{K-1} [e_k]^2 = \arg \min_{\bf w} \sum_{k=0}^{K-1} \left[s_k - {\hat s}_k \right]^2
\end{equation}
2.2. Vector Notation
In order to solve the LS problem it is convenient to...
Can you write Python code for it?


I am working on the file "R3.Least_Squares/regresion_LS.ipynb".
The first task is:
This notebook covers the problem of fitting parametric regression models with a minimum least-squares criterion. The material presented here is based on the first lectures of this Machine Learning course. In particular, you can refer to the following presentation:  Probabilistic Regression.
1. A parametric approach to the regression problem
We have already presented the goal of regression. Given that we have access to a set of training points, ${{\bf x}k, s_k}{k=0}^{K-1}$, the goal is to learn a function $f({\bf x})$ that we can use to make good predictions for an arbitrary input vector.
The following plot illustrates a regression example for unidimensional input data. We have also generated three different regression curves corresponding to polynomia of degrees 1, 2, and 3 with random coefficients.
Can you write Python code for it?

K = 35
n_grid = 200
frec = 3
std_n = 0.3

# Location of the training points
X_tr = (3 * np.random.random((K, 1)) - 0.5)

# Labels are obtained from a sinusoidal function, and contaminated by noise
S_tr = np.cos(frec*X_tr) + std_n * np.random.randn(K, 1)

# Equally spaced points in the X-axis
X_grid = np.linspace(np.min(X_tr),np.max(X_tr),n_grid)

# Gererate random prediction curves
f1 = np.random.random() + np.random.random()*X_grid
f2 = np.random.random() + np.random.random()*X_grid + \
     np.random.random()*(X_grid**2)
f3 = np.random.random() + np.random.random()*X_grid + \
     np.random.random()*(X_grid**2) + np.random.random()*(X_grid**3)

plt.plot(X_tr,S_tr,'b.')
plt.plot(X_grid,f1.T,'g-',label='Arbitrary Linear function')
plt.plot(X_grid,f2.T,'r-',label='Arbitrary Quadratic function')
plt.plot(X_grid,f3.T,'m-',label='Arbitrary Cubic function')

plt.legend(loc='best')
plt.show()


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML-checkpoint.ipynb".
The first task is:
1.1. Represent a scatter plot of the data points
Can you write Python code for it?

# <SOL>
plt.figure()
plt.scatter(X, s)
plt.xlabel('x')
plt.ylabel('s')
plt.show()
# </SOL>


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML-checkpoint.ipynb".
The first task is:
1.2. Compute the ML estimate.
Can you write Python code for it?

# Note that, to use lstsq, the input matrix must be K x 1
Xcol = X[:,np.newaxis]

# Compute the ML estimate using linalg.lstsq from Numpy.
wML = np.linalg.lstsq(Xcol, s)[0]


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML-checkpoint.ipynb".
The first task is:
1.3. Plot the likelihood as a function of parameter $w$ along the interval $-0.5\le w \le 2$, verifying that the ML estimate takes the maximum value.
Can you write Python code for it?

K = len(s)
wGrid = np.arange(-0.5, 2, 0.01)

p = []
for w in wGrid:
    d = s - X*w
    # p.append(<FILL IN>)
    p.append((1.0/(np.sqrt(2*np.pi)*sigma_eps))**K * np.exp(-np.dot(d, d)))

# Compute the likelihood for the ML parameter wML
# d = <FILL IN>
d = s-X*wML
# pML = [<FILL IN>]
pML = [(1.0/(np.sqrt(2*np.pi)*sigma_eps))**K * np.exp(-np.dot(d, d))]

# Plot the likelihood function and the optimal value
plt.figure()
plt.plot(wGrid, p)
plt.stem(wML, pML)
plt.xlabel('$w$')
plt.ylabel('Likelihood function')
plt.show()


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML-checkpoint.ipynb".
The first task is:
1.4. Plot the prediction function over the data scatter plot
Can you write Python code for it?

x = np.arange(0, 1.2, 0.01)
# sML = <FILL IN>
sML = wML * x

plt.figure()
plt.scatter(X, s)
# plt.plot(<FILL IN>)
plt.plot(x, sML)
plt.xlabel('x')
plt.ylabel('s')
plt.axis('tight')
plt.show()


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML-checkpoint.ipynb".
The first task is:
Exercise 2:
Assume the dataset $\mathcal{D} = {(x^{(k)}, s^{(k)}, k=1,\ldots, K}$ contains i.i.d. samples from a distribution with posterior density given by
$$
p(s|x, w) = w x \exp(- w x s), \qquad s\ge0, \,\, x\ge 0, \,\, w\ge 0
$$

2.1. Determine an expression for the likelihood function

Solution:
The likelihood function is
$$
p({\bf s}|w, {\bf X})
    = \prod_{k=1}^K w x^{(k)} \exp(- w x^{(k)} s^{(k)})
    = w^K \left(\prod_{k=1}^K x^{(k)}\right)  \exp\left(- w \sum_{k=1}^K x^{(k)} s^{(k)}\right)
$$

2.2. Draw the likelihood function for the dataset in Exercise 1 in the range $0\le w\le 6$.
Can you write Python code for it?

K = len(s)
wGrid = np.arange(0, 6, 0.01)

p = []
Px = np.prod(X)
xs = np.dot(X,s)
for w in wGrid:
    # p.append(<FILL IN>)
    p.append((w**K)*Px*np.exp(-w*xs))

plt.figure()
# plt.plot(<FILL IN>)
plt.plot(wGrid, p)
plt.xlabel('$w$')
plt.ylabel('Likelihood function')
plt.show()


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML_student-checkpoint.ipynb".
The first task is:
1.1. Represent a scatter plot of the data points
Can you write Python code for it?


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML_student-checkpoint.ipynb".
The first task is:
1.2. Compute the ML estimate.
Can you write Python code for it?

# Note that, to use lstsq, the input matrix must be K x 1
Xcol = X[:,np.newaxis]

# Compute the ML estimate using linalg.lstsq from Numpy.
# wML = <FILL IN>


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML_student-checkpoint.ipynb".
The first task is:
1.3. Plot the likelihood as a function of parameter $w$ along the interval $-0.5\le w \le 2$, verifying that the ML estimate takes the maximum value.
Can you write Python code for it?

K = len(s)
wGrid = np.arange(-0.5, 2, 0.01)

p = []
for w in wGrid:
    d = s - X*w
    # p.append(<FILL IN>)

# Compute the likelihood for the ML parameter wML
# d = <FILL IN>
# pML = [<FILL IN>]

# Plot the likelihood function and the optimal value
plt.figure()
plt.plot(wGrid, p)
plt.stem(wML, pML)
plt.xlabel('$w$')
plt.ylabel('Likelihood function')
plt.show()


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML_student-checkpoint.ipynb".
The first task is:
2.3. Determine the coefficient $w_\text{ML}$ of the linear prediction function (i.e., using ${\bf Z}={\bf X}$).
(Hint: you can maximize the log of the likelihood function instead of the likelihood function in order to simplify the differentiation)

Solution:
<SOL>
</SOL>

2.4. Compute $w_\text{ML}$ for the dataset in Exercise 1
Can you write Python code for it?


I am working on the file "R4.ML_Regression/.ipynb_checkpoints/Regression_ML_student-checkpoint.ipynb".
The first task is:
Exercise 2:
Assume the dataset $\mathcal{D} = {(x^{(k)}, s^{(k)}, k=1,\ldots, K}$ contains i.i.d. samples from a distribution with posterior density given by
$$
p(s|x, w) = w x \exp(- w x s), \qquad s\ge0, \,\, x\ge 0, \,\, w\ge 0
$$

2.1. Determine an expression for the likelihood function

Solution:
<SOL>
</SOL>

2.2. Draw the likelihood function for the dataset in Exercise 1 in the range $0\le w\le 6$.
Can you write Python code for it?

K = len(s)
wGrid = np.arange(0, 6, 0.01)

p = []
Px = np.prod(X)
xs = np.dot(X,s)
for w in wGrid:
    # p.append(<FILL IN>)

plt.figure()
# plt.plot(<FILL IN>)
plt.xlabel('$w$')
plt.ylabel('Likelihood function')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
2.1. Model assumptions
In order to solve exercise 4 we have taken advantage of the statistical independence of the noise components. Some independence assumptions are required in general to compute the ML estimate in other scenarios.
In order to estimate ${\bf w}$ from the training data in a mathematicaly rigorous and compact form let us group the target variables into a vector
$$
{\bf s} = \left(s_0, \dots, s_{K-1}\right)^\top
$$
and the input vectors into a matrix
$$
{\bf X} = \left({\bf x}0, \dots, {\bf x}{K-1}\right)^\top
$$
We will make the following assumptions:

A1. All samples in ${\cal D}$ have been generated by the same distribution, $p({\bf x}, s \mid {\bf w})$
A2. Input variables ${\bf x}$ do not depend on ${\bf w}$. This implies that
$$
p({\bf X} \mid {\bf w}) = p({\bf X})
$$
A3. Targets $s_{0},\ldots, s_{K-1}$ are statistically independent, given ${\bf w}$ and the inputs ${\bf x}0,\ldots, {\bf x}{K-1}$, that is:
$$
p({\bf s} \mid {\bf X}, {\bf w}) = \prod_{k=0}^{K-1} p(s_k \mid {\bf x}_k, {\bf w})
$$

Since ${\cal D} = ({\bf X}, {\bf s})$, we can write
$$p(\mathcal{D}|{\bf w})
    = p({\bf s}, {\bf X}|{\bf w})
    = p({\bf s} | {\bf X}, {\bf w}) p({\bf X}|{\bf w})
$$
Using assumption A2,
$$
p(\mathcal{D}|{\bf w})
    = p({\bf s} | {\bf X}, {\bf w}) p({\bf X})
$$
and, finally, using assumption A3, we can express the estimation problem as the computation of
\begin{align}
\hat{\bf w}{\text{ML}}
                &= \arg \max{\bf w} p({\bf s}|{\bf X},{\bf w})  \
   \qquad \quad &= \arg \max_{\bf w} \prod_{k=0}^{K-1} p(s_k \mid {\bf x}k, {\bf w})  \
   \qquad \quad &= \arg \max{\bf w} \sum_{k=0}^{K-1}\log  p(s_k \mid {\bf x}_k, {\bf w})
\end{align}
Any of the last three terms can be used to optimize ${\bf w}$. The sum in the last term is usually called the log-likelihood function, $L({\bf w})$, whereas the product in the previous line is simply referred as the likelihood function.
2.2. Summary.
Let's summarize what we need to do in order to design a regression algorithm based on ML...
Can you write Python code for it?


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
4.1. Represent a scatter plot of the data points
Can you write Python code for it?


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
4.2. Compute the ML estimate
Can you write Python code for it?

# wML = <FILL IN>
wML = np.sum(X*s) / np.sum(X*X)

print("The ML estimate is {}".format(wML))


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
4.3. Plot the likelihood as a function of parameter $w$ along the interval $-0.5\le w \le 2$, verifying that the ML estimate takes the maximum value.
Can you write Python code for it?

sigma_eps = 1
K = len(s)
wGrid = np.arange(-0.5, 2, 0.01)

p = []
for w in wGrid:
    d = s - X*w
    # p.append(<FILL IN>)
    p.append((1.0/(np.sqrt(2*np.pi)*sigma_eps))**K * np.exp(-np.dot(d, d) / (2*sigma_eps**2)))

# Compute the likelihood for the ML parameter wML
# d = <FILL IN>
d = s-X*wML
# pML = [<FILL IN>]
pML = [(1.0/(np.sqrt(2*np.pi)*sigma_eps))**K * np.exp(-np.dot(d, d) / (2*sigma_eps**2))]

# Plot the likelihood function and the optimal value
plt.figure()
plt.plot(wGrid, p)
plt.stem([wML], pML)
plt.xlabel('$w$')
plt.ylabel('Likelihood function')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
4.4. Plot the prediction function on top of the data scatter plot
Can you write Python code for it?

xgrid = np.arange(0, 1.2, 0.01)
# sML = <FILL IN>
sML = wML * xgrid

plt.figure()
plt.scatter(X, s)
# plt.plot(<FILL IN>)
plt.plot(xgrid, sML)
plt.xlabel('x')
plt.ylabel('s')
plt.axis('tight')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
5.1. Compute the ML estimate.
Can you write Python code for it?

# Compute the extended input matrix Z
nx = len(X)
# Z = <FILL IN>
Z = np.hstack((np.ones((nx, 1)), X[:,np.newaxis], X[:,np.newaxis]**2))

# Compute the ML estimate using linalg.lstsq from Numpy.
# wML = <FILL IN>
wML = np.linalg.lstsq(Z, s)[0]
print(wML)


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
6.3. Determine the maximum likelihood coefficient, $w_\text{ML}$.

(Hint: you can maximize the log-likelihood function instead of the likelihood function in order to simplify the differentiation)
Solution:
<SOL>
Applyng the logarithm to the likelihood function we get
$$
\log p({\bf s}|{\bf w}, {\bf X})
    = K\log  w + \sum_{k=0}^{K-1} \log\left(x_k\right)  - w {\bf X}^\top {\bf s}
$$
which is minimum for
$$
w_\text{ML} = \frac{K}{{\bf X}^\top {\bf s}}
$$
</SOL>

6.4. Compute $w_\text{ML}$ for the dataset in Exercise 4
Can you write Python code for it?

# wML = <FILL IN>
wML = np.float(K) /xs
print(wML)


I am working on the file "R4.ML_Regression/Regression_ML_professor.ipynb".
The first task is:
6.5. Assuming $w = w_\text{ML}$, compute the prediction function based on the estimate $s_{MSE}$

Solution:
<SOL>
$$
\hat{s}{\text{MSE}} = \mathbb{E}{s |x, w} = \int s w x \exp(w\text{ML} x s) ds = \frac{1}{w_\text{ML} x}
$$
</SOL>

6.6. Plot the prediction function obtained in ap. 6.5, and compare it with the linear predictor in exercise 4
Can you write Python code for it?

xgrid = np.arange(0.1, 1.2, 0.01)
# sML = <FILL IN>
sML = 1 / (wML * xgrid)

plt.figure()
plt.scatter(X, s)
# plt.plot(<FILL IN>)
plt.plot(xgrid, sML)
plt.xlabel('x')
plt.ylabel('s')
plt.axis('tight')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
4.3. Plot the likelihood as a function of parameter $w$ along the interval $-0.5\le w \le 2$, verifying that the ML estimate takes the maximum value.
Can you write Python code for it?

sigma_eps = 1
K = len(s)
wGrid = np.arange(-0.5, 2, 0.01)

p = []
for w in wGrid:
    d = s - X*w
    # p.append(<FILL IN>)

# Compute the likelihood for the ML parameter wML
# d = <FILL IN>
# pML = [<FILL IN>]

# Plot the likelihood function and the optimal value
plt.figure()
plt.plot(wGrid, p)
plt.stem([wML], pML)
plt.xlabel('$w$')
plt.ylabel('Likelihood function')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
4.4. Plot the prediction function on top of the data scatter plot
Can you write Python code for it?

xgrid = np.arange(0, 1.2, 0.01)
# sML = <FILL IN>

plt.figure()
plt.scatter(X, s)
# plt.plot(<FILL IN>)
plt.xlabel('x')
plt.ylabel('s')
plt.axis('tight')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
5.1. Compute the ML estimate.
Can you write Python code for it?

# Compute the extended input matrix Z
nx = len(X)
# Z = <FILL IN>

# Compute the ML estimate using linalg.lstsq from Numpy.
# wML = <FILL IN>
print(wML)


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
5.3. Plot the prediction function over the data scatter plot
Can you write Python code for it?

xgrid = np.arange(0, 1.2, 0.01)
nx = len(xgrid)
# Compute the input matrix for the grid data in x
# Z = <FILL IN>

# sML = <FILL IN>

plt.figure()
plt.scatter(X, s)
# plt.plot(<FILL IN>)
plt.xlabel('x')
plt.ylabel('s')
plt.axis('tight')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
6.3. Determine the maximum likelihood coefficient, $w_\text{ML}$.

(Hint: you can maximize the log-likelihood function instead of the likelihood function in order to simplify the differentiation)
Solution:
<SOL>
</SOL>

6.4. Compute $w_\text{ML}$ for the dataset in Exercise 4
Can you write Python code for it?


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
6.5. Assuming $w = w_\text{ML}$, compute the prediction function based on the estimate $s_{MSE}$

Solution:
<SOL>
</SOL>

6.6. Plot the prediction function obtained in ap. 6.5, and compare it with the linear predictor in exercise 4
Can you write Python code for it?

xgrid = np.arange(0.1, 1.2, 0.01)
# sML = <FILL IN>

plt.figure()
plt.scatter(X, s)
# plt.plot(<FILL IN>)
plt.xlabel('x')
plt.ylabel('s')
plt.axis('tight')
plt.show()


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
A quick note on the mathematical notation
In this notebook we will make extensive use of probability distributions. In general, we will use capital leters
${\bf X}$, $S$, $E$ ..., to denote random variables, and lower-case letters ${\bf x}$, $s$, $\epsilon$ ..., to denote the values they can take.
In general, we will use letter $p$ for probability density functions (pdf). When necessary, we will use, capital subindices to make the random variable explicit. For instance, $p_{{\bf X}, S}({\bf x}, s)$ would be the joint pdf of random variables ${\bf X}$ and $S$ at values ${\bf x}$ and $s$, respectively.
However, to avoid a notation overload, we will omit subindices when they are clear from the context. For instance, we will use $p({\bf x}, s)$ instead of $p_{{\bf X}, S}({\bf x}, s)$.
1. Model-based parametric regression
1.1. The regression problem
Given an observation vector ${\bf x}$, the goal of the regression problem is to find a function $f({\bf x})$ providing good predictions about some unknown variable $s$. To do so, we assume that a set of labelled training examples, ${{\bf x}k, s_k}{k=0}^{K-1}$ is available.
The predictor function should make good predictions for new observations ${\bf x}$ not used during training. In practice, this is tested using a second set (the test set) of labelled samples.
1.2. The underlying model assumption
Many regression algorithms are grounded on the idea that all samples from the training set have been generated
independently by some common stochastic process.
<img src="figs/data_model.png" width=180>
If $p({\bf x}, s)$ were known, we could apply estimation theory to estimate $s$ for a given ${\bf x}$ using $p$. For instance, we could apply any of the following classical estimates:

Maximum A Posterior (MAP): $$\hat{s}_{\text{MAP}} = \arg\max_s p(s| {\bf x})$$
Minimum Mean Square Error (MSE): $$\hat{s}_{\text{MSE}} = \mathbb{E}{S |{\bf x}} = \int s \, p(s| {\bf x}) \, ds $$

Note that, since these estimators depend on $p(s |{\bf x})$, knowing the posterior distribution...
Can you write Python code for it?

# wML = <FILL IN>

print("The ML estimate is {}".format(wML))


I am working on the file "R4.ML_Regression/Regression_ML_student.ipynb".
The first task is:
Exercise 6:
Assume the dataset $\mathcal{D} = {(x_k, s_k, k=0,\ldots, K-1}$ contains i.i.d. samples from a distribution with posterior density given by
$$
p(s \mid x, w) = w x \exp(- w x s), \qquad s\ge0, \,\, x\ge 0, \,\, w\ge 0
$$

6.1. Determine an expression for the likelihood function

Solution:
<SOL>
</SOL>

6.2. Draw the likelihood function for the dataset in Exercise 4 in the range $0\le w\le 6$.
Can you write Python code for it?


I am working on the file "R5.Bayesian_Regression/.ipynb_checkpoints/Bayesian_regression-checkpoint.ipynb".
The first task is:
1. Model-based parametric regression
1.1. The regression problem.
Given an observation vector ${\bf x}$, the goal of the regression problem is to find a function $f({\bf x})$ providing good predictions about some unknown variable $s$. To do so, we assume that a set of labelled training examples, ${{\bf x}^{(k)}, s^{(k)}}_{k=1}^K$ is available.
The predictor function should make good predictions for new observations ${\bf x}$ not used during training. In practice, this is tested using a second set (the test set) of labelled samples.
NOTE: In the following, we will use capital letters, ${\bf X}$, $S$, ..., to denote random variables, and lower-case letters ${\bf x}$, s, ..., to the denote the values they can take. When there is no ambigüity, we will remove subindices of the density functions, $p_{{\bf X}, S}({\bf x}, s)= p({\bf x}, s)$ to simplify the mathematical notation.
1.2. Model-based parametric regression
Model-based regression methods assume that all data in the training and test dataset habe been generated by some stochastic process. In parametric regression, we assume that the probability distribution generating the data has a known parametric form, but the values of some parameters are unknown.
In particular, in this notebook we will assume the target variables in all pairs $({\bf x}^{(k)}, s^{(k)})$ from the training and test sets have been generated independently from some posterior distribution $p(s| {\bf x}, {\bf w})$, were ${\bf w}$ is some unknown parameter. The training dataset is used to estimate ${\bf w}$.
Once $p(s|{\bf x},{\bf w})$ is known or can be estimated, Estimation Theory can be applied to estimate $s$ for any input ${\bf x}$. For instance, any of these classical estimates can be used:

Maximum A Posterior (MAP): $\qquad\hat{s}_{\text{MAP}} = \arg\max_s p(s| {\bf x}, {\bf w})$
Minimum Mean Square Error (MSE): $\qquad\hat{s}_{\text{MSE}} = \mathbb{E}{S |{\bf x}, {\bf w}}$

<img src="figs/ParametricReg.png", width=300>
1.3.1. Maximum Likelihood (ML) parameter estimation
One way to...
Can you write Python code for it?

n_points = 20
n_grid = 200
frec = 3
std_n = 0.2
degree = 3
nplots = 20

#Prior distribution parameters
sigma_eps = 0.1
mean_w = np.zeros((degree+1,))
sigma_p = 0.03  ### Try increasing this value
var_w = sigma_p**2 * np.eye(degree+1)

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)

xmin = np.min(X_tr)
xmax = np.max(X_tr)
X_grid = np.linspace(xmin-0.2*(xmax-xmin), xmax+0.2*(xmax-xmin),n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

for k in range(nplots):

    #Draw weigths fromt the prior distribution
    w_iter = np.random.multivariate_normal(mean_w, var_w)
    S_grid_iter = np.polyval(w_iter,X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

ax.set_xlim(xmin-0.2*(xmax-xmin), xmax+0.2*(xmax-xmin))
ax.set_ylim(S_tr[0]-2,S_tr[-1]+2)
ax.set_xlabel('$x$')
ax.set_ylabel('$s$')
plt.show()


I am working on the file "R5.Bayesian_Regression/.ipynb_checkpoints/Bayesian_regression-checkpoint.ipynb".
The first task is:
2.2. Summary
Summarizing, the steps to design a Bayesian parametric regresion algorithm are the following:

Assume a parametric data model $p(s| {\bf x},{\bf w})$ and a prior distribution $p({\bf w})$.
Using the data model and the i.i.d. assumption, compute $p({\bf s}|{\bf w})$.
Applying the bayes rule, compute the posterior distribution $p({\bf w}|{\bf s})$.
Compute the MSE estimate of $s$ given ${\bf x}$.

3. Bayesian regression for a Gaussian model.
We will apply the above steps to derive a Bayesian regression algorithm for a Gaussian model.
3.1. Step 1: The Gaussian model.
Let as assume that the likelihood function is given by the Gaussian model described in Sec. 1.3.2.
$$
s~|~{\bf w} \sim {\cal N}\left({\bf z}^\top{\bf w}, \sigma_\varepsilon^2 {\bf I} \right)
$$
and that the prior is also Gaussian
$$
{\bf w} \sim {\cal N}\left({\bf 0},{\bf V}_{p} \right)
$$
3.2. Step 2: Complete data likelihood
Using the i.i.d. assumption,
$$
{\bf s}~|~{\bf w} \sim {\cal N}\left({\bf Z}{\bf w},\sigma_\varepsilon^2 {\bf I} \right)
$$
3.3. Step 3: Posterior weight distribution
The posterior distribution of the weights can be computed using the Bayes rule
$$p({\bf w}|{\bf s}) = \frac{p({\bf s}|{\bf w})~p({\bf w})}{p({\bf s})}$$
Since both $p({\bf s}|{\bf w})$ and $p({\bf w})$ follow a Gaussian distribution, we know also that the joint distribution and the posterior distribution of ${\bf w}$ given ${\bf s}$ are also Gaussian. Therefore,
$${\bf w}~|~{\bf s} \sim {\cal N}\left({\bf w}\text{MSE}, {\bf V}{\bf w}\right)$$
After some algebra, it can be shown that mean and the covariance matrix of the distribution are:
$${\bf V}{\bf w} = \left[\frac{1}{\sigma\varepsilon^2} {\bf Z}^{\top}{\bf Z}
                  + {\bf V}_p^{-1}\right]^{-1}$$
$${\bf w}\text{MSE} = {\sigma\varepsilon^{-2}} {\bf V}_{\bf w} {\bf Z}^\top {\bf s}$$
Exercise 1:
Consider the dataset with one-dimensional inputs given by
Can you write Python code for it?

# True data parameters
w_true = 3
std_n = 0.4

# Generate the whole dataset
n_max = 64
X_tr = 3 * np.random.random((n_max,1)) - 0.5
S_tr =  w_true * X_tr + std_n * np.random.randn(n_max,1)


I am working on the file "R5.Bayesian_Regression/.ipynb_checkpoints/Bayesian_regression-checkpoint.ipynb".
The first task is:
Exercise 2:
Note that, in the example above, the model assumptions are correct: the target variables have been generated by a linear model with noise standard deviation sigma_n which is exactly equal to the value assumed by the model, stored in variable sigma_eps. Check what happens if we take sigma_eps=4*sigma_n or sigma_eps=sigma_n/4.

Does the algorithm fails in that cases?
What differences can you observe with respect to the ideal case sigma_eps=sigma_n?

3.4. Step 4: MSE estimate
Noting that
$$
\mathbb{E}{s|{\bf w}, {\bf x}} = {\bf w}^\top {\bf z}
$$
we can write
$$
\hat{s}\text{MSE}
   = \int {\bf w}^\top {\bf z} p({\bf w}|{\bf s}) d{\bf w}
   = \left(\int {\bf w} p({\bf w}|{\bf s}) d{\bf w}\right)^\top {\bf z}
   = {\bf w}\text{MSE}^\top {\bf z}
$$
where
$$
{\bf w}\text{MSE}
   = \int {\bf w} p({\bf w}|{\bf s}) d{\bf w}
   = {\sigma\varepsilon^{-2}} {\bf V}_{\bf w} {\bf Z}^\top {\bf s}
$$
Therefore, in the Gaussian case, the weighted integration of prediction function is equivalent to apply a unique model, with weights ${\bf w}_\text{MSE}$.
Exercise 3:
Plot the minimum MSE predictions of $s$ for inputs $x$ in the interval [-1, 3].
Can you write Python code for it?

# <SOL>
x = np.array([-1.0, 3.0])
s_pred = w_MSE * x

plt.figure()
plt.plot(X_tr, S_tr,'b.')
plt.plot(x, s_pred)
plt.show()
# </SOL>


I am working on the file "R5.Bayesian_Regression/.ipynb_checkpoints/Bayesian_regression-checkpoint.ipynb".
The first task is:
Not only do we obtain a better predictive model, but we also have confidence intervals (error bars) for the predictions.
4 Maximum evidence model selection
We have already addressed with Bayesian Inference the following two issues:

For a given degree, how do we choose the weights?

Should we focus on just one model, or can we use several models at once?

However, we still needed some assumptions: a parametric model (i.e., polynomial function and a priori degree selection) and several parameters needed to be adjusted.
Though we can recur to cross-validation, Bayesian inference opens the door to other strategies.

We could argue that rather than keeping single selections of these parameters, we could use simultaneously several sets of parameters (and/or several parametric forms), and average them in a probabilistic way ... (like we did with the models)

We will follow a simpler strategy, selecting just the most likely set of parameters according to an ML criterion

4.1 Model evidence
The evidence of a model is defined as
$$L = p({\bf s}~|~{\cal M})$$
where ${\cal M}$ denotes the model itself and any free parameters it may have. For instance, for the polynomial model we have assumed so far, ${\cal M}$ would represent the degree of the polynomia, the variance of the additive noise, and the a priori covariance matrix of the weights
Applying the Theorem of Total probability, we can compute the evidence of the model as
$$L = \int p({\bf s}~|~{\bf f},{\cal M}) p({\bf f}~|~{\cal M}) d{\bf f} $$
For the linear model $f({\bf x}) = {\bf w}^\top{\bf z}$, the evidence can be computed as
$$L = \int p({\bf s}~|~{\bf w},{\cal M}) p({\bf w}~|~{\cal M}) d{\bf w} $$
It is important to notice that these probability density functions are exactly the ones we computed on the previous section. We are just making explicit that they depend on a particular model and the selection of its parameters. Therefore:

$p({\bf s}~|~{\bf w},{\cal M})$ is the likelihood of ${\bf w}$

$p({\bf w}~|~{\cal M})$ is the a...
Can you write Python code for it?

from math import pi

n_points = 15
frec = 3
std_n = 0.2
max_degree = 12

#Prior distribution parameters
sigma_eps = 0.2
mean_w = np.zeros((degree+1,))
sigma_p = 0.5

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)

#Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z=np.asmatrix(Z)

#Evaluate the posterior evidence

logE = []
for deg in range(max_degree):
    Z_iter = Z[:,:deg+1]
    logE_iter = -((deg+1)*np.log(2*pi)/2) \
                -np.log(np.linalg.det((sigma_p**2)*Z_iter.dot(Z_iter.T) + (sigma_eps**2)*np.eye(n_points)))/2 \
                -S_tr.T.dot(np.linalg.inv((sigma_p**2)*Z_iter.dot(Z_iter.T) + (sigma_eps**2)*np.eye(n_points))).dot(S_tr)/2
    logE.append(logE_iter[0,0])

plt.plot(np.array(range(max_degree))+1,logE)
plt.xlabel('Polynomia degree')
plt.ylabel('log evidence')
plt.show()


I am working on the file "R5.Bayesian_Regression/.ipynb_checkpoints/Bayesian_regression-checkpoint.ipynb".
The first task is:
Posterior distribution of the target

Since $f^ = f({\bf x}^) = {\bf w}^\top{\bf z}$, $f^*$ is also a Gaussian variable whose posterior mean and variance can be calculated as follows:

$$\mathbb{E}{{{\bf z}^}^\top {\bf w}~|~{\bf s}, {\bf z}^} =
       {{\bf z}^}^\top \mathbb{E}{{\bf w}|{\bf s}} =
       {\sigma_\varepsilon^{-2}} {{\bf z}^}^\top {\bf V}_{\bf w} {\bf Z}^\top {\bf s}$$
$$\text{Cov}\left[{{\bf z}^}^\top {\bf w}~|~{\bf s}, {\bf z}^\right] =
       {{\bf z}^}^\top \text{Cov}\left[{\bf w}~|~{\bf s}\right] {{\bf z}^} =
       {{\bf z}^}^\top {\bf V}_{\bf w} {{\bf z}^}$$

Therefore, $f^~|~{\bf s}, {\bf x}^ \sim {\cal N}\left({\sigma_\varepsilon^{-2}} {{\bf z}^}^\top  {\pmb\Sigma}_{\bf w} {\bf Z}^\top {\bf s}, {{\bf z}^}^\top {\pmb \Sigma}_{\bf w} {{\bf z}^*} \right)$

Finally, for $s^ = f^ + \varepsilon^$, the posterior distribution is  $s^~|~{\bf s}, {\bf z}^ \sim {\cal N}\left({\sigma_\varepsilon^{-2}} {{\bf z}^}^\top  {\pmb\Sigma}{\bf w} {\bf Z}^\top {\bf s}, {{\bf z}^}^\top {\pmb \Sigma}_{\bf w} {{\bf z}^} + \sigma\varepsilon^2\right)$
Can you write Python code for it?

n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
degree = 12
nplots = 6

#Prior distribution parameters
sigma_eps = 0.1
mean_w = np.zeros((degree+1,))
sigma_p = .5 * np.eye(degree+1)

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-1,3,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

#Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z=np.asmatrix(Z)

#Compute posterior distribution parameters
Sigma_w = np.linalg.inv(np.dot(Z.T,Z)/(sigma_eps**2) + np.linalg.inv(sigma_p))
posterior_mean = Sigma_w.dot(Z.T).dot(S_tr)/(sigma_eps**2)
posterior_mean = np.array(posterior_mean).flatten()

#Plot the posterior mean
#Note that polyval assumes the first element of weight vector is the coefficient of
#the highest degree term. Thus, we need to reverse w_iter
S_grid_iter = np.polyval(posterior_mean[::-1],X_grid)
ax.plot(X_grid,S_grid_iter,'g-',label='Predictive mean, BI')

#Plot confidence intervals for the Bayesian Inference
std_x = []
for el in X_grid:
    x_ast = np.array([el**k for k in range(degree+1)])
    std_x.append(np.sqrt(x_ast.dot(Sigma_w).dot(x_ast)[0,0]))
std_x = np.array(std_x)
plt.fill_between(X_grid, S_grid_iter-std_x, S_grid_iter+std_x,
    alpha=0.2, edgecolor='#1B2ACC', facecolor='#089FFF',
    linewidth=4, linestyle='dashdot', antialiased=True)

#We plot also the least square solution
w_LS = np.polyfit(X_tr.flatten(), S_tr.flatten(), degree)
S_grid_iter = np.polyval(w_LS,X_grid)
ax.plot(X_grid,S_grid_iter,'m-',label='LS regression')

ax.set_xlim(-1,3)
ax.set_ylim(S_tr[0]-2,S_tr[-1]+2)
ax.legend(loc='best')


I am working on the file "R5.Bayesian_Regression/.ipynb_checkpoints/Bayesian_regression-checkpoint.ipynb".
The first task is:
To do so, compute the posterior weight distribution using the first $k$ samples in the complete dataset, for $k = 1,2,4,8,\ldots 128$. Draw all these posteriors along with the prior distribution in the same plot.
Can you write Python code for it?

# No. of points to analyze
n_points = [1, 2, 4, 8, 16, 32, 64]

# Prepare plots
w_grid = np.linspace(2.7, 3.4, 5000)   # Sample the w axis
plt.figure()

# Compute the prior distribution over the grid points in w_grid
# p = <FILL IN>
p = 1.0/(sigma_p*np.sqrt(2*np.pi)) * np.exp(-(w_grid**2)/(2*sigma_p**2))
plt.plot(w_grid, p,'g-')

for k in n_points:

    # Select the first k samples
    Zk = X_tr[0:k, :]
    Sk = S_tr[0:k]

    # Parameters of the posterior distribution
    # 1. Compute the posterior variance.
    #    (Make sure that the resulting variable, Var_w, is a 1x1 numpy array.)
    # Var_w = <FILL IN>
    Var_w = np.linalg.inv(np.dot(Zk.T, Zk)/(sigma_eps**2) + np.linalg.inv(Var_p))

    # 2. Compute the posterior mean.
    #    (Make sure that the resulting variable, w_MSE, is a scalar)
    # w_MSE = <FILL IN>
    w_MSE = (Var_w.dot(Zk.T).dot(Sk)/(sigma_eps**2)).flatten()

    # Compute the posterior distribution over the grid points in w_grid
    sigma_w = np.sqrt(Var_w.flatten())     # First we take a scalar standard deviation
    # p = <FILL IN>
    p = 1.0/(sigma_w*np.sqrt(2*np.pi)) * np.exp(-((w_grid-w_MSE)**2)/(2*sigma_w**2))

    plt.plot(w_grid, p,'g-')
    plt.fill_between(w_grid, 0, p, alpha=0.8, edgecolor='#1B2ACC', facecolor='#089FFF',
        linewidth=1, antialiased=True)
    plt.xlim(w_grid[0], w_grid[-1])
    plt.ylim(0, np.max(p))
    plt.xlabel('$w$')
    plt.ylabel('$p(w|s)$')

    display.clear_output(wait=True)
    display.display(plt.gcf())
    time.sleep(2.0)

# Remove the temporary plots and fix the last one
display.clear_output(wait=True)
plt.show()