I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
4. Regularization and MAP estimation.
An alternative to the ML estimation of the weights in logistic regression is Maximum A Posteriori estimation. Modelling the logistic regression weights as a random variable with prior distribution $p_{\bf W}({\bf w})$, the MAP estimate is defined as
$$
\hat{\bf w}_{\text{MAP}} = \arg\max p({\bf w}|{\mathcal S})
$$
The posterior density $p({\bf w}|{\mathcal S})$ is related to the likelihood function and the prior density of the weights, $p_{\bf W}({\bf w})$ through the Bayes rule
$$
p({\bf w}|{\mathcal S}) =
    \frac{p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
          p_{\bf W}({\bf w})}
         {p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)}\right)}
$$
In general, the denominator in this expression cannot be computed analytically. However, it is not required for MAP estimation because it does not depend on ${\bf w}$. Thus, we can compute the MAP estimate as
\begin{align}
\hat{\bf w}{\text{MAP}}
  &= \arg\max{\bf w}
         p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
         p_{\bf W}({\bf w}) \
  &= \arg\max_{\bf w} \left{
         \log\left(p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right) \right)
         + \log\left( p_{\bf W}({\bf w})\right)
         \right}  \
  &= \arg\min_{\bf w} \left{L({\bf w}) - \log\left( p_{\bf W}({\bf w})\right)
         \right}
\end{align}
where $L(·)$ is the negative log-likelihood function.
Two common choices for the prior are

Gaussian: ${\bf W}$ is a zero-mean Gaussian random variable with variance matrix $v{\bf I}$. Thus:
$$
p_{\bf W}({\bf w}) = \frac{1}{(2\pi v)^{N/2}} \exp\left(-\frac{1}{2v}\|{\bf w}\|^2\right)
$$
and the MAP estimate is
\begin{align}
\hat{\bf w}{\text{MAP}}
  &= \arg\min{\bf w} \left{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2
         \right}
\end{align}
where $C = 2v$. Noting that
$$\nabla_{\bf w}\left{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2\right}
= - {\bf Z}^\intercal \left({\bf...
Can you write Python code for it?

def logregFit2(Z_tr, Y_tr, rho, n_it, C=1e4):

    # Compute Z's
    r = 2.0/C
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):
        p_tr = logistic(np.dot(Z_tr, w))

        sk = np.multiply(p_tr, 1-p_tr)
        S = np.diag(np.ravel(sk.T))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p_tr)) - np.dot((1-Y_tr).T, np.log(1-p_tr))

        # Update weights
        invH = np.linalg.inv(r*np.identity(n_dim) + np.dot(Z_tr.T, np.dot(S, Z_tr)))

        w += rho*np.dot(invH, np.dot(Z_tr.T, Y_tr - p_tr))

    return w, nll_tr

# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
C = 1000
g = 4

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(X_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(X_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit2(Z_tr, Y_tr2, rho, n_it, C)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
In order to apply the gradient descent rule, we need to define two methods:
 - A fit method, that receives the training data and returns the model weights and the value of the negative log-likelihood during all iterations.
 - A predict method, that receives the model weight and a set of inputs, and returns the posterior class probabilities for that input, as well as their corresponding class predictions.
Can you write Python code for it?

def logregFit(Z_tr, Y_tr, rho, n_it):

    # Data dimension
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):

        # Compute posterior probabilities for weight w
        p1_tr = logistic(np.dot(Z_tr, w))
        p0_tr = logistic(-np.dot(Z_tr, w))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p1_tr)) - np.dot((1-Y_tr).T, np.log(p0_tr))

        # Update weights
        w += rho*np.dot(Z_tr.T, Y_tr - p1_tr)

    return w, nll_tr

def logregPredict(Z, w):

    # Compute posterior probability of class 1 for weights w.
    p = logistic(np.dot(Z, w))

    # Class
    D = [int(round(pn)) for pn in p]

    return p, D


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
Now, we can normalize training and test data. Observe in the code that the same transformation should be applied to training and test data. This is the reason why normalization with the test data is done using the means and the variances computed with the training set.
Can you write Python code for it?

# Normalize data
Xn_tr, mx, sx = normalize(X_tr)
Xn_tst, mx, sx = normalize(X_tst, mx, sx)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
Now, we select two classes and two attributes.
Can you write Python code for it?

# Select attributes
i = 0 # Try 0,1,2,3
j = 1 # Try 0,1,2,3 with j!=i

# Select two classes
c0 = 'Iris-versicolor'
c1 = 'Iris-virginica'

# Select two coordinates
ind = [i, j]

# Take training test
X_tr = np.array([[xTrain_all[n][i] for i in ind] for n in range(nTrain_all)
                  if cTrain_all[n]==c0 or cTrain_all[n]==c1])
C_tr = [cTrain_all[n] for n in range(nTrain_all)
          if cTrain_all[n]==c0 or cTrain_all[n]==c1]
Y_tr = np.array([int(c==c1) for c in C_tr])
n_tr = len(X_tr)

# Take test set
X_tst = np.array([[xTest_all[n][i] for i in ind] for n in range(nTest_all)
                 if cTest_all[n]==c0 or cTest_all[n]==c1])
C_tst = [cTest_all[n] for n in range(nTest_all)
         if cTest_all[n]==c0 or cTest_all[n]==c1]
Y_tst = np.array([int(c==c1) for c in C_tst])
n_tst = len(X_tst)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
The following figure generates a plot of the normalized training data.
Can you write Python code for it?

# Separate components of x into different arrays (just for the plots)
x0c0 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==0]
x1c0 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==0]
x0c1 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==1]
x1c1 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==1]

# Scatterplot.
labels = {'Iris-setosa': 'Setosa',
          'Iris-versicolor': 'Versicolor',
          'Iris-virginica': 'Virginica'}
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
Visualizing the posterior map we can se that the polynomial transformations produces nonlinear decision boundaries.
Can you write Python code for it?

# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 8, 4  # Set figure size
for i in [1, 2]:
    ax = plt.subplot(1,2,i)
    ax.plot(x0c0, x1c0,'r.', label=labels[c0])
    ax.plot(x0c1, x1c1,'g+', label=labels[c1])
    ax.set_xlabel('$x_' + str(ind[0]) + '$')
    ax.set_ylabel('$x_' + str(ind[1]) + '$')
    ax.axis('equal')
    if i==1:
        ax.contourf(xx, yy, pp, cmap=plt.cm.copper)
    else:
        ax.legend(loc='best')
        ax.contourf(xx, yy, np.round(pp), cmap=plt.cm.copper)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/RegresionLogistica-checkpoint.ipynb".
The first task is:
We can test the behavior of the gradient descent method by fitting a logistic regression model with ${\bf z}({\bf x}) = (1, {\bf x}^\intercal)^\intercal$.
Can you write Python code for it?

# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 200   # Number of iterations

# Compute Z's
Z_tr = np.c_[np.ones(n_tr), Xn_tr]
Z_tst = np.c_[np.ones(n_tst), Xn_tst]
n_dim = Z_tr.shape[1]

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The optimal weights are:"
print w
print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
1. Introduction
  [Source:  sklearn documentation ]
Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.
The advantages of support vector machines are:

Effective in high dimensional spaces.
Still effective in cases where number of dimensions is greater than the number of samples.
Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
Versatile: different Kernel functions can be specified for the decision function.

The disadvantages of support vector machines include:

SVMs do not directly provide probability estimates.

2. Motivating Support Vector Machines
  [Source:  notebook  by  Jake Vanderplas]
Support Vector Machines (SVMs) are a kind of discriminative classifiers: that is, they draw a boundary between clusters of data without making any explicit assumption about the probability model generating underlying the data generation process.
Let's show a quick example of support vector classification. First we need to create a dataset:
Can you write Python code for it?

X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
A discriminative classifier attempts to draw a line between the two sets of data. Immediately we see a problem: such a line is ill-posed! For example, we could come up with several possibilities which perfectly discriminate between the classes in this example:
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
If we plot this along with our data, we can see the effect of it:
Can you write Python code for it?

def plot_3D(elev=30, azim=30):
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, cmap='spring')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('r')

interact(plot_3D, elev=[-90, 90], azip=(-180, 180));


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
Let's use IPython's interact functionality to explore how the distribution of points affects the support vectors and the discriminative fit. (This is only available in IPython 2.0+, and will not work in a static view)
Can you write Python code for it?

def plot_svm(N=10):
    X, y = make_blobs(n_samples=200, centers=2,
                      random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    clf = svm.SVC(kernel='linear')
    clf.fit(X, y)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
    plt.xlim(-1, 4)
    plt.ylim(-1, 6)
    plot_svc_decision_function(clf, plt.gca())
    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
                s=200, facecolors='none')

interact(plot_svm, N=[10, 200], kernel='linear')


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
Notice here that if we want to maximize this width, the middle fit is clearly the best. This is the intuition of the SVM, which optimize a linear discriminant model in conjunction with a margin representing the perpendicular distance between the datasets.
3. Linear SVM
  [Source: adapted from  Wikipedia ]
In order to present the SVM in a formal way, consider a training dataset $\mathcal{S} = \left{ (\mathbf{x}^{(k)}, y^{(k)}) \mid \mathbf{x}^{(k)}\in \Re^N,\, y^{(k)} \in {-1,1}, k=1,\ldots, K\right}$, where the binary symmetric label $y^{(k)}\in {-1,1}$ indicates the class to which the point $\mathbf{x}^{(k)}$ belongs. Each $\mathbf{x}^{(k)}$ is a $p$-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y^{(k)}=1$ from those having $y^{(k)}=-1$.
Any hyperplane can be written as the set of points $\mathbf{x}$ satisfying
$$
\mathbf{w}^\intercal \mathbf{x} - b=0,
$$
where ${\mathbf{w}}$ denotes the (not necessarily normalized) normal vector to the hyperplane. The parameter $\tfrac{b}{\|\mathbf{w}\|}$ determines the offset of the hyperplane from the origin along the normal vector ${\mathbf{w}}$.
If the training data are linearly separable, we can select two parallel hyperplanes in a way that they separate the data and there are no points between them, and then try to maximize their distance. The region bounded by them is called "the margin". These hyperplanes can be described by the equations
$$\mathbf{w}^\intercal \mathbf{x} - b=1$$
and
$$\mathbf{w}^\intercal \mathbf{x} - b=-1.$$
Note that the two equations above can represent any two parallel hyperplanes in $\Re^N$. Essentially, the direction of vector $\mathbf{w}$ determines the orientation of the hyperplanes, whereas parameter $b$ and the norm of $\mathbf{w}$ can be used to select their exact location.
To compute the distance between the hyperplanes, we can obtain the projection of vector ${\mathbf...
Can you write Python code for it?

clf = svm.SVC(kernel='linear')
clf.fit(X, y)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
Notice the unique thing about SVM is that only the support vectors matter: that is, if you moved any of the other points without letting them cross the decision boundaries, they would have no effect on the classification results!
5. Non-separable problems.
  [Source: adapted from  Wikipedia ]
In 1995, Corinna Cortes and Vladimir N. Vapnik suggested a modified maximum margin idea that allows for mislabeled examples. If there exists no hyperplane that can split the positive and negative samples, the Soft Margin
 method will choose a hyperplane that splits the examples as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples. The method introduces non-negative slack variables, $\xi^{(k)}$, which measure the degree of misclassification of the data $\mathbf{x}^{(k)}$
$$y^{(k)}(\mathbf{w}^\intercal\mathbf{x}^{(k)} - b) \ge 1 - \xi^{(k)} \quad 1 \le k \le K.
$$
The objective function is then increased by a function which penalizes non-zero $\xi^{(k)}$, and the optimization becomes a trade off between a large margin and a small error penalty. If the penalty function is linear, the optimization problem becomes:
$$(\mathbf{w}^,\mathbf{\xi}^, b^*) = \arg\min_{\mathbf{w},\mathbf{\xi}, b } \left{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{k=1}^K \xi^{(k)} \right} \
\text{subject to: } \quad y^{(k)}(\mathbf{w}\cdot\mathbf{x_i} - b) \ge 1 - \xi^{(k)}, \quad \xi^{(k)} \ge 0, \quad  k=1,\ldots, K.
$$
This constraint along with the objective of minimizing $\|\mathbf{w}\|$ can be solved using KKT multipliers as done above. One then has to solve the following problem:
$$
\arg\min_{\mathbf{w}, \mathbf{\xi}, b } \max_{\boldsymbol{\alpha}, \boldsymbol{\beta} }
\left{ \frac{1}{2}\|\mathbf{w}\|^2
+ C \sum_{k=1}^K \xi^{(k)}
- \sum_{k=1}^K {\alpha^{(k)}[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) -1 + \xi^{(k)}]}
- \sum_{k=1}^K \beta^{(k)} \xi^{(k)} \right }\
\text{subject to: } \quad...
Can you write Python code for it?

X, y = make_circles(100, factor=.1, noise=.1)

clf = svm.SVC(kernel='linear').fit(X, y)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
These are three very different separators which perfectly discriminate between these samples. Depending on which you choose, a new data point will be classified almost entirely differently! How can we improve on this?
Support Vector Machines (SVM) select the boundary decision maximizing the margin. The margin of a classifier is defined as twice the maximum signed distance between the decision boundary and the training data. By signed we mean that the distance to misclassified samples is counted negatively. Thus, if the classification problem is "separable" (i.d. if there exist a decision boundary with zero errors in the training set), the SVM will choose the zero-error decision boundary that is "as far as possible" from the training data.
In summary, what a SVM does is to not only draw a line, but consider the "sample free" region about the line. Here's an example of what it might look like:
Can you write Python code for it?

xfit = np.linspace(-1, 3.5)
plt.scatter(X[:,0], X[:,1], c=y, s=50, cmap='copper')

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-k')
    plt.fill_between(xfit, yfit-d, yfit+d, edgecolor='none',
                     color='#AAAAAA', alpha=0.4)

plt.xlim(-1, 3.5)
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SVM-checkpoint.ipynb".
The first task is:
We can see that with this additional dimension, the data becomes trivially linearly separable!
This is a relatively simple kernel; SVM has a more sophisticated version of this kernel built-in to the process. This is accomplished by using the Gaussian kernel (kernel='rbf'), short for radial basis function:
Can you write Python code for it?

clf = svm.SVC(kernel='rbf')
clf.fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SupportVectorMachines-checkpoint.ipynb".
The first task is:
1. Introduction
Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.
The advantages of support vector machines are:

Effective in high dimensional spaces.
Still effective in cases where number of dimensions is greater than the number of samples.
Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.

The disadvantages of support vector machines include:

If the number of features is much greater than the number of samples, the method is likely to give poor performances.
SVMs do not directly provide probability estimates, these are calculated using an expensive five-fold cross-validation (see Scores and probabilities, below).
The support vector machines in scikit-learn support both dense (numpy.ndarray and convertible to that by numpy.asarray) and sparse (any scipy.sparse) sample vectors as input. However, to use an SVM to make predictions for sparse data, it must have been fit on such data. For optimal performance, use C-ordered numpy.ndarray (dense) or scipy.sparse.csr_matrix (sparse) with dtype=float64.

SVM implementations in Scikit Learn
SVC, NuSVC and LinearSVC are classes capable of performing multi-class classification on a dataset.
SVC and NuSVC are similar methods, but accept slightly different sets of parameters and have different mathematical formulations (see section Mathematical formulation). On the other hand, LinearSVC is another implementation of Support Vector Classification for the case of a linear kernel. Note that LinearSVC does not accept keyword kernel, as this is assumed to be linear. It also lacks some of the members of SVC and NuSVC, like support_.
As other classifiers, SVC, NuSVC and LinearSVC take as input two arrays: an array X of size [n_samples, n_features] holding the training...
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SupportVectorMachines-checkpoint.ipynb".
The first task is:
3.2.1 Example: Iris Dataset.
As an illustration, consider the  Iris dataset , taken from the  UCI Machine Learning repository. This data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant (setosa, versicolor or virginica). Each instance contains 4 measurements of given flowers: sepal length, sepal width, petal length and petal width, all in centimeters.
We will try to fit the logistic regression model to discriminate between two classes using only two attributes.
First, we load the dataset and split them in training and test subsets.
Can you write Python code for it?

# Adapted from a notebook by Jason Brownlee
def loadDataset(filename, split):
    xTrain = []
    cTrain = []
    xTest = []
    cTest = []

    with open(filename, 'rb') as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
    for i in range(len(dataset)-1):
        for y in range(4):
            dataset[i][y] = float(dataset[i][y])
        item = dataset[i]
        if random.random() < split:
            xTrain.append(item[0:4])
            cTrain.append(item[4])
        else:
            xTest.append(item[0:4])
            cTest.append(item[4])
    return xTrain, cTrain, xTest, cTest

with open('iris.data', 'rb') as csvfile:
    lines = csv.reader(csvfile)

xTrain_all, cTrain_all, xTest_all, cTest_all = loadDataset('iris.data', 0.66)
nTrain_all = len(xTrain_all)
nTest_all = len(xTest_all)
print 'Train: ' + str(nTrain_all)
print 'Test: ' + str(nTest_all)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SupportVectorMachines-checkpoint.ipynb".
The first task is:
3.2.3. Free parameters
Under certaing conditions, the gradient descent method can be shown to converge assymtotically (i.e. as the number of iterations goes to infinity) to the ML estimate of the logistic model. However, in practice, the final estimate of the weights ${\bf w}$ depend on several factors:

Number of iterations
Initialization
Learning step

Exercise: Visualize the variability of gradient descent caused by initializations. To do so, fix the number of iterations to 200 and the learning step, and execute the gradient descent 100 times, storing the training error rate of each execution. Plot the histogram of the error rate values.
Note that you can do this exercise with a loop over the 100 executions, including the code in the previous code slide inside the loop, with some proper modifications. To plot a histogram of the values in array p with nbins, you can use plt.hist(p, n)
3.2.3.1. Learning step
The learning step, $\rho$, is a free parameter of the algorithm. Its choice is critical for the convergence of the algorithm. too large values of $\rho$ make the algorithm diverge. For too small values, the convergence is too slown and more iterations are required for a good convergence.
Exercise 3: Observe the evolution of the negative log-likelihood with the number of iterations, for different values of $\rho$. It is easy to check that, for $\rho$ large enough, the gradient descent method does not converge. Can you estimate (through manual observation) and approximate value of $\rho$ stating a boundary between convergence and non-convergence?
Exercise 4: In this exercise we explore the influence of the learning step more sistematically. Use the code in the previouse exercises to compute, for every value of $\rho$, the average error rate over 100 executions. Plot the average error rate vs. $\rho$.
Note that you should explore the values of $\rho$ in a logarithmic scale. For instance, you can take $\rho = 1, 1/10, 1/100, 1/1000, \ldots$
In practice, the selection of $\rho$ may be a matter of trial an...
Can you write Python code for it?

# Create a regtangular grid.
x_min, x_max = Xn_tr[:, 0].min(), Xn_tr[:, 0].max()
y_min, y_max = Xn_tr[:, 1].min(), Xn_tr[:, 1].max()
dx = x_max - x_min
dy = y_max - y_min
h = dy /400
xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                     np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
X_grid = np.array([xx.ravel(), yy.ravel()]).T

# Compute Z's
Z_grid = np.c_[np.ones(X_grid.shape[0]), X_grid]

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)

# Put the result into a color plot
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
pp = pp.reshape(xx.shape)
plt.contourf(xx, yy, pp, cmap=plt.cm.copper)


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SupportVectorMachines-checkpoint.ipynb".
The first task is:
3.2.5. Polynomial Logistic Regression
The error rates of the logitic regression model can be potentially reduce by using polynomial transformations.
To compute the polinomial transformation up to a given degree, we can use the PolynomialFeatures method in sklearn.preprocessing.
Can you write Python code for it?

# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
g = 5 # Degree of polynomial

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The optimal weights are:"
print w
print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])


I am working on the file "C4.Classification_SVM/.ipynb_checkpoints/SupportVectorMachines-checkpoint.ipynb".
The first task is:
Now, we can normalize training and test data. Observe in the code that the same transformation should be applied to training and test data. This is the reason why normalization with the test data is done using the means and the variances computed with the training set.
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_professor.ipynb".
The first task is:
A discriminative classifier attempts to draw a line between the two sets of data. Immediately we see an inconvenience: such problem is ill-posed! For example, we could come up with several possibilities which perfectly discriminate between the classes in this example:
Can you write Python code for it?

xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')

for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-k')

plt.xlim(-1, 3.5);
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()


I am working on the file "C4.Classification_SVM/svm_professor.ipynb".
The first task is:
Clearly, no linear discrimination will ever separate these data.
One way we can adjust this is to apply a kernel, which is some functional transformation of the input data.
For example, one simple model we could use is a radial basis function
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_professor.ipynb".
The first task is:
Let's use IPython's interact functionality to explore how the distribution of points affects the support vectors and the discriminative fit.
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_professor.ipynb".
The first task is:
Notice here that if we want to maximize this width, the middle fit is clearly the best. This is the intuition of the SVM, which optimizes a linear discriminant model in conjunction with a margin representing the perpendicular distance between the datasets.
3. Linear SVM
  [Source: adapted from Wikipedia]
In order to present the SVM in a formal way, consider a training dataset $\mathcal{D} = \left{ (\mathbf{x}_k, y_k) \mid \mathbf{x}_k\in \Re^M,\, y_k \in {-1,1}, k=0,\ldots, {K-1}\right}$, where the binary symmetric label $y_k\in {-1,1}$ indicates the class to which the point $\mathbf{x}_k$ belongs. Each $\mathbf{x}_k$ is an $M$-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y_k=1$ from those having $y_k=-1$.
Any hyperplane can be written as the set of points $\mathbf{x}$ satisfying
$$
\mathbf{w}^\intercal \mathbf{x} - b=0,
$$
where ${\mathbf{w}}$ denotes the (not necessarily normalized) normal vector to the hyperplane. The parameter $\tfrac{b}{\|\mathbf{w}\|}$ determines the offset of the hyperplane from the origin along the normal vector ${\mathbf{w}}$.
If the training data are linearly separable, we can select two parallel hyperplanes in a way that they separate the data and there are no points between them, and then try to maximize their distance. The region bounded by them is called "the margin". These hyperplanes can be described by the equations
$$\mathbf{w}^\intercal \mathbf{x} - b=1$$
and
$$\mathbf{w}^\intercal \mathbf{x} - b=-1.$$
Note that the two equations above can represent any two parallel hyperplanes in $\Re^M$. Essentially, the direction of vector $\mathbf{w}$ determines the orientation of the hyperplanes, whereas parameter $b$ and the norm of $\mathbf{w}$ can be used to select their exact location.
To compute the distance between the hyperplanes, we can obtain the projection of vector ${\mathbf x}_1 - {\mathbf x}_2$, where ${\mathbf x}_1$ and ${\mathbf x}_2$ are points from each of the...
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_professor.ipynb".
The first task is:
Notice the unique thing about SVM is that only the support vectors matter: that is, if you moved any of the other points without letting them cross the decision boundaries, they would have no effect on the classification results!
5. Non-separable problems.
  [Source: adapted from Wikipedia]
In 1995, Corinna Cortes and Vladimir N. Vapnik suggested a modified maximum margin idea that allows for mislabeled examples. If there exists no hyperplane that can split the positive and negative samples, the Soft Margin
 method will choose a hyperplane that splits the examples as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples. The method introduces non-negative slack variables, $\xi_k$, which measure the degree of misclassification of the data $\mathbf{x}_k$
$$y_k(\mathbf{w}^\top\mathbf{x}_k - b) \ge 1 - \xi_k \quad k=0,\ldots, K-1.
$$
The objective function is then increased by a function which penalizes non-zero $\xi_k$, and the optimization becomes a trade off between a large margin and a small error penalty. If the penalty function is linear, the optimization problem becomes:
$$(\mathbf{w}^,\mathbf{\xi}^, b^*) = \arg\min_{\mathbf{w},\mathbf{\xi}, b } \left{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{k=0}^{K-1} \xi_k \right} \
\text{subject to: } \quad y_k(\mathbf{w}^\intercal\mathbf{x}_k - b) \ge 1 - \xi_k, \quad \xi_k \ge 0, \quad  k=0,\ldots, K-1.
$$
This constraint along with the objective of minimizing $\|\mathbf{w}\|$ can be solved using KKT multipliers as done above. One then has to solve the following problem:
$$
\arg\min_{\mathbf{w}, \mathbf{\xi}, b } \max_{\boldsymbol{\alpha}, \boldsymbol{\beta} }
\left{ \frac{1}{2}\|\mathbf{w}\|^2
+ C \sum_{k=0}^{K-1} \xi_k
- \sum_{k=0}^{K-1} {\alpha_k\left[y_k(\mathbf{w}^\top \mathbf{x}k - b) -1 + \xi_k\right]}
- \sum{k=0}^{K-1} \beta_k \xi_k \right }\
\text{subject to: } \quad
\alpha_k, \beta_k \ge 0.
$$
A similar analysis to that in the separable case can be applied to show that the...
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_professor.ipynb".
The first task is:
We can see that with this additional dimension, the data becomes trivially linearly separable!
This is a relatively simple kernel; SVM has a more sophisticated version of this kernel built-in to the process. This is accomplished by using the Gaussian kernel (kernel='rbf'), short for radial basis function:
Can you write Python code for it?

clf = svm.SVC(kernel='rbf', C=10)
clf.fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');


I am working on the file "C4.Classification_SVM/svm_student.ipynb".
The first task is:
Clearly, no linear discrimination will ever separate these data.
One way we can adjust this is to apply a kernel, which is some functional transformation of the input data.
For example, one simple model we could use is a radial basis function
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_student.ipynb".
The first task is:
If we plot this along with our data, we can see the effect of it:
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_student.ipynb".
The first task is:
Notice here that if we want to maximize this width, the middle fit is clearly the best. This is the intuition of the SVM, which optimizes a linear discriminant model in conjunction with a margin representing the perpendicular distance between the datasets.
3. Linear SVM
  [Source: adapted from Wikipedia]
In order to present the SVM in a formal way, consider a training dataset $\mathcal{D} = \left{ (\mathbf{x}_k, y_k) \mid \mathbf{x}_k\in \Re^M,\, y_k \in {-1,1}, k=0,\ldots, {K-1}\right}$, where the binary symmetric label $y_k\in {-1,1}$ indicates the class to which the point $\mathbf{x}_k$ belongs. Each $\mathbf{x}_k$ is an $M$-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y_k=1$ from those having $y_k=-1$.
Any hyperplane can be written as the set of points $\mathbf{x}$ satisfying
$$
\mathbf{w}^\intercal \mathbf{x} - b=0,
$$
where ${\mathbf{w}}$ denotes the (not necessarily normalized) normal vector to the hyperplane. The parameter $\tfrac{b}{\|\mathbf{w}\|}$ determines the offset of the hyperplane from the origin along the normal vector ${\mathbf{w}}$.
If the training data are linearly separable, we can select two parallel hyperplanes in a way that they separate the data and there are no points between them, and then try to maximize their distance. The region bounded by them is called "the margin". These hyperplanes can be described by the equations
$$\mathbf{w}^\intercal \mathbf{x} - b=1$$
and
$$\mathbf{w}^\intercal \mathbf{x} - b=-1.$$
Note that the two equations above can represent any two parallel hyperplanes in $\Re^M$. Essentially, the direction of vector $\mathbf{w}$ determines the orientation of the hyperplanes, whereas parameter $b$ and the norm of $\mathbf{w}$ can be used to select their exact location.
To compute the distance between the hyperplanes, we can obtain the projection of vector ${\mathbf x}_1 - {\mathbf x}_2$, where ${\mathbf x}_1$ and ${\mathbf x}_2$ are points from each of the...
Can you write Python code for it?


I am working on the file "C4.Classification_SVM/svm_student.ipynb".
The first task is:
Notice that the dashed lines touch a couple of the points: these points are the support vectors. In scikit-learn, these are stored in the support_vectors_ attribute of the classifier:
Can you write Python code for it?

plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, marker='s');
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)


I am working on the file "C4.Classification_SVM/svm_student.ipynb".
The first task is:
To better visualize what's happening here, let's create a quick convenience function that will plot SVM decision boundaries for us:
Can you write Python code for it?

def plot_svc_decision_function(clf, ax=None):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    x = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)
    y = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)
    Y, X = np.meshgrid(y, x)
    P = np.zeros_like(X)
    for i, xi in enumerate(x):
        for j, yj in enumerate(y):
            P[i, j] = clf.decision_function(np.array([xi, yj]).reshape(1,-1))
    # plot the margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
<font color='olive'> 3.4.4. Multi Layer networks using nn.Sequential </font>

PyTorch simplifies considerably the implementation of neural network training, since we do not need to implement derivatives ourselves

We can also make a simpler implementation of multilayer networks using nn.Sequential function

It returns directly a network with the requested topology, including parameters and forward evaluation method
Can you write Python code for it?

my_net = nn.Sequential(
    nn.Linear(X_train_torch.size()[1], 200),
    nn.ReLU(),
    nn.Linear(200,50),
    nn.ReLU(),
    nn.Linear(50,20),
    nn.ReLU(),
    nn.Linear(20,y_train_torch.size()[1])
)

opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 200

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    print(f'Current epoch: {epoch + 1}  \r', end="")

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()

print('Validation accuracy with this net:', acc_val[-1])


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Dogs vs Cats data set

Dataset is taken from  Kaggle
25000 pictures of dogs and cats
Binary problem
Input data are images, we create also a flattened version
Original images are RGB, and arbitrary size
Preprocessed images are $64 \times 64$ and gray scale
Can you write Python code for it?

# Preprocessing of original Dogs and Cats Pictures
# Adapted from https://medium.com/@mrgarg.rajat/kaggle-dogs-vs-cats-challenge-complete-step-by-step-guide-part-1-a347194e55b1
# RGB channels are collapsed in GRAYSCALE
# Images are resampled to 64x64

"""
import os, cv2  # cv2 -- OpenCV

train_dir = './data/DogsCats/train/'
rows = 64
cols = 64
train_images = sorted([train_dir+i for i in os.listdir(train_dir)])

def read_image(file_path):
    image = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)
    return cv2.resize(image, (rows, cols),interpolation=cv2.INTER_CUBIC)

def prep_data(images):
    m = len(images)
    X = np.ndarray((m, rows, cols), dtype=np.uint8)
    y = np.zeros((m,))
    print("X.shape is {}".format(X.shape))

    for i,image_file in enumerate(images) :
        image = read_image(image_file)
        X[i,] = np.squeeze(image.reshape((rows, cols)))
        if 'dog' in image_file.split('/')[-1].lower():
            y[i] = 1
        elif 'cat' in image_file.split('/')[-1].lower():
            y[i] = 0

        if i%5000 == 0 :
            print("Proceed {} of {}".format(i, m))

    return X,y

X_train, y_train = prep_data(train_images)
np.save('./data/DogsCats/X.npy', X_train)
np.save('./data/DogsCats/Y.npy', y_train)
"""

DogsCatsX = np.load('./data/DogsCats/X.npy')
DogsCatsY = np.load('./data/DogsCats/Y.npy')
K = DogsCatsX.shape[0]
img_size = DogsCatsX.shape[1]
DogsCatsX_flatten = DogsCatsX.reshape(K,img_size*img_size)

print('Size of Input Data Matrix:', DogsCatsX.shape)
print('Size of Flattned Input Data Matrix:', DogsCatsX_flatten.shape)
print('Size of label Data Matrix:', DogsCatsY.shape)
selected = [260, 16000]
plt.subplot(1, 2, 1), plt.imshow(DogsCatsX[selected[0]].reshape(img_size, img_size)), plt.axis('off')
plt.subplot(1, 2, 2), plt.imshow(DogsCatsX[selected[1]].reshape(img_size, img_size)), plt.axis('off')
plt.show()
print('Labels corresponding to figures:', DogsCatsY[selected,])


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Exercise

Study the behavior of the algorithm changing the number of iterations and the learning rate

Obtain the confusion matrix, and study which classes are more difficult to classify

Think about the differences between using this 10-class network, vs training 10 binary classifiers, one for each class

As in linear logistic regression note that we covered the following aspects of neural network design, implementation, and training:

Network topology (In this case, a linear network with one layer and $M$ ouptuts)
Activation functions (softmax activation)
Parameter initialization ($\bf W$/$b$)
Obtaining the network prediction using forward computation
Loss function
Parameter gradient calculus using backward computation
Optimization method for parameters update (here, GD)

 2.3. Multi Layer Networks (Deep Networks)
Previous networks are constrained in the sense that they can only implement linear classifiers. In this section we analyze how we can extend them to implement non-linear classification:
* Fixed non-linear transformations of inputs: ${\bf z} = {\bf{f}}({\bf x})$

Parametrize the transformation using additional non-linear layers

When counting layers, we normally ignore the input layer, since there is no computation involved

Intermediate layers are normally referred to as "hidden" layers
Non-linear activations result in an overall non-linear classifier
We can still use Gradient Descent Optimization as long as the network loss derivatives with respect to all parameters exist and are continuous
This is already deep learning. We can have two layers or more, each with different numbers of neurons. But as long as derivatives with respect to parameters can be calculated, the network can be optimized
Finding an appropriate number of layers for a particular problem, as well as the number of neurons per layer, requires exploration
The more data we have for training the network, the...
Can you write Python code for it?

# Define some useful functions
def logistic(t):
    return 1.0 / (1 + np.exp(-t))

def forward(W1, b1, w2, b2, x):
    #Calcula la salida de la red
    h = x.dot(W1.T) + b1
    y_hat = logistic(h.dot(w2) + b2)
    #Provide also hidden units value for backward gradient step
    return h, y_hat

def backward(y, y_hat, h, x, w2):
    #Calcula los gradientes
    w2_grad = h.T.dot(y_hat - y) / len(y)
    b2_grad = np.sum(y_hat - y) / len(y)
    W1_grad = ((w2[np.newaxis,] * ((1 - h)**2) * (y_hat - y)[:,np.newaxis]).T.dot(x)) / len(y)
    b1_grad = ((w2[np.newaxis,] * ((1 - h)**2) * (y_hat - y)[:,np.newaxis]).sum(axis=0)) / len(y)
    return w2_grad, b2_grad, W1_grad, b1_grad

def accuracy(y, y_hat):
    return np.mean(y == (y_hat >= 0.5))

def loss(y, y_hat):
    return - np.sum(y * np.log(y_hat) + (1 - y) * np.log(1 - y_hat)) / len(y)

def evaluate_model(
    X_train, X_val, y_train, y_val, n_h=5, epochs=1000, rho=.005):

    W1 = .01 * np.random.randn(n_h, X_train.shape[1])
    b1 = .01 * np.random.randn(n_h)
    w2 = .01 * np.random.randn(n_h)
    b2 = .01 * np.random.randn(1)

    loss_train = np.zeros(epochs)
    loss_val = np.zeros(epochs)
    acc_train = np.zeros(epochs)
    acc_val = np.zeros(epochs)

    for epoch in np.arange(epochs):
        print(f'Current epoch: {epoch + 1}  \r', end="")

        h, y_hat_train = forward(W1, b1, w2, b2, X_train)
        dum, y_hat_val = forward(W1, b1, w2, b2, X_val)
        w2_grad, b2_grad, W1_grad, b1_grad = backward(y_train, y_hat_train, h, X_train, w2)
        W1 = W1 - rho/10 * W1_grad
        b1 = b1 - rho/10 * b1_grad
        w2 = w2 - rho * w2_grad
        b2 = b2 - rho * b2_grad

        loss_train[epoch] = loss(y_train, y_hat_train)
        loss_val[epoch] = loss(y_val, y_hat_val)
        acc_train[epoch] = accuracy(y_train, y_hat_train)
        acc_val[epoch] = accuracy(y_val, y_hat_val)

    return loss_train, loss_val, acc_train, acc_val


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Exercise

Initialize a tensor x with the upper right $5 \times 10$ submatrix of flattened digits
Compute output vector y applying a function of your choice to x
Compute scalar value z as the sum of all elements in y squared
Check that x.grad calculation is correct using the backward method
Try to run your cell multiple times to see if the calculation is still correct. If not, implement the necessary mnodifications so that you can run the cell multiple times, but the gradient does not change from run to run

Note: The backward method can only be run on scalar variables
 3.4. Feed Forward Network using PyTorch

In this section we will change our code for a neural network to use tensors instead of numpy arrays. We will work with the sign digits datasets.

We will introduce all concepts using a single layer perceptron (softmax regression), and then implement networks with additional hidden layers

 3.4.1. Using Automatic differentiation

We start by loading the data, and converting to tensors.

As a first step, we refactor our code to use tensor operations

We do not need to pay too much attention to particular details regarding tensor operations, since these will not be necessary when moving to higher PyTorch abstraction levels

We do not need to implement gradient calculation. PyTorch will take care of that
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

dataset = 'digits'

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(digitsX_flatten)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, digitsY, test_size=0.2, random_state=42, shuffle=True)

#Convert to Torch tensors
X_train_torch = torch.from_numpy(X_train)
X_val_torch = torch.from_numpy(X_val)
y_train_torch = torch.from_numpy(y_train)
y_val_torch = torch.from_numpy(y_val)

# Define some useful functions
def softmax(t):
    """Compute softmax values for each sets of scores in t"""
    return t.exp() / t.exp().sum(-1).unsqueeze(-1)

def model(w,b,x):
    #Calcula la salida de la red
    return softmax(x.mm(w) + b)

def accuracy(y, y_hat):
    return (y.argmax(axis=-1) == y_hat.argmax(axis=-1)).float().mean()

def nll(y, y_hat):
    return -(y * y_hat.log()).mean()


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Results in Dogs vs Cats dataset ($epochs = 1000$ and $\rho = 0.05$)
Can you write Python code for it?

dataset = 'DogsCats'

X_train, X_val, y_train, y_val = get_dataset(dataset)
loss_train, loss_val, acc_train, acc_val = evaluate_model(
    X_train, X_val, y_train, y_val, n_h=5, epochs=1000, rho=0.05)

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
 3. Implementing Deep Networks with PyTorch

Pytorch is a Python library that provides different levels of abstraction for implementing deep neural networks

The main features of PyTorch are:

Definition of numpy-like n-dimensional tensors. They can be stored in / moved to GPU for parallel execution of operations
Automatic calculation of gradients, making backward gradient calculation transparent to the user
Definition of common loss functions, NN layers of different types, optimization methods, data loaders, etc, simplifying NN implementation and training
Provides different levels of abstraction, thus a good balance between flexibility and simplicity

This notebook provides just a basic review of the main concepts necessary to train NNs with PyTorch taking materials from:

Learning PyTorch with Examples, by Justin Johnson
What is torch.nn really?, by Jeremy Howard
Pytorch Tutorial for Deep Learning Lovers, by Kaggle user kanncaa1

 3.1. Installation and PyTorch introduction

PyTorch can be installed with or without GPU support

If you have an Anaconda installation, you can install from the command line, using the instructions of the project website

PyTorch is also preinstalled in Google Collab with free GPU access

Follow RunTime -> Change runtime type, and select GPU for HW acceleration

Please, refer to Pytorch getting started tutorial for a quick introduction regarding tensor definition, GPU vs CPU storage of tensors, operations, and bridge to Numpy

 3.2. Torch tensors (very) general overview

We can create tensors with different construction methods provided by the library, either to create new tensors from scratch or from a Numpy array
Can you write Python code for it?

import torch

x = torch.rand((100,200))
digitsX_flatten_tensor = torch.from_numpy(digitsX_flatten)

print(x.type())
print(digitsX_flatten_tensor.size())


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Adding underscore performs operations "in place", e.g., x.add_(y)

If a GPU is available, tensors can be moved to and from the GPU device

Operations on tensors stored in a GPU will be carried out using GPU resources and will typically be highly parallelized
Can you write Python code for it?

if torch.cuda.is_available():
    device = torch.device('cuda')
    x = x.to(device)
    y = x.add(x)
    y = y.to('cpu')
else:
    print('No GPU card is available')


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
It is important to deactivate gradient updates after the network has been evaluated on training data, and gradients of the loss function have been computed
Can you write Python code for it?

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Note network optimization is carried out outside torch.no_grad() but network evaluation (other than forward output calculation for the training patterns) still need to deactivate gradient updates
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_professor.ipynb".
The first task is:
Syntaxis is a bit different because input variables are tensors, not arrays

This time we did not need to implement the backward function
Can you write Python code for it?

# Parameter initialization
W = .1 * torch.randn(X_train_torch.size()[1], y_train_torch.size()[1])
W.requires_grad_()
b = torch.zeros(y_train_torch.size()[1], requires_grad=True)

epochs = 500
rho = .5

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

# Network training
for epoch in range(epochs):

    print(f'Current epoch: {epoch + 1}  \r', end="")

    #Compute network output and cross-entropy loss
    pred = model(W,b,X_train_torch)
    loss = nll(y_train_torch, pred)

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = model(W, b, X_val_torch)
        loss_val[epoch] = nll(y_val_torch, pred_val).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        W -= rho * W.grad
        b -= rho * b.grad
        #Reset gradients
        W.grad.zero_()
        b.grad.zero_()


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
<font color='olive'> 3.4.2. Using torch nn module </font>

PyTorch nn module provides many attributes and methods that make the implementation and training of Neural Networks simpler

nn.Module and nn.Parameter allow to implement a more concise training loop

nn.Module is a PyTorch class that will be used to encapsulate and design a specific neural network, thus, it is central to the implementation of deep neural nets using PyTorch

nn.Parameter allow the definition of trainable network parameters. In this way, we will simplify the implementation of the training loop.

All parameters defined with nn.Parameter will have requires_grad = True
Can you write Python code for it?

from torch import nn

class my_multiclass_net(nn.Module):
    def __init__(self, nin, nout):
        """This method initializes the network parameters
        Parameters nin and nout stand for the number of input parameters (features in X)
        and output parameters (number of classes)"""
        super().__init__()
        self.W = nn.Parameter(.1 * torch.randn(nin, nout))
        self.b = nn.Parameter(torch.zeros(nout))

    def forward(self, x):
        return softmax(x.mm(self.W) + self.b)

    def softmax(t):
        """Compute softmax values for each sets of scores in t"""
        return t.exp() / t.exp().sum(-1).unsqueeze(-1)

my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])

epochs = 500
rho = .5

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    print(f'Current epoch: {epoch + 1}  \r', end="")

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = nll(y_train_torch, pred)

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = nll(y_val_torch, pred_val).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        for p in my_net.parameters():
            p -= p.grad * rho
        #Reset gradients
        my_net.zero_grad()

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
<font color='olive'> 3.4.4. Multi Layer networks using nn.Sequential </font>

PyTorch simplifies considerably the implementation of neural network training, since we do not need to implement derivatives ourselves

We can also make a simpler implementation of multilayer networks using nn.Sequential function

It returns directly a network with the requested topology, including parameters and forward evaluation method
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
 Exercise
Implement network training with other optimization methods. You can refer to the official documentation and select a couple of methods. You can also try to implement adaptive learning rates using torch.optim.lr_scheduler

Each epoch of the previous implementation of network training was actually implementing Gradient Descent

In SGD only a minibatch of training patterns are used at every iteration

In each epoch we iterate over all training patterns sequentially selecting non-overlapping minibatches

Overall, convergence is usually faster than when using Gradient Descent

Torch provides methods that simplify the implementation of this strategy
Can you write Python code for it?

from torch.utils.data import TensorDataset, DataLoader

train_ds = TensorDataset(X_train_torch, y_train_torch)
train_dl = DataLoader(train_ds, batch_size=64)

from torch import optim

my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])
opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 200

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    print(f'Current epoch: {epoch + 1}  \r', end="")

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Dogs vs Cats data set

Dataset is taken from  Kaggle
25000 pictures of dogs and cats
Binary problem
Input data are images, we create also a flattened version
Original images are RGB, and arbitrary size
Preprocessed images are $64 \times 64$ and gray scale
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Exercise

Study the behavior of the algorithm changing the number of iterations and the learning rate

Obtain the confusion matrix, and study which classes are more difficult to classify

Think about the differences between using this 10-class network, vs training 10 binary classifiers, one for each class

As in linear logistic regression note that we covered the following aspects of neural network design, implementation, and training:

Network topology (In this case, a linear network with one layer and $M$ ouptuts)
Activation functions (softmax activation)
Parameter initialization ($\bf W$/$b$)
Obtaining the network prediction using forward computation
Loss function
Parameter gradient calculus using backward computation
Optimization method for parameters update (here, GD)

 2.3. Multi Layer Networks (Deep Networks)
Previous networks are constrained in the sense that they can only implement linear classifiers. In this section we analyze how we can extend them to implement non-linear classification:
* Fixed non-linear transformations of inputs: ${\bf z} = {\bf{f}}({\bf x})$

Parametrize the transformation using additional non-linear layers

When counting layers, we normally ignore the input layer, since there is no computation involved

Intermediate layers are normally referred to as "hidden" layers
Non-linear activations result in an overall non-linear classifier
We can still use Gradient Descent Optimization as long as the network loss derivatives with respect to all parameters exist and are continuous
This is already deep learning. We can have two layers or more, each with different numbers of neurons. But as long as derivatives with respect to parameters can be calculated, the network can be optimized
Finding an appropriate number of layers for a particular problem, as well as the number of neurons per layer, requires exploration
The more data we have for training the network, the...
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Exercise

Initialize a tensor x with the upper right $5 \times 10$ submatrix of flattened digits
Compute output vector y applying a function of your choice to x
Compute scalar value z as the sum of all elements in y squared
Check that x.grad calculation is correct using the backward method
Try to run your cell multiple times to see if the calculation is still correct. If not, implement the necessary mnodifications so that you can run the cell multiple times, but the gradient does not change from run to run

Note: The backward method can only be run on scalar variables
 3.4. Feed Forward Network using PyTorch

In this section we will change our code for a neural network to use tensors instead of numpy arrays. We will work with the sign digits datasets.

We will introduce all concepts using a single layer perceptron (softmax regression), and then implement networks with additional hidden layers

 3.4.1. Using Automatic differentiation

We start by loading the data, and converting to tensors.

As a first step, we refactor our code to use tensor operations

We do not need to pay too much attention to particular details regarding tensor operations, since these will not be necessary when moving to higher PyTorch abstraction levels

We do not need to implement gradient calculation. PyTorch will take care of that
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Exercises

Train the network using other settings for:

The number of iterations
The learning step
The number of neurons in the hidden layer

You may find divergence issues for some settings

Related to the use of the hyperbolic tangent function in the hidden layer (numerical issues)
This is also why learning step was selected smaller for the hidden layer
Optimized libraries rely on certain modifications to obtain more robust implementations

Try to solve both problems using scikit-learn implementation

You can also explore other activation functions
You can also explore other solvers to speed up convergence
You can also adjust the size of minibatches
Take a look at the early_stopping parameter

 2.4. Multi Layer Networks for Regression

Deep Learning networks can be used to solve regression problems with the following common adjustments

Linear activation for the output unit

Square loss:
$$l(y, \hat y) = (y - \hat y)^2, \qquad \text{where} \qquad y, \hat y \in \Re$$

 2.5. Activation Functions
You can refer to the Dive into Deep Learning book for a more detailed discussion on common actiation functions for the hidden units.
We extract some information about the very important ReLU function

The most popular choice, due to both simplicity of implementation and its good performance on a variety of predictive tasks, is the rectified linear unit (ReLU). ReLU provides a very simple nonlinear transformation. Given an element $x$, the function is defined as the maximum of that element and 0.
When the input is negative, the derivative of the ReLU function is 0, and when the input is positive, the derivative of the ReLU function is 1. When the input takes value precisely equal to 0, we say that the derivative is 0 when the input is...
Can you write Python code for it?

x_array = np.linspace(-6,6,100)
y_array = np.clip(x_array, 0, a_max=None)
plt.plot(x_array, y_array)
plt.title('ReLU activation function')
plt.show()


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Results in Binary Sign Digits Dataset ($epochs = 10000$ and $\rho = 0.001$)
Can you write Python code for it?

dataset = 'digits'
X_train, X_val, y_train, y_val = get_dataset(dataset, forze_binary=True)
loss_train, loss_val, acc_train, acc_val = evaluate_model(
    X_train, X_val, y_train, y_val, n_h=5, epochs=10000, rho=0.001)

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Results in Dogs vs Cats dataset ($epochs = 1000$ and $\rho = 0.05$)
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
 1. Introduction and purpose of this Notebook
 1.1. About Neural Networks

Neural Networks (NN) have become the state of the art for many machine learning problems

Natural Language Processing
Computer Vision
Image Recognition

They are in widespread use for many applications, e.g.,

Language translation (Google Neural Machine Translation System)
Automatic speech recognition (Hey Siri! DNN overview)
Autonomous navigation (Facebook Robot Autonomous 3D Navigation)
Automatic plate recognition

Feed Forward Neural Networks have been around since 1960 but only recently (last 10-12 years) have they met their expectations, and improve other machine learning algorithms

Computation resources are now available at large scale
Cloud Computing (AWS, Azure)
From MultiLayer Perceptrons to Deep Learning
Big Data sets
This has also made possible an intense research effort resulting in
Topologies better suited to particular problems (CNNs, RNNs)
New training strategies providing better generalization

In parallel, Deep Learning Platforms have emerged that make design, implementation, training, and production of DNNs feasible for everyone
 1.2. Scope

To provide just an overview of most important NNs and DNNs concepts
Connecting with already studied methods as starting point
Introduction to PyTorch
Providing links to external sources for further study

 1.3. Outline

Introduction and purpose of this Notebook
Introduction to Neural Networks
Implementing Deep Networks with PyTorch

 1.4. Other resources

We point here to external resources and tutorials that are excellent material for further study of...
Can you write Python code for it?

digitsX = np.load('./data/Sign-language-digits-dataset/X.npy')
digitsY = np.load('./data/Sign-language-digits-dataset/Y.npy')
K = digitsX.shape[0]
img_size = digitsX.shape[1]
digitsX_flatten = digitsX.reshape(K,img_size*img_size)

print('Size of Input Data Matrix:', digitsX.shape)
print('Size of Flattned Input Data Matrix:', digitsX_flatten.shape)
print('Size of label Data Matrix:', digitsY.shape)
selected = [260, 1400]
plt.subplot(1, 2, 1), plt.imshow(digitsX[selected[0]].reshape(img_size, img_size)), plt.axis('off')
plt.subplot(1, 2, 2), plt.imshow(digitsX[selected[1]].reshape(img_size, img_size)), plt.axis('off')
plt.show()
print('Labels corresponding to figures:', digitsY[selected,])


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Adding underscore performs operations "in place", e.g., x.add_(y)

If a GPU is available, tensors can be moved to and from the GPU device

Operations on tensors stored in a GPU will be carried out using GPU resources and will typically be highly parallelized
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Note network optimization is carried out outside torch.no_grad() but network evaluation (other than forward output calculation for the training patterns) still need to deactivate gradient updates
Can you write Python code for it?


I am working on the file "C5.Classification_NN/NeuralNetworks_student.ipynb".
The first task is:
Tensors can be converted back to numpy arrays

Note that in this case, a tensor and its corresponding numpy array will share memory

Operations and slicing use a syntax similar to numpy
Can you write Python code for it?

print('Size of tensor x:', x.size())
print('Tranpose of vector has size', x.t().size()) #Transpose and compute size
print('Extracting upper left matrix of size 3 x 3:', x[:3,:3])
print(x.mm(x.t()).size())  #mm for matrix multiplications
xpx = x.add(x)
xpx2 = torch.add(x,x)
print((xpx!=xpx2).sum())   #Since all are equal, count of different terms is zero


I am working on the file "CAD Widget Example.ipynb".
The first task is:
Display the widget:
Can you write Python code for it?

cadExample

from IPython.html.widgets import interact
@interact(near=(1, 100), far=(100, 400))
def cam(near, far):
    cadExample.camera_near, cadExample.camera_far = near, far


I am working on the file "CAIQ.ipynb".
The first task is:
Al incrementar el número de sustancias, de ensayos, de experimentos, etc... Se comienza a complicar el procesamiento de los datos, por tanto, conviene integrar PyTher con librerías especializadas para el procesamiento de datos como Pandas para obtener resultados más eficientes.
Función = data_temperature(components, temperature, Vapour_Pressure, temp_enter)

Por ejemplo, la función data_temperature() organiza los resultados obtenidos (Vapour_Pressure), diferenciando las temperaturas fuera del intervalo de aplicación de la correlación termodinámica (NaN), de forma estructurada y poder exportarlo en un formato conveniente:

(csv, txt, hoja de cálculo, etc).
Can you write Python code for it?

temp_enter = thermodynamic_correlations.temperature_enter
thermodynamic_correlations.data_temperature(components, temperature, Vapour_Pressure, temp_enter)


I am working on the file "CAIQ.ipynb".
The first task is:
Documentación
PyTher cuenta con documentación abierta (incluyendo errores cometidos, porque open science también es publicar los muchos desaciertos que allanaron el camino de los pocos aciertos) y disponible en Internet con una revisión y construcción continua...
Can you write Python code for it?

from IPython.display import IFrame
documentation_pyther = "http://pyther.readthedocs.io/es/latest/index.html"
IFrame(documentation_pyther, width=950, height=350)


I am working on the file "CAIQ.ipynb".
The first task is:
Especificar multiples sustancias puras y varias condiciones
Es común que se requiera analizar multiples sustancias de forma simultanea con la especificación de varias condiciones de un modelo (por ejemplo valores de temperaturas), de tal forma que se obtendran una gran cantidad de valores (por ejemplo una propiedad termodinámica) que conforman un set de datos (en principio desestructurados), que en general demanda mucho tiempo en revisar y (limpiar datos) para encontrar resultados que no interesan o son invalidos (por ejemplo predicciones fuera del rango de aplicación de un modelos termodinámico)
Veamos un caso típico...
Se obtiene una cierta cantidad de datos desestructurados y sin formato que dificulta diferenciar los resultados "buenos de los malos"
Can you write Python code for it?

thermodynamic_correlations = pt.Thermodynamic_correlations()

components = ["METHANE", "n-TETRACOSANE", "n-PENTACOSANE", "ETHANE", "ISOBUTANE", "PROPANE", "3-METHYLHEPTANE"]
property_thermodynamics = "Vapour_Pressure"
temperature = [180.4, 181.4, 185.3, 210, 800]

Vapour_Pressure = thermodynamic_correlations.property_cal(components, property_thermodynamics, temperature)
print("Vapour Pressure = {0}". format(Vapour_Pressure))


I am working on the file "CAIQ.ipynb".
The first task is:
Pero, los más interesante es el procesamiento de datos, por ejemplo...
Can you write Python code for it?

import numpy as np
import pandas as pd
import pyther as pt
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "CALPHAD2015-Demo.ipynb".
The first task is:
Al-Zn (S. Mey et al., 1993)
Next, we show values from Mey et al.'s 1993 assessment of the Al-Zn system. We have not done any fitting here; we are just showing how arbitrary values for the parameters may be specified and easily used to plot the phase diagram. Note the optional parameters keyword argument specified for each call to Model. This feature is what makes it easy to perform arbitrary variable substitutions in our thermodynamic models.
Can you write Python code for it?

from pycalphad import Model, Database, binplot

db_alzn = Database('alzn_noparams.tdb')

assessed_parameters = {
    'LIQALZN0H': 10465.3,
    'LIQALZN0S': 3.39259,
    'FCCALZN0H': 7297.5,
    'FCCALZN0S': -0.47512,
    'FCCALZN1H': 6612.9,
    'FCCALZN1S': 4.5911,
    'FCCALZN2H': -3097.2,
    'FCCALZN2S': -3.306635,
    'HCPALZN0H': 18821.0,
    'HCPALZN0S': 8.95255,
    'HCPALZN3H': -702.8,
    'HCPALZN3S': 0.0
    }

# Initialize phase models
assessed_models = {
    'LIQUID': Model(db_alzn, ['AL', 'ZN'], 'LIQUID', parameters=assessed_parameters),
    'FCC_A1': Model(db_alzn, ['AL', 'ZN', 'VA'], 'FCC_A1', parameters=assessed_parameters),
    'HCP_A3': Model(db_alzn, ['AL', 'ZN', 'VA'], 'HCP_A3', parameters=assessed_parameters)
    }

fig = plt.figure(figsize=(9,6))
ax = binplot(db_alzn, ['AL', 'ZN', 'VA'] , ['LIQUID', 'FCC_A1', 'HCP_A3'], \
                   'X(ZN)', 300, 1000, ax=plt.gca(), model=assessed_models)
ax = ax.set_title('Al-Zn', fontsize=20)

blank_parameters = {
    'LIQALZN0H': 0.0,
    'LIQALZN0S': 0.0,
    'FCCALZN0H': 0.0,
    'FCCALZN0S': 0.0,
    'FCCALZN1H': 0.0,
    'FCCALZN1S': 0.0,
    'FCCALZN2H': 0.0,
    'FCCALZN2S': 0.0,
    'HCPALZN0H': 0.0,
    'HCPALZN0S': 0.0,
    'HCPALZN3H': 0.0,
    'HCPALZN3S': 0.0
    }

models = {
    'LIQUID': Model(db_alzn, ['AL', 'ZN'], 'LIQUID', parameters=blank_parameters),
    'FCC_A1': Model(db_alzn, ['AL', 'ZN', 'VA'], 'FCC_A1', parameters=blank_parameters),
    'HCP_A3': Model(db_alzn, ['AL', 'ZN', 'VA'], 'HCP_A3', parameters=blank_parameters)
    }

fig = plt.figure(figsize=(9,6))
ax = binplot(db_alzn, ['AL', 'ZN', 'VA'] , ['LIQUID', 'FCC_A1', 'HCP_A3'], \
             'X(ZN)', 300, 1000, ax=plt.gca(), model=models)
ax = ax.set_title('Al-Zn with endmembers only', fontsize=20)


I am working on the file "CALPHAD2015-Demo.ipynb".
The first task is:
Energy Surfaces of Al-Ni (N. Dupin et al., 2001)
Can you write Python code for it?

%matplotlib inline
from pycalphad import energy_surf, Database
from pycalphad.plot.utils import phase_legend
import numpy as np
import matplotlib.pyplot as plt
db_alni = Database('NI_AL_DUPIN_2001.TDB')
my_phases_alni = ['LIQUID', 'FCC_L12', 'BCC_B2', 'AL3NI5', 'AL3NI2', 'AL3NI1']

fig = plt.figure(figsize=(9,6))
ax = fig.gca()
%time nrg_df = energy_surf(db_alni, ['AL', 'NI', 'VA'], my_phases_alni, T=1000.0)

legend_handles, colorlist = phase_legend(my_phases_alni)
group_df = nrg_df.groupby('Phase')
for phase, phase_df in group_df:
    ax = phase_df.plot(kind='scatter', x='X(AL)', y='GM', ax=ax, color=colorlist[phase.upper()], s=1, xlim=(0,1), ylim=(-11e4, -3e4))
ax = ax.legend(handles=legend_handles, loc='center left', bbox_to_anchor=(1, 0.6))
plt.title('Al-Ni Gibbs Energy at 1000 K', fontsize=20)
plt.show()

nrg_df

nrg_df.to_excel('alni_energies.xlsx')


I am working on the file "CB/Appendix2.ipynb".
The first task is:
Սկզբունքը նույնն է նաև մնացած բոլոր հմտությունների որոնման համար
Ակտիվ գործատուներ
Can you write Python code for it?

exp_company=[]
for i in  list_soup:
    m=i.findAll('p',attrs={'align':'center'})
    for j in m:
        exp_company.append(j.text)
list_zangi=[]
for i in exp_company:
    exp=re.findall('Zangi',i)
    if len(exp)==1:
        list_zangi.append(exp)
print 'Zangi-ի կողմից հայատարարված աշխատատեղեր՝ ' + str(len(list_zangi))

list_llc=[]
for i in exp_company:
    exp=re.findall('.*\s*Tech[a-z]*',i)
    if len(exp)==1:
        list_llc.append(exp)
print '"Tech" անունով ընկերությունների կողմից հայատարարված աշխատատեղեր՝ ' + str(len(list_llc))


I am working on the file "CCL/CCLSteps.ipynb".
The first task is:
Algorithm
Can you write Python code for it?

# Pad with zero for border control
def padwithzeros(vector, pad_width, iaxis, kwargs):
     vector[:pad_width[0]] = 0
     vector[-pad_width[1]:] = 0
     return vector

data = saved_pattern(which=3)
data = np.pad(data, 1, padwithzeros)
label = 1
uf = UnionFind([label])

# First Pass
for (i,j), x in np.ndenumerate(data[1:-1, 1:-1]):
    if data[i+1,j+1]==0:
        pass
    elif data[i,j+1]==0 and data[i+1,j]==0:
        label+=1
        data[i+1,j+1]=label
    elif data[i,j+1]!=0 and data[i+1,j]!=0:
        data[i+1,j+1]=data[i,j+1]
        uf.union(data[i,j+1],data[i+1,j])
    elif data[i,j+1]!=0:
        data[i+1,j+1]=data[i,j+1]
    elif data[i+1,j]!=0:
        data[i+1,j+1]=data[i+1,j]

# Second Pass
for (i,j), x in np.ndenumerate(data[1:-1, 1:-1]):
    if (data[i+1,j+1] in uf.component_mapping().keys()) and data[i+1,j+1] != min(uf.component(data[i+1,j+1])):
        #print(data[i+1,j+1] in uf.component_mapping().keys())
        data[i+1,j+1] = min(uf.component(data[i+1,j+1]))

# Visualize
color_table(data[1:-1, 1:-1].astype(int), bkg_colors=['white', 'blue', 'red', 'green'])
plt.show()


I am working on the file "CCL/CCLSteps.ipynb".
The first task is:
Step-wise algorithm
This is intended for presentation and pedagogical purposes.
This version of the two pass algorithm can be runned by the user step by step.
Requires jupyter widgets to be installed
For an interactive notebook ipywidgets need to be installed and properly configured.
If the following code doesn't work or throw an error try to uninstall and reinstall ipywidget by running the following commands.

pip uninstall ipywidgets widgetsnbextension
pip install ipywidgets
jupyter nbextension enable --py --sys-prefix widgetsnbextension
Can you write Python code for it?

from ipywidgets import widgets
from ipywidgets import *
from IPython.display import display

# Pad with zero for border control
def padwithzeros(vector, pad_width, iaxis, kwargs):
     vector[:pad_width[0]] = 0
     vector[-pad_width[1]:] = 0
     return vector

data = saved_pattern(which=3)
data = np.pad(data, 1, padwithzeros)
label = 1
uf = UnionFind([label])

steps = [x for x,k in np.ndenumerate(data[1:-1, 1:-1])]
# First Pass
def first_pass(step, fig, ax):
    global label
    i,j = steps[step]
    if data[i+1,j+1]==0:
        data[i+1,j+1]=-1
        pass
    elif data[i,j+1] in [0,-1] and data[i+1,j] in [0,-1]:
        label+=1
        data[i+1,j+1]=label
    elif data[i,j+1] not in [0,-1] and data[i+1,j] not in [0,-1]:
        data[i+1,j+1]=data[i,j+1]
        uf.union(data[i,j+1],data[i+1,j])
    elif data[i,j+1] not in [0,-1]:
        data[i+1,j+1]=data[i,j+1]
    elif data[i+1,j] not in [0,-1]:
        data[i+1,j+1]=data[i+1,j]
    return color_tables(data[1:-1, 1:-1].astype(int), fig, ax, bkg_colors=['white', 'blue', 'red', 'green'])

# Second Pass
for (i,j), x in np.ndenumerate(data[1:-1, 1:-1]):
    if (data[i+1,j+1] in uf.component_mapping().keys()) and data[i+1,j+1] != min(uf.component(data[i+1,j+1])):
        #print(data[i+1,j+1] in uf.component_mapping().keys())
        data[i+1,j+1] = min(uf.component(data[i+1,j+1]))

# Visualize
#color_table(data[1:-1, 1:-1].astype(int), bkg_colors=['white', 'blue', 'red', 'green'])

%matplotlib notebook
from ipywidgets import *
import numpy as np
import matplotlib.pyplot as plt

button = widgets.Button(description="Next Step")
display(button)
step = 0
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
def next_step(a):
    global step
    if step > 36:
        print('Nothing else to do')
        pass
    else:
        global fig, ax
        #fig = plt.figure()
        #ax = fig.add_subplot(1, 1, 1)
        fig = first_pass(step, fig, ax)
        fig.canvas.draw()
        step += 1
button.on_click(next_step)


I am working on the file "CCL/CCLSteps.ipynb".
The first task is:
Two pass algorithm ###
Import the unionfind data structure, needed for the second pass.
Can you write Python code for it?

from unionfind import UnionFind
data = saved_pattern(which=3)
#data = test_pattern()


I am working on the file "CDO_Tranches.ipynb".
The first task is:
Notice that Emin is multiplied by the total notional of the tranche K!!!!
$AA(K)) = \frac{C(T)-\sqrt({1-\beta^2})\phi(\frac{K}{(1-R)})}{\beta}$
$E[min(L_T,K)] = K[(1-R)\Phi_2^{-1}(C(T),-AA(K),-\beta)+\Phi(AA(K)]$
Can you write Python code for it?

def Emin(K,R,CT,beta):
    mean=np.array([0,0])
    cov = np.array([[1,-beta],[-beta,1]])
    AAK = AA(CT=CT,x=K,R=R)
    upp = np.array([(x,y) for x,y in zip(CT,-AAK)])
    A = np_cdf(mean=mean, diag_sigma=cov, value=upp, name=None)
    A = np.asarray(A, dtype=np.float32).reshape([-1, 1])
    B = norm.cdf(AAK)
    B = np.asarray(B, dtype=np.float32).reshape([-1, 1])
    return K*((1-R)*A + B)

def Q(K1,K2,CT,R,beta):
    QQ1 = Emin(K=K1,R=R,CT=CT,beta=beta)
    QQ2 = Emin(K=K2,R=R,CT=CT,beta=beta)
    QQ =1- (QQ2-QQ1)/(K2-K1)
    return QQ,QQ1,QQ2

K1=0.03
K2 = 0.06
R=0.4
beta=0.3
QQ,QQ1,QQ2= Q(K1,K2,CT,R,beta)

x=CT
fig = plt.figure()
fig.suptitle('CDO Tranche Loss Function', fontsize=14, fontweight='bold')
ax = fig.add_subplot(111)
ax.set_xlabel('CDO Tranche Loss versus Default Barrier Height')
ax.set_ylabel('Accumulated Tranche Loss')
ax.plot(x,QQ1)
ax.plot(x,QQ2)
plt.show()

fig = plt.figure()
fig.suptitle('CDO Tranche Survival Function', fontsize=14, fontweight='bold')
ax = fig.add_subplot(111)
ax.set_xlabel('CDO Tranche Loss versus Default Barrier Height')
ax.set_ylabel('Tranche Survival Function')
ax.plot(x,QQ)
plt.show()


I am working on the file "CDO_Tranches.ipynb".
The first task is:
Question
I am reviewing this Tuesday’s slide. In page 11(lecture 9), the tranche notional is calculated as J(K1,K2)=Q(K1,K2,T)*(K2-K1). I do not quite understand what this notional mean.
According to page 9, tranche notional loss = E[min(LT,K2)]-E[min(LT,K1)]. Is this notional the same one as J(K1,K2)?
If they are the same, then J(K1,K2)=[1-Q(K1,K2,T)] * (K2-K1), by using Q(K1,K2,T) = 1-{E[min(LT,K2)]-E[min(LT,K1)] / (K2-K1)}. Is this right?
Can you write Python code for it?

R=0.4
PD= 0.05 # Probability of default per year
PD = np.array([0.05*x for x in range(1,100) ]) # Probability of default per year
beta = 0.3
CT = norm.ppf(PD)
x = 0.5

def AA(CT,x,R):
    AAA = CT - np.sqrt(1-beta*beta)*norm.ppf(x/(1-R))
    AAA[AAA < -10]=-10
    AAA[AAA > 10] = 10
    return AAA


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
And the gradient of the objective function:
Can you write Python code for it?


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
Let's take a look at the data:
Can you write Python code for it?

pl.subplot(111)
pl.plot(t[np.where(t<2003.)],y[np.where(t<2003.)],ls=':')
pl.ylabel(r"CO$_2$ concentration, ppm", fontsize=20)
pl.xlabel("year", fontsize=20)
pl.title(r"Mauna Loa CO$_2$ data: 1958 to 2003")
pl.axis([1958.,2025.,310.,420.])
pl.grid()
pl.show()


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
Set up the various covariance kernels. These are given by Eq. 5.15 - 5.18 in RW. Note that we've given each of these kernels arguments. These will act as our initial guess at parameter values.
Can you write Python code for it?

k1 = 66.0**2 * kernels.ExpSquaredKernel(67.0**2)
k2 = 2.4**2 * kernels.ExpSquaredKernel(90**2) * kernels.ExpSine2Kernel(2.0 / 1.3**2, 1.0)
k3 = 0.66**2 * kernels.RationalQuadraticKernel(0.78, 1.2**2)
k4 = 0.18**2 * kernels.ExpSquaredKernel(1.6**2) + kernels.WhiteKernel(0.19)


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
So how did we do? We can compare our prediction to the actual measurements from 2005 until now...
Can you write Python code for it?

ax = pl.subplot(111)

# plot the training values
pl.plot(t_to_2003,y_to_2003,ls=':',c='b')

# shade in the area inside a one standard deviation bound:
ax.fill_between(x,mu-std,mu+std,facecolor='lightgrey', lw=0, interpolate=True)

# plot the full dataset
pl.plot(t[np.where(t>2003)],y[np.where(t>2003)],ls='-',c='r')

pl.ylabel(r"CO$_2$ concentration, ppm")
pl.xlabel("year")
pl.title(r"Mauna Loa CO$_2$ data - Comparison")
pl.axis([1958.,2025.,310.,420.])
pl.grid()

# display the figure:
pl.show()


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
Then let's import all the libraries we need...
Can you write Python code for it?

import numpy as np
import george
from george import kernels
import statsmodels.api as sm
import scipy.optimize as op
import pylab as pl


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
Then run the optimization routine:
Can you write Python code for it?

# initial guess at parameters (see above):
p0 = gp.kernel.vector

# if you want to view your initial guess values uncomment the line below
# print p0

# run optimization:
results = op.minimize(nll, p0, jac=grad_nll)

print np.exp(results.x)


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
Then use this kernel to calculate the predicted expectation value and variance of the data over a range of times:
Can you write Python code for it?

# range of times for prediction:
x = np.linspace(max(t_to_2003), 2025, 2000)

# calculate expectation and variance at each point:
mu, cov = gp.predict(y_to_2003, x)
std = np.sqrt(np.diag(cov))


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
Then we can immediately calculate our prediction out to 2025...
Can you write Python code for it?

# range of times for prediction:
x = np.linspace(max(t_to_2003), 2025, 2000)

# calculate expectation and variance at each point:
mu, cov = gp.predict(y_to_2003, x)
std = np.sqrt(np.diag(cov))

ax = pl.subplot(111)

# plot the original values
pl.plot(t_to_2003,y_to_2003,ls=':')

# shade in the area inside a one standard deviation bound:
ax.fill_between(x,mu-std,mu+std,facecolor='lightgrey', lw=0, interpolate=True)

pl.ylabel(r"CO$_2$ concentration, ppm")
pl.xlabel("year")
pl.title(r"Mauna Loa CO$_2$ data - Initial Prediction")
pl.axis([1958.,2025.,310.,420.])
pl.grid()

# display the figure:
pl.show()


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
We can immediately compute the covariance matrix because we've entered some initial guess parameters as the arguments in our kernels.
Can you write Python code for it?


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
We then feed this kernel to the george library.
Can you write Python code for it?

#gp = george.GP(kernel, mean=np.mean(y_to_2005))
gp = george.GP(kernel, mean=0.0)


I am working on the file "CDT-KickOff/LECTURE/GPMCarbonDioxide.ipynb".
The first task is:
You need to compute the GP once before starting the optimization. We've actually already done this above, but it doesn't hurt to run it again.
Can you write Python code for it?


I am working on the file "CDT-KickOff/LECTURE/GPMIntro.ipynb".
The first task is:
If we then take the final realization, which has $\lambda = 5$, and select 5 points from it randomly we can calculate the posterior mean and variance at every point based on those five input data.
The mean and variance are given by Eq. 3.8 & 3.9 in Roberts et al. (2012) or Eq. 2.25 & 2.26 in Rasmussen & Williams.
First let's select our  training data points and our test data points:
Can you write Python code for it?

# set number of training points
nx_training = 5

# randomly select the training points:
tmp = np.random.uniform(low=0.0, high=2000.0, size=nx_training)
tmp = tmp.astype(int)

condition = np.zeros_like(x1)
for i in tmp: condition[i] = 1.0

y_train = y1[np.where(condition==1.0)]
x_train = x1[np.where(condition==1.0)]
y_test = y1[np.where(condition==0.0)]
x_test = x1[np.where(condition==0.0)]


I am working on the file "CDT-KickOff/LECTURE/GPMIntro.ipynb".
The first task is:
Let's plot this up:
Can you write Python code for it?

# mu and sig are currently lists - turn them into numpy arrays:
mu=np.array(mu);sig=np.array(sig)

# make some plots:

# left-hand plot
ax = pl.subplot(121)
pl.scatter(x_train,y_train)  # plot the training points
pl.plot(x1,y1,ls=':')        # plot the original data they were drawn from
pl.title("Input")

# right-hand plot
ax = pl.subplot(122)
pl.plot(x_test,mu,ls='-')     # plot the predicted values
pl.plot(x_test,y_test,ls=':') # plot the original values

# shade in the area inside a one standard deviation bound:
ax.fill_between(x_test,mu-sig,mu+sig,facecolor='lightgrey', lw=0, interpolate=True)
pl.title("Predicted")

pl.scatter(x_train,y_train)  # plot the training points

# display the plot:
pl.show()


I am working on the file "CDT-KickOff/LECTURE/GPMIntro.ipynb".
The first task is:
We can use this kernel to calculate  the value of each element in our covariance matrix:
$\mathbf{K(x,x)} = \left(
\begin{array}{cccc}
k(x_1,x_1) & k(x_1,x_2) & ... & k(x_1,x_n) \
k(x_2,x_1) & k(x_2,x_2) & ... & k(x_2,x_n) \
\vdots & \vdots & \vdots & \vdots \
k(x_n,x_1) & k(x_n,x_2) & ... & k(x_n,x_n)
\end{array}
\right).$
We can then populate a covariance matrix, $K(\mathbf{x},\mathbf{x})$, for our data:
Can you write Python code for it?

def make_K(x, h, lam):

    """
    Make covariance matrix from covariance kernel
    """

    # for a data array of length x, make a covariance matrix x*x:
    K = np.zeros((len(x),len(x)))

    for i in range(0,len(x)):
        for j in range(0,len(x)):

            # calculate value of K for each separation:
            K[i,j] = cov_kernel(x[i],x[j],h,lam)

    return K


I am working on the file "CDT-KickOff/LECTURE/GPMIntro.ipynb".
The first task is:
We then use our training data points to define a covariance matrix:
Can you write Python code for it?

# define the covariance matrix:
K = make_K(x_train,h,lam)

# take the inverse:
iK = np.linalg.inv(K)


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
I'm going to set bounds on the available parameters space, i.e. our prior volume, using the ranges taken from Table 4 of https://arxiv.org/pdf/1706.05459.pdf
Can you write Python code for it?

import scipy.optimize as op

# extract our initial guess at parameters
# from the george kernel and put it in a
# vector:
p0 = gp.kernel.vector

# set prior ranges
# Note that these are in *logarithmic* space
bnds = ((-20.,0.),(2.,8.),(0.,3.),(np.log(0.5),np.log(100.)),(-20.,0))

# run optimization:
results = op.minimize(nll, p0, method='L-BFGS-B', jac=grad_nll, bounds=bnds)

# print the value of the optimised parameters:
print np.exp(results.x)


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
Import some libraries:
Can you write Python code for it?


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
In the paper there are two suggested kernels for modelling the covariance of the Kepler data (Eqs. 55 & 56). In the paper the authors fit Eq 56 - here we are going to fit Eq. 55.
We can do this using a combination of kernels from the george library.
Exponential Squared Kernel:
$$
k_1(x_i,x_j)=h_1 \exp(−\frac{(x_i-x_j)^2}{2 \sigma^2})
$$
Exp-Sine-Squared Kernel:
$$
k_2(x_i,x_j)=h_2 \exp \left( \Gamma \sin^2 \left[ \frac{\pi}{P}|x_i - x_j|\right] \right)
$$
Our combined kernel is therefore:
$$
k(x_i,x_j)=h \exp(−\frac{(x_i-x_j)^2}{2 \sigma^2})\exp \left( \Gamma \sin^2 \left[ \frac{\pi}{P}|x_i - x_j|\right] \right)
$$
where $h = h_1 h_2$.
However, following https://arxiv.org/pdf/1706.05459.pdf, we are also going to add a white noise kernel in:
$$
k_3(x_i,x_j)=c \delta_{ij}
$$
So our final kernel will be:
$$
k = (k_1k_2) + k_3
$$
Can you write Python code for it?

# h =1.0; sigma = 1.0; Gamma = 2.0/1.0^2; P = 3.8
k1 = 1.0**2 * kernels.ExpSquaredKernel(1.0**2) \
                        * kernels.ExpSine2Kernel(2.0 / 1.0**2, 1.0)

k2 = kernels.WhiteKernel(0.01)

kernel = k1+k2

# first we feed our combined kernel to the George library:
gp = george.GP(kernel, mean=0.0)

# then we compute the covariance matrix:
gp.compute(time)

t = np.arange(np.min(time),np.max(time),0.1)

# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)
std = np.sqrt(np.diag(cov))

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
Make a plot like the one in Figure 7:
Can you write Python code for it?

pl.subplot(111)
pl.scatter(time,value,s=0.2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
Read the Kepler data from the file:
Can you write Python code for it?

time=[];value=[]
while True:
    line = datafile.readline()
    if not line: break

    items=line.split()
    if (items[0][0]!='|'):
        time.append(float(items[1]))
        value.append(float(items[2]))

time=np.array(time)
value=np.array(value)


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
Specify the datafile containing Kepler data for the object KIC 1430163:
Can you write Python code for it?

filename="KIC1430163.tbl"
datafile = open(filename,'r')


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurve.ipynb".
The first task is:
The key parameter here is the period, which is the fourth number along. We expect this to be about 3.9 and... it is already coming out with that kind of value!
From the paper:
This star has a published rotation period of 3.88 ± 0.58 days, measured using traditional periodogram and autocorrelation function approaches applied to Kepler data from Quarters 0–16 (Mathur et al. 2014), covering about four years.
Let's now pass these optimised parameters to george and recompute our prediction:
Can you write Python code for it?

# pass the parameters to the george kernel:
gp.kernel[:] = results.x

t = np.arange(np.min(time),np.max(time),0.1)

# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)
std = np.sqrt(np.diag(cov))

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:

Can you write Python code for it?

import emcee

# we need to define three functions:
# a log likelihood, a log prior & a log posterior.


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
First we need to define a log(likelihood). We'll use the log(likelihood) implemented in the george library, which implements:
$$
\ln L = -\frac{1}{2}(y - \mu)^{\rm T} C^{-1}(y - \mu) - \frac{1}{2}\ln |C\,| + \frac{N}{2}\ln 2\pi
$$
(see Eq. 5 in https://arxiv.org/pdf/1706.05459.pdf).
Can you write Python code for it?

# set the loglikelihood:
def lnlike(p, x, y):

    ln_a = p[0]
    ln_b = p[1]
    ln_c = -1.*p[2]  # we pass ln(c) from the CustomKernel, ln(c) = -ln(L)
    ln_p = np.log(p[3]) # we're sampling linearly from P so we need to log it

    p0 = np.array([ln_a,ln_b,ln_c,ln_p])

    # update kernel parameters:
    gp.set_parameter_vector(p0)

    # calculate the likelihood:
    ll = gp.log_likelihood(y)

    # return
    return ll if np.isfinite(ll) else 1e25