I am working on the file "Python/5 Linear Regression/Vectorization-with-Numpy.ipynb".
The first task is:
We notice that <tt>NumPy</tt> based computation is much faster than the list based computation.  Similar observations can be made when a function is applied to all elements of an array.  For big arrays, using the vectorized functions offered by <tt>NumPy</tt> is usually much faster than applying the function to all elements of a list.
Can you write Python code for it?

import math

%%timeit
for i, x in enumerate(la):
    lb[i] = math.sin(x)

%%timeit
b = np.sin(a)


I am working on the file "Python/55_VH_Resample.ipynb".
The first task is:
Because we are resampling the moving image using the physical location of the fixed image without any transformation (identity), most of the resulting volume is empty. The image content appears in slice 57 and below.
Can you write Python code for it?

myshow(out)

# combine the two images using a checkerboard pattern:
# because the moving image is single channel with a high dynamic range we rescale it to [0,255] and repeat
# the channel 3 times
vis = sitk.CheckerBoard(
    fixed,
    sitk.Compose([sitk.Cast(sitk.RescaleIntensity(out), sitk.sitkUInt8)] * 3),
    checkerPattern=[15, 10, 1],
)

myshow(vis)


I am working on the file "Python/55_VH_Resample.ipynb".
The first task is:
Load the RGB cryosectioning of the Visible Human Male dataset. The data is about 1GB so this may take several seconds, or a bit longer if this is the first time the data is downloaded from the midas repository.
Can you write Python code for it?

fixed = sitk.ReadImage(fdata("vm_head_rgb.mha"))

moving = sitk.ReadImage(fdata("vm_head_mri.mha"))

print(fixed.GetSize())
print(fixed.GetOrigin())
print(fixed.GetSpacing())
print(fixed.GetDirection())

print(moving.GetSize())
print(moving.GetOrigin())
print(moving.GetSpacing())
print(moving.GetDirection())

import sys

resample = sitk.ResampleImageFilter()
resample.SetReferenceImage(fixed)
resample.SetInterpolator(sitk.sitkBSpline)
resample.AddCommand(
    sitk.sitkProgressEvent,
    lambda: print(f"\rProgress: {100*resample.GetProgress():03.1f}%...", end=""),
)
resample.AddCommand(sitk.sitkProgressEvent, lambda: sys.stdout.flush())
out = resample.Execute(moving)


I am working on the file "Python/55_VH_Resample.ipynb".
The first task is:
Write the image to the Output directory: (1) original as a single image volume and (2) as a series of smaller JPEG images which can be constructed into an animated GIF.
Can you write Python code for it?

import os

sitk.WriteImage(vis, os.path.join(OUTPUT_DIR, "example_resample_vis.mha"))

temp = sitk.Shrink(vis, [3, 3, 2])
sitk.WriteImage(
    temp, [os.path.join(OUTPUT_DIR, f"r{i:03d}.jpg") for i in range(temp.GetSize()[2])]
)


I am working on the file "Python/6 Classification/Digit-Recognition-SVM.ipynb".
The first task is:
Digit Recognition using a Support Vector Machine
Can you write Python code for it?

import gzip
import pickle
import numpy       as np
import sklearn.svm as svm


I am working on the file "Python/6 Classification/Digit-Recognition-SVM.ipynb".
The first task is:
The function $\texttt{load_data}()$ returns a pair of the form
$$ (\texttt{training_data}, \texttt{test_data}) $$
where

 $\texttt{training_data}$ is a list containing 60,000 pairs $(\textbf{x}, y)$ s.t. $\textbf{x}$ is a 784-dimensional `numpy.ndarray` containing the input image and $y$ is the digit that is supposed to be shown in the image $\textbf{x}$.

 $\texttt{test_data}$ is a list containing 10,000 pairs $(\textbf{x}, y)$.  In each case,
     $\textbf{x}$ is a 784-dimensional `numpy.ndarry` containing the input image,
     and $y$ is the corresponding digit value.

Can you write Python code for it?

def load_data():
    with gzip.open('../mnist.pkl.gz', 'rb') as f:
        train, validate, test = pickle.load(f, encoding="latin1")
    X_train = np.array([np.reshape(x, (784, )) for x in train[0]])
    X_test  = np.array([np.reshape(x, (784, )) for x in test [0]])
    Y_train = np.array(train[1])
    Y_test  = np.array(test [1])
    return (X_train, X_test, Y_train, Y_test)

X_train, X_test, Y_train, Y_test = load_data()


I am working on the file "Python/6 Classification/Digit-Recognition-SVM.ipynb".
The first task is:
We define a support vector machine with Gaussian kernel.
Can you write Python code for it?

M = svm.SVC(kernel='rbf', gamma=0.05, C=5)

%%time
M.fit(X_train, Y_train)

M.score(X_train, Y_train)

M.score(X_test, Y_test)


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
As a first attempt to solve the problem we will use the last character of a name as its feature.  We have to compute the conditional probability for every possible letter that occurs as the last letter of a name.  The general formula to compute the conditional probability of a feature $f$ given a class $C$ is the following:
$$ P(f\;|\;C) \approx
 \frac{\mathtt{card}\bigl({t \in T \;|\; \mathtt{class}(t) = C \wedge \mathtt{has}(t, f) }\bigr)}{
       \mathtt{card}\bigl({t \in T \;|\; \mathtt{class}(t) = C }\bigr)}
$$
The function conditional_prop takes a character $c$ and a gender $g$ and determines the conditional probability of seeing $c$ as a last character of a name that has the gender $g$.
Can you write Python code for it?

def conditional_prop(c, g):
    if g == 'f':
        return len([n for n in FemaleNames if n[-1] == c]) / len(FemaleNames)
    else:
        return len([n for n in MaleNames   if n[-1] == c]) / len(MaleNames)


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
Gender Estimation for First Names
This notebook gives a simple example for a naive Bayes classifier.  We try to predict the gender of a first name.  In order to train our classifier, we need a training set of names that are marked as being either male.  We happen to have two text files, names-female.txt and names-male.txt containing female and male first names.  We start by defining the function read_names.  This function reads a file of strings and returns a list of all the names given in the file.  Care is taken that the newline character at the end of each line is discarded.
Can you write Python code for it?

def read_names(file_name):
    Result = []
    with open(file_name, 'r') as file:
        for name in file:
            Result.append(name[:-1]) # discard newline
    return Result

FemaleNames = read_names('names-female.txt')
MaleNames   = read_names('names-male.txt')


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
Let us check the overall accuracy of our classifier with respect to the training set.
Can you write Python code for it?

total   = 0
correct = 0
for n in FemaleNames:
    if classify(n) == 'f':
        correct += 1
    total += 1
for n in MaleNames:
    if classify(n) == 'm':
        correct += 1
    total += 1
accuracy = correct / total
accuracy


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
Let us compute the prior probabilities $P(\texttt{Female})$ and $P(\texttt{Male})$ for the classes $\texttt{Female}$ and $\texttt{Male}$.  In the lecture it was shown that the prior probability of a class $C$ in a training set $T$ is given as:
$$ P(C) \approx
   \frac{\mathtt{card}\bigl({t \in T \;|\; \mathtt{class}(t) = C }\bigr)}{\mathtt{card}(T)}
$$
Therefore, these probabilities are computed as follows.
Can you write Python code for it?

pFemale = len(FemaleNames) / (len(FemaleNames) + len(MaleNames))
pMale   = len(MaleNames)   / (len(FemaleNames) + len(MaleNames))
pFemale


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
Next, we define a dictionary Conditional_Probability.  For every character $c$ and every gender $g \in {\texttt{'f'}, \texttt{'m'}}$, the entry $\texttt{Conditional_Probability}[(c,g)]$ is the conditional probability of observing the last character $c$ if the gender is known to be $g$.
Can you write Python code for it?

Conditional_Probability = {}
for c in 'abcdefghijklmnopqrstuvwxyz':
    for g in ['f', 'm']:
        Conditional_Probability[c, g] = conditional_prop(c, g)


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
Now that have both the prior probabilities $P(\texttt{'f'})$ and $P(\texttt{'m'})$ and also all the conditional probabilities $P(c|g)$, we are ready to implement our naive Bayes classifier.
Can you write Python code for it?

def classify(name):
    last   = name[-1]
    female = Conditional_Probability[(last, 'f')] * pFemale
    male   = Conditional_Probability[(last, 'm')] * pMale
    if female >= male:
        return 'f'
    else:
        return 'm'


I am working on the file "Python/6 Classification/Gender-Estimation.ipynb".
The first task is:
We test our classifier with two common names.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Gradient-Ascent.ipynb".
The first task is:
Clearly, this function has a maximum somewhere between 0.7 and 0.8.  Let us use gradient ascent to find it.  In order to do so, we have to provide the derivative of this function.  We have
$$ \frac{\mathrm{d}f}{\mathrm{d}x} = \cos(x) - x. $$
Can you write Python code for it?


I am working on the file "Python/6 Classification/Gradient-Ascent.ipynb".
The first task is:
Let us plot the derivative together with the function.
Can you write Python code for it?

X2 = np.arange(0.4, 1.1, 0.01)
Ys = fs(X2)
plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.title('lambda x: sin(x) - x**2/2 and its derivative')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('x')
plt.ylabel('y')
plt.xticks(np.arange(-0.5, 1.81, step=0.1))
plt.yticks(np.arange(-0.6, 0.61, step=0.1))
plt.plot(X, Y, color='b')
plt.plot(X2, Ys, color='r')

x_max, _, cnt = findMaximum(f, fs, 0.0, 1e-15)

x_max, cnt


I am working on the file "Python/6 Classification/Gradient-Ascent.ipynb".
The first task is:
Let us plot this function.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn           as sns

X = np.arange(-0.5, 1.8, 0.01)
Y = f(X)
plt.figure(figsize=(15, 10))
sns.set(style='whitegrid')
plt.title('lambda x: sin(x) - x**2/2')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('x')
plt.ylabel('y')
plt.xticks(np.arange(-0.5, 1.81, step=0.1))
plt.plot(X, Y, color='b')


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
Classifying Flowers using a Support Vector Machine
I have adapted this notebook from https://scikit-learn.org/stable/auto_examples/svm/plot_iris.html.
In this notebook we will use different support vector machine to classify flowers from the iris dataset
using only the petal length and the petal width.
We will show how to plot the decision boundary of a support vector machine.
We start with the usual imports for reading the data and plotting.
Can you write Python code for it?

import numpy             as np
import pandas            as pd
import matplotlib.pyplot as plt
import seaborn           as sns


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
Given a model $M$ and a two-dimensional design matrix $X$, this function plots the data from $X$ and a decision boundary.
Can you write Python code for it?

def plot_data_and_boundary(X, M, title):
    X0, X1 = X[:, 0], X[:, 1]
    XX, YY = make_meshgrid(X0, X1)
    plt.figure(figsize=(15, 10))
    sns.set(style='darkgrid')
    plot_contour(M, XX, YY)
    plt.scatter(X0, X1, c=y, edgecolors='k')
    plt.xlim(XX.min(), XX.max())
    plt.ylim(YY.min(), YY.max())
    plt.xlabel('Petal length')
    plt.ylabel('Petal width')
    plt.xticks()
    plt.yticks()
    plt.title(title)


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
In order to use support vector machines we have to import the module svm from SciKit-Learn.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
Let us load the data and store it in a data frame.
Can you write Python code for it?

IrisDF = pd.read_csv('iris.csv')
IrisDF.head()


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
Since we want to have a two dimensional model, we will only use the petal length and the petal width.
Can you write Python code for it?

X = np.array(IrisDF[['petal_length', 'petal_width']])
y = np.array([name_to_number(name) for name in IrisDF['species']])

X.shape, y.shape


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
The class SVC uses a One-vs-One classifier, i.e. in this example it builds three support vector machines:
  -  The first SVM separates setosa from versicolor.
  -  The second SVM separates setosa from virginica.
  -  The third  SVM separates virginica from versicolor.
Let's have a Gaussian kernel function next.
Can you write Python code for it?

M = svm.SVC(kernel='rbf', gamma=1.5, C=10000)
M.fit(X, y)
M.score(X, y)

plot_data_and_boundary(X, M, 'Support Vector Machine with Gaussian Kernel')

M = svm.SVC(kernel='poly', degree=3, gamma='auto', C=10000)
M.fit(X, y)
M.score(X, y)

plot_data_and_boundary(X, M, 'Support Vector Machine with Polynomial Kernel of Degree 3')


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
The function $\texttt{name_to_number}(name)$ converts the name of the flower into a number.
Can you write Python code for it?

def name_to_number(name):
    if name == 'setosa':
        return 0
    if name == 'versicolor':
        return 1
    return 2


I am working on the file "Python/6 Classification/Iris-Classification-with-SVM.ipynb".
The first task is:
The function $\texttt{plot_contour}(M, X1, X2)$ is used to plot the decision boundaries for the classifier $M$.
$X$ and $Y$ are meshgrids for the x and y axis.
Can you write Python code for it?

def plot_contour(M, X, Y):
    Z = M.predict(np.c_[X.ravel(), Y.ravel()])
    Z = Z.reshape(X.shape)
    plt.contour(X, Y, Z)


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
In order to plot the <bf style="color:red;">losers</bf> in a different color, we split the data frame ExamDF into winners and losers.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
In order to plot the decision boundary, we have to extract the coefficients of the new model.
Can you write Python code for it?

ϑ0     = M.intercept_[0]
ϑ1, ϑ2 = M.coef_[0]

plt.figure(figsize=(10, 6))
sns.set(style='darkgrid')
plt.title('Pass/Fail vs. Hours of Study and IQ')
plt.axvline(x=0.0, c='k')
plt.axhline(y=85.0, c='k')
plt.xlabel('Hours of Study')
plt.ylabel('IQ')
plt.xticks(np.arange(0.0, 6.0, step=0.25))
plt.yticks(np.arange(85, 126, step=2.0))
plt.scatter(X_pass[:,0], X_pass[:,1], color='blue') # plot student who passed in blue
plt.scatter(X_fail[:,0], X_fail[:,1], color='red')  # plot the losers red
H = np.arange(1.35, 3.5, 0.05)
P = -(ϑ0 + ϑ1 * H)/ϑ2
plt.plot(H, P, color='green')


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
It seems that three students are misclassified, but one of them is only misclassified by a small margin
as the data points are very close to the green borderline.
Can you write Python code for it?

errors   = np.sum(np.abs(Y - M.predict(X)))
accuracy = (len(Y) - errors) / len(Y)
accuracy


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
Logistic Regression with SciKit-Learn
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
Next, we train the model with the data we have.  The parameter C is a regularization parameter.  Setting it to a high values avoids any regularization.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
Now only one student is misclassified and hence the accuracy goes up to nearly $95\%$.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
Now we are ready to plot the data.
Can you write Python code for it?

plt.figure(figsize=(14, 10))
sns.set(style='darkgrid')
plt.title('Pass/Fail vs. Hours of Study and IQ')
plt.axvline(x=0.0, c='k')
plt.axhline(y=85.0, c='k')
plt.xlabel('Hours of Study')
plt.ylabel('IQ')
plt.xticks(np.arange(0.0, 6.0, step=0.25))
plt.yticks(np.arange(85, 126, step=2.0))
plt.scatter(X_pass[:,0], X_pass[:,1], color='b') # plot student who passed in blue
plt.scatter(X_fail[:,0], X_fail[:,1], color='r') # plot the losers red


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
The data is read with the library pandas.
Can you write Python code for it?

ExamDF = pd.read_csv('exam-iq.csv')
ExamDF


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
The data we want to investigate is stored in the file 'exam-iq.csv'.  The first column of this file is an integer from the set ${0,1}$.  The number is $0$ if the corresponding student has failed the exam and is $1$ otherwise.  The second column is a floating point number that lists the number of hours that the student has studied.  The third column is an integer value specifying the IQ of the student.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
The fact that we did not predict the correct result for the student who learned for $2.25$ hours and who has an IQ of $120$ is due to the outlier that we have in our data set.  Fortunately, the outlier happens to be the last student in the dataset.  We will remove this student when training our model.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
There is one student who failed although he has an IQ of 125 and he did study for $3.5$ hours.  Maybe he was still drunk when he had to write the exam.  The student with an IQ of 104 who did pass while only studying for $2$ hours might just have gotten lucky.
We import the module linear_model from SciKit-Learn.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
We built a logistic regression model.   The first parameter C is the so called regularization parameter.  If we set it to a high value, then we do not regularize.  The second parameter tol is the tolerance.  It specifies when gradient descent should stop.
Can you write Python code for it?

M = lm.LogisticRegression(C=100_000, tol=1e-8)


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
We can select both winners and losers by indexing with Boolean arrays.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
We extract the data from the data frame and convert it into NumPy arrays.  Furthermore, the pass/fail results are converted into floats.
Can you write Python code for it?

X = np.array(ExamDF[['Hours','IQ']])
Y = np.array(ExamDF['Pass'], dtype=float)


I am working on the file "Python/6 Classification/Logistic-Regression-with-SciKit-Learn.ipynb".
The first task is:
We extract the parameters that we have learned.
Can you write Python code for it?

ϑ0     = M.intercept_[0]
ϑ1, ϑ2 = M.coef_[0]


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
As we are using NumPy to compute $\exp(t)$, we can feed this function with a numpy array to compute the sigmoid function for every element of the array:
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
As we have no real clue about the weights, we set them to $0$ initially.
Can you write Python code for it?

import gradient_ascent

start   = np.zeros((2,))
eps     = 10 ** -8
f       = lambda w: ll(X, y, w)
gradF   = lambda w: gradLL(X, y, w)
w, _, _ = gradient_ascent.findMaximum(f, gradF, start, eps, True)
beta    = w[1]
gamma   = w[0]
print(f'model: P(pass|hours) = S({beta} + {gamma} * hours)')


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
Let us compute
$$ \ln\bigl(S(-1000)\bigr) = \ln\Bigl(\frac{1}{1 + \exp(1000)}\Bigr) = - \ln\bigl(1 + \exp(1000)\bigr) \approx -1000. $$
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
Let us plot this function together with the data.
Can you write Python code for it?

plt.figure(figsize=(15, 9))
sns.set_style('whitegrid')
plt.title('Pass/Fail vs. Hours of Study')
H = np.arange(0.0, 6.0, 0.05)
P = sigmoid(beta + gamma * H)
sns.lineplot(x=H, y=P, color='r')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('Hours of Study')
plt.ylabel('Probability of Passing the Exam')
plt.xticks(np.arange(0.0, 6.0, step=0.5))
plt.yticks(np.arange(-0.0, 1.01, step=0.1))
plt.scatter(x, (y + 1) / 2, color='b')
plt.savefig('exam-probability.pdf')


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
Logistic Regression
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
Next, we define the natural logarithm of the sigmoid function.  If we implement this as log(sigmoid(t)) we will get overflow issues for negative values of $t$ such that $t < -1000$ as the expression np.exp(-t) will overflow.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
The file exam.csv contains fictional data about an exam. The first column contains the number 0 if the student has failed the exam and 1 otherwise.  The second column contains the number of hours the student has studied for the given exam.
Can you write Python code for it?

!cat exam.csv || type exam.csv

with open('exam.csv') as file:
    reader = csv.reader(file, delimiter=',')
    count  = 0  # line count
    Pass   = []
    Hours  = []
    for row in reader:
        if count != 0:  # skip header
            Pass .append(float(row[0]))
            Hours.append(float(row[1]))
        count += 1


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
The function $\mathtt{gradLL}(\mathbf{x}, \mathbf{y}, \mathbf{w})$ computes the gradient of
the log-likelihood according to the formula
$$ \frac{\partial\quad}{\partial\, w_j}\ell\ell(\mathbf{X},\mathbf{y};\mathbf{w}) =
   \sum\limits_{i=1}^N y_i \cdot x_{i,j} \cdot  S(-y_i \cdot \mathbf{x}_i \cdot \mathbf{w}).
$$
The different components of this gradient are combined into a vector.
The arguments are the same as the arguments to the function $\ell\ell$ that computes the log-likelihood, i.e.
* $\textbf{X}$ is the feature matrix, $\textbf{X}[i]$ is the $i$-th feature vector.
* $\textbf{y}$ is the output vector, $\textbf{y}[i] \in {-1,+1}$ for all $i$.
* $\textbf{w}$ is the weight vector.
Can you write Python code for it?

def gradLL(X, y, w):
    Gradient = []
    for j in range(len(X[0])):
        L = [y[i] * X[i][j] * sigmoid(-y[i] * (X[i] @ w)) for i in range(len(X))]
        Gradient.append(sum(L))
    return np.array(Gradient)


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
The number of students is stored in the variable n.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
Therefore, if $t < -100$ we have:
$$
\begin{array}{lcl}
         \ln\left(\large\frac{1}{1+\exp(-t)}\right)
  & = & -\ln\bigl(1+\exp(-t)\bigr) \
  & \approx & -\ln\bigl(\exp(-t)\bigr)  \
  & = & t
\end{array}
$$
Hence $\ln\bigl(S(t)\bigr) \approx t$ for $t < -100$. The following implementation uses this approximation.
Can you write Python code for it?

def logSigmoid(t):
    if t > -100:
        return -np.log(1.0 + np.exp(-t))
    else:
        return t

logSigmoid(-1000)


I am working on the file "Python/6 Classification/Logistic-Regression.ipynb".
The first task is:
We append the number $1.0$ in every row of X.
Can you write Python code for it?

X = np.append(X, np.ones((n, 1)), axis=-1)
X


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
All of our features occur in the list of the 30 most important features.  In order to improve our model we could try to use a classifier that is different from NaiveBayesClassifier.  For example, the MaxentClassifier is more sophisticated than the NaiveBayesClassifier.  However, this classifier also takes a much longer time to train.
Can you write Python code for it?

train_set_features = [(gender_features(n), g) for (n, g) in train_set]
classifier = nltk.MaxentClassifier.train(train_set_features)
nltk.classify.accuracy(classifier, train_set_features)


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Given that this is our first attempt, an accuracy of 80% is not too bad. After all, so far we are using just a single feature.  The question is, whether our classifier is able to generalize its predictions to examples it has not seen before.  In order to answer this question we have to use the test set.  Again, we first have to transform the names from the test set into features.
Can you write Python code for it?

test_set_features = [(gender_features(n), g) for (n, g) in test_set]
nltk.classify.accuracy(classifier, test_set_features)


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
In fact, our new features overfit the training data and do not generalize.  Hence, we conclude that having a suffix of three characters is not helpful.
My final attempt to solve to improve the accuracy contains three ideas:

Instead of just using the first character as a feature, we should use the first two characters.
        After all, we are also using the last two characters.

Often, the way the vowels of a name are connected gives a hint about the gender.

In the same way, the consonants might be helpful.  However, we will only use the set of all consonants
        occurring in a name, not the order in which they appear.

Furthermore, in order to reduce the overfitting we will drop the features that check the occurrence of each character individually.
The function find_vowels$(s)$ takes a string $s$ and strips out all characters that are not vowels.
Can you write Python code for it?

def find_vowels(s):
    return ''.join([c for c in s if c in 'aeiouy'])

find_vowels('Hugo')


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
It seems that the suffix is by far more important than anything else.  Therefore, we try a brute force approach and increase the length of the suffix feature to three characters.  After all three is more than two, so this should be an improvement.  However, we have to take care of the fact that some names have a length of just two characters.  Our new implementation of gender_features deals with this case.
Can you write Python code for it?

def gender_features(name):
    features = {}
    features["first" ] = name[0].lower()
    if len(name) >= 3:
        features["suffix"] = name[-3:].lower()
    else:
        features["suffix"] = name[-2:].lower()
    for letter in string.ascii_lowercase:
        features["has(%s)" % letter] = (letter in name.lower())
    return features

train_set_features = [(gender_features(n), g) for (n, g) in train_set]

classifier = nltk.NaiveBayesClassifier.train(train_set_features)

nltk.classify.accuracy(classifier, train_set_features)


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Let us load these names into lists.
Can you write Python code for it?

nltk.download('names')

FemaleNames = names.words('female.txt')
MaleNames   = names.words('male.txt'  )
print('Number of female first names:', len(FemaleNames))
print('Number of   male first names:', len(MaleNames))


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Let's test this function on the name 'Hugo'.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Next, we check the accuracy on the test set.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Next, we need to decide the features that we want to use in order predict the gender of a name.  Our first attempt to predict the gender of a word uses just a single feature. This feature is the substring containing the last two characters of the name.
The classifiers that are already implemented in NLTK assume a special format for the features: The features of an object to be classified have to be implemented as a dictionary.  The keys of the features are supposed to be short descriptions of the features.  Later, we will try to increase the accuracy of our prediction by adding more features.
Can you write Python code for it?

def gender_features(word):
    return { 'ending': word[-2:] }


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Next, we train a NaiveBayesClassifier with the new features.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Obviously, this is an improvement, but this improvement is less than what we might have hoped for.  Let us check the 30 most important features.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
Our goal is to test whether it is possible to predict the gender of a given name using a Naive Bayes classifier.  In order to be able to make a quantitative assessment of the accuracy of the classifier, we have to split our data into a training dataset and a testing dataset.  To minimize any bias, the assignment of the names into those datasets should be done randomly.  In order for our results to be reproducible, we set a seed for the random number generator.  This ensures that the random number generator will always behave the same way.
Can you write Python code for it?

import random
random.seed(1)
random.shuffle(Names)
len(Names)


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
The Natural Language Toolkit (NLKT)
This notebook introduces the Natural Language ToolKit (NLKT).  Our first example is concerned with classification: We want to see whether it is possible to predict the gender of a given first name.  This example is taken from Chapter 6 of the NLTK book.  To begin with, we import the module nltk.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
The classifier has correctly predicted the gender of 'Hugo' to be male.  But before we get too excited, we should check the accuracy of the classifier on the training set.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
The function find_consonants$(s)$ takes a string $s$ and returns a set of its consonants.
Can you write Python code for it?

def find_consonants(s):
    return frozenset({c for c in s if c not in 'aeiouy'})

find_consonants('Hugo')

def gender_features(name):
    name     = name.lower()
    features = {}
    features["first" ] = name[:2]
    features["suffix"] = name[-2:]
    features["vowels"] = find_vowels(name)
    features["consonants"] = find_consonants(name)
    return features

train_set_features = [(gender_features(n), g) for (n, g) in train_set]
classifier = nltk.NaiveBayesClassifier.train(train_set_features)
nltk.classify.accuracy(classifier, train_set_features)

test_set_features = [(gender_features(n), g) for (n, g) in test_set]
nltk.classify.accuracy(classifier, test_set_features)

classifier.show_most_informative_features(40)


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
The performance on the test set is slightly worse, but given that we have a bias of 20%, there is no need to worry about a variance of 2% at this point.  Of course, this remark only holds if we assume that the so called Bayes optimal error is close to 0%.  If, instead, the Bayes optimal error would be, say, 19%, then we can never achieve an accuracy that is better than 81%. In that case the difference of 2% between the test set and the training set would have to be investigated further, because it is then more promising to reduce this error than to try to reduce the 1% that separates the error on the training set from the best possible error.  Of course, initially we do not know the Bayes optimal error.  For now I am just assuming that it is 15% or less.
The NaiveBayesClassifier has a useful method called $\texttt{show_most_informative_features}(n)$ which shows the $n$ most important features.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
This looks like an improvement.  Let's check the accuracy on the test set.
Can you write Python code for it?


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
We combine these two lists into one list of tagged names, where a tagged name is a pair of the form
$$ (\textrm{name}, \textrm{gender}) \quad \mbox{such that $\textrm{gender} \in {\texttt{'f'},\texttt{'m'}}$.}$$
Can you write Python code for it?

Names = [(n, 'm') for n in MaleNames] + [(n, 'f') for n in FemaleNames]


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
We have to transform the names in our training set into features in order to train a classifier.
Can you write Python code for it?

train_set_features = [(gender_features(n), g) for (n, g) in train_set]
train_set_features[:10]


I am working on the file "Python/6 Classification/NLTK-Introduction.ipynb".
The first task is:
With this new implementation of the function gender_features we have 28 features, which is a lot more than what we had in our first model.  But do these additional features actually improve the performance of our model?  We can only answer this question if we train the model and test it.  Let us compute the features of the training set.
Can you write Python code for it?

train_set_features = [(gender_features(n), g) for (n, g) in train_set]


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Clearly, pure logistic regression is not working for this example.  The reason is, that a linear decision boundary is not able to separate the positive examples from the negative examples.  Let us add polynomial features.  This enables us to create more complex decision boundaries.
The function $\texttt{extend}(X)$ takes a feature matrix $X$ that is supposed to contain two features $x$ and $y$.  It creates the new features $x^2$, $y^2$ and $x\cdot y$ and returns a new feature matrix that also contains these additional features.
Can you write Python code for it?

def extend(X):
    n  = len(X)
    fx = np.reshape(X[:,0], (n, 1)) # extract first column
    fy = np.reshape(X[:,1], (n, 1)) # extract second column
    return np.hstack([fx, fy, fx*fx, fy*fy, fx*fy]) # stack everthing horizontally

X_train_quadratic = extend(X_train)
X_test_quadratic  = extend(X_test)

M, score, accuracy = logistic_regression(X_train_quadratic, Y_train, X_test_quadratic, Y_test)
score, accuracy


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Given that there are only two classes, the accuracy of our first model is quite poor.
Let us extract the coefficients so we can plot the decision boundary.
Can you write Python code for it?

ϑ0     = M.intercept_[0]
ϑ1, ϑ2 = M.coef_[0]

plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.title('A Classification Problem')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('x axis')
plt.ylabel('y axis')
plt.xticks(np.arange(-0.9, 1.1, step=0.1))
plt.yticks(np.arange(-0.8, 1.2, step=0.1))
plt.scatter(X_pass[:,0], X_pass[:,1], color='b')
plt.scatter(X_fail[:,0], X_fail[:,1], color='r')
H = np.arange(-0.8, 1.0, 0.05)
P = -(ϑ0 + ϑ1 * H)/ϑ2
plt.plot(H, P, color='green')


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
In order to build a logistic regression classifier, we import the module linear_model from SciKit-Learn.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
In order to plot the instances according to their class we divide the feature matrix $X$ into two parts. $\texttt{X_pass}$ contains those examples that have class $1$, while $\texttt{X_fail}$ contains those examples that have class $0$.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Let us check whether regularization can help.  Below, the regularization parameter prevents the decision boundary from becoming to wiggly and thus the accuracy on the test set can increase.  The function below plots all the data.
Can you write Python code for it?

def plot_nth_degree_boundary_all(n, C):
    poly         = PolynomialFeatures(n, include_bias=False)
    X_train_poly = poly.fit_transform(X_train)
    X_test_poly  = poly.fit_transform(X_test)
    M, score, accuracy = logistic_regression(X_train_poly, Y_train, X_test_poly, Y_test, C)
    print('The accuracy on the training set is:', score)
    print('The accuracy on the test     set is:', accuracy)
    Z = polynomial_grid(n, M)
    plt.figure(figsize=(15, 10))
    sns.set(style='darkgrid')
    plt.title('A Classification Problem')
    plt.axvline(x=0.0, c='k')
    plt.axhline(y=0.0, c='k')
    plt.xlabel('x axis')
    plt.ylabel('y axis')
    plt.xticks(np.arange(-0.9, 1.11, step=0.1))
    plt.yticks(np.arange(-0.8, 1.21, step=0.1))
    plt.scatter(X_pass[:,0], X_pass[:,1], color='b')
    plt.scatter(X_fail[:,0], X_fail[:,1], color='r')
    CS = plt.contour(A, B, Z, 0, colors='green')

plot_nth_degree_boundary_all(14, 100.0)

plot_nth_degree_boundary_all(20, 100000.0)


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Let us plot the data.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn           as sns

plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.title('A Classification Problem')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('x axis')
plt.ylabel('y axis')
plt.xticks(np.arange(-0.9, 1.1, step=0.1))
plt.yticks(np.arange(-0.8, 1.2, step=0.1))
plt.scatter(X_pass[:,0], X_pass[:,1], color='b')
plt.scatter(X_fail[:,0], X_fail[:,1], color='r')


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Let us test this for the polynomial logistic regression model of degree $4$.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Let's check this out for $n=4$.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Plotting the decision boundary starts to get tedious.
Can you write Python code for it?

a    = np.arange(-1.0, 1.0, 0.005)
b    = np.arange(-1.0, 1.0, 0.005)
A, B = np.meshgrid(a,b)
Z    = ϑ0 + ϑ1 * A + ϑ2 * B + \
       ϑ3 * A**2 + ϑ4 * A * B + ϑ5 * B**2 + \
       ϑ6 * A**3 + ϑ7 * A**2 * B + ϑ8 * A * B**2 + ϑ9 * B**3 + \
       ϑ10 * A**4 + ϑ11 * A**3 * B + ϑ12 * A**2 * B**2 + ϑ13 * A * B**3 + ϑ14 * B**4

plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.title('A Classification Problem')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('x axis')
plt.ylabel('y axis')
plt.xticks(np.arange(-0.9, 1.1, step=0.1))
plt.yticks(np.arange(-0.8, 1.2, step=0.1))
plt.scatter(X_pass[:,0], X_pass[:,1], color='b')
plt.scatter(X_fail[:,0], X_fail[:,1], color='r')
CS = plt.contour(A, B, Z, 0, colors='green')


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
Polynomial Logistic Regression
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
The accuracy on the training set has increased, but we observe that the accuracy on the training set is actually not improving. Again, we proceed to plot the decision boundary.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
The data we want to investigate is stored in the file 'fake-data.csv'.   It is data that I have found somewhere.  I am not sure whether this data is real or fake.  Therefore, I won't discuss the attributes of the data.  The point of the data is that it is a classification problem that can not be solved with
ordinary logistic regression.  We will introduce polynomial logistic regression to solve this problem.
Can you write Python code for it?

DF = pd.read_csv('fake-data.csv')
DF.head()


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
The decision boundary looks strange.  Let's get bold and try to add features of a higher power.
However, in order to understand what is happening, we will only plot the training data.
Can you write Python code for it?

X_pass_train = X_train[Y_train == 1.0]
X_fail_train = X_train[Y_train == 0.0]


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
The function $\texttt{polynomial_grid}(n, M)$ takes a number $n$ and a model $M$.  It returns a meshgrid that can be used to plot the decision boundary of the model.
Can you write Python code for it?

def polynomial_grid(n, M):
    Θ    = [M.intercept_[0]] + list(M.coef_[0])
    a    = np.arange(-1.0, 1.0, 0.005)
    b    = np.arange(-1.0, 1.0, 0.005)
    A, B = np.meshgrid(a,b)
    return eval(polynomial(n))


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
This seems to be the same shape that we have seen earlier.  It looks like the function $\texttt{plot_nth_degree_boundary}(n)$ is working.  Let's try higher degree polynomials.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
We captured one more of the training examples.  Let's get bold, we want a $100\%$ training accuracy.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
We extract the target column and convert it into a NumPy array.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
We use this function to build a model for our data.  Initially, we will take all the available data to create the model.
Can you write Python code for it?

M, score, accuracy = logistic_regression(X, Y, X, Y)
score, accuracy


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
We want to split the data into a training set and a test set.
The training set will be used to compute the parameters of our model, while the
testing set is only used to check the accuracy.  SciKit-Learn has a predefined method
train_test_split that can be used to randomly split data into a training set and a test set.
Can you write Python code for it?

from sklearn.model_selection import train_test_split


I am working on the file "Python/6 Classification/Polynomial-Logistic-Regression.ipynb".
The first task is:
We will split the data at a ratio of $4:1$, i.e. $80\%$ of the data will be used for training, while the remaining $20\%$ is used to test the accuracy.
Can you write Python code for it?

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=1)


I am working on the file "Python/6 Classification/Sigmoid.ipynb".
The first task is:
It can be shown that the derivative of the sigmoid function satisfies the equation
$$ \frac{\mathrm{d}S}{\mathrm{d}x}(t) = S(t) \cdot \bigl(1 - S(t)\bigr) $$
Can you write Python code for it?

def sigmoidPrime(t):
    """
    compute the derivative of the sigmoid function
    """
    return sigmoid(t) * (1 - sigmoid(t))


I am working on the file "Python/6 Classification/Sigmoid.ipynb".
The first task is:
Let us plot this function.
Can you write Python code for it?

n = 1000
X = np.linspace(-8.0, 8.0, num=n)
Y = sigmoid(X)
plt.figure(figsize=(15, 10))
sns.set(style='darkgrid')
plt.margins(0.02)
plt.plot(X, Y)
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel("x")
plt.ylabel("y")
plt.yticks(np.arange(0.0, 1.01, step=0.1))
plt.title("The Sigmoid Function.")


I am working on the file "Python/6 Classification/Sigmoid.ipynb".
The first task is:
The Gauss function with standard deviation $\sigma$ is defined as
$$ t \mapsto \frac{1}{\sqrt{2\cdot\pi}\cdot\sigma} \cdot \exp\left(-\frac{t^2}{2\cdot \sigma^2}\right)$$
Can you write Python code for it?

def gauss(t):
    "Return the Gauss function with standard deviation sqrt(1/3) * pi"
    s = sqrt(1/3) * pi
    return np.exp(-t*t/(2*s*s)) / sqrt(2 * pi) / s


I am working on the file "Python/6 Classification/Sigmoid.ipynb".
The first task is:
The Sigmoid Function
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn           as sns
import numpy             as np
from math import *


I am working on the file "Python/6 Classification/Sigmoid.ipynb".
The first task is:
The sigmoid function $S:\mathbb{R} \rightarrow [0,1]$ is defined as:
$$ S(t) = \frac{1}{1 + \exp(-t)}. $$
Can you write Python code for it?

def sigmoid(t):
    "Returns the sigmoid function S(t)"
    return 1 / (1 + np.exp(-t))


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
Common_Words is a numpy array of the 2500 most common words found in all of our emails.
Can you write Python code for it?

M            = 2500             # number of the most common words to use
Word_Counter = read_all_files()
Common_Words = np.array(list({ w for w, _ in Word_Counter.most_common(M) }))

Common_Words


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
Step 2: Transform Files into Frequency Vectors
Index_Dict is a dictionary that maps from the most common words to their index in the array Common_Words.
Can you write Python code for it?

Index_Dict = { w: i for i, w in enumerate(Common_Words) }
Index_Dict


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
Step 3: Compute the Inverse Document Frequency
In natural language processing, the notion term is used as a synonym for word.
Given a term $t$ and a document $d$, the term frequency $\texttt{tf}(t, d)$ is defined as
$$ \texttt{tf}(t, d) = \frac{d.\texttt{count}(t)}{\texttt{len}(d)}, $$
where $d.\texttt{count}(t)$ counts the number of times $t$ appears in $d$ and $\texttt{len}(d)$ is the length of the list representing $d$.
A corpus is a set of documents.  Given a term $t$ and a corpus $\mathcal{C}$, the inverse document frequency
$\texttt{idf}(t,\mathcal{C})$ is defined as
$$ \texttt{idf}(t,\mathcal{C}) = \ln\left(\frac{\texttt{card}(\mathcal{C}) + 1}{\texttt{card}\bigl({ d \in \mathcal{C} \mid t \in d }\bigr) + 1}\right). $$
The addition of $1$ in both nominator and denominator is called Laplace smoothing.  This is necessary to prevent a division by zero error
for those terms $t$ that do not occur in the list Common_Words.
Step 4: Compute the Feature Matrix
The function $\texttt{feature_matrix}(\texttt{spam_dir}, \texttt{ham_dir})$ takes two directories that contain spam and ham, respectively.
It computes a matrix $X$ and a vector $Y$, where $X$ is the feature matrix and for
every row $r$ of the feature matrix, $Y[r]$ is 1 if the mail is ham and 0 if it's spam.
The way $X$ is computed is quite inefficient, it would have been better to initialize $X$ as a matrix with the shape $(N,M)$, where $N$ is the number of mails and $M$ is the number of common words.
Can you write Python code for it?

def feature_matrix(spam_dir, ham_dir):
    X = []
    Y = []
    for fn in os.listdir(spam_dir):
        X.append(get_word_vector(spam_dir + fn))
        Y.append(0)
    for fn in os.listdir(ham_dir):
        X.append(get_word_vector(ham_dir + fn))
        Y.append(+1)
    X = np.array(X)
    Y = np.array(Y)
    return X, Y


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
Step 5: Train and Test a Support Vector Machine
Can you write Python code for it?


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
The function $\texttt{get_word_set}(\texttt{fn})$ takes a filename $\texttt{fn}$ as its argument.  It reads the file and returns a set of all words that are found in this file.  The words are transformed to lower case.
Can you write Python code for it?

def get_words_set(fn):
    with open(fn) as file:
        text = file.read()
        text = text.lower()
        return set(re.findall(r"[\w']+", text))


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
The function $\texttt{transform_to_vector}(L)$ takes a list of words $L$ and transforms this list into a vector $\mathbf{v}$.  If
$\texttt{CommonWords}[i] = w$, then $\mathbf{v}[i]$ specifies the number of times that $w$ occurs in $L$.
Can you write Python code for it?

def transform_to_vector(L):
    Result = np.zeros((len(Common_Words, )))
    for w in L:
        if w in Index_Dict:
            Result[Index_Dict[w]] += 1
    return Result


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
The function read_all_files reads all files contained in those directories that are stored in the list Directories.
It returns a Counter.  For every word $w$ this counter contains the number of files that contain $w$.
Can you write Python code for it?

def read_all_files():
    Words = Counter()
    for directory in Directories:
        for file_name in os.listdir(directory):
            Words.update(get_words_set(directory + file_name))
    return Words


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
Train an SVM and compute the accuracy on the training data.
Can you write Python code for it?

M = svm.SVC(kernel='linear', C=100000)
M.fit  (X_train, Y_train)
M.score(X_train, Y_train)


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
Up to now, the feature matrix contains only the term frequencies.  Next we multiply with the inverse document frequencies.
Can you write Python code for it?

N, _ = X_train.shape
IDF  = {}
for w, i in Index_Dict.items():
    IDF[w] = np.log((N + 1) / (Word_Counter[w] + 1))
    X_train[:, i] = X_train[:, i] * IDF[w]


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
We build the feature matrix for the test set.
Can you write Python code for it?

X_test, Y_test = feature_matrix(spam_dir_test, ham__dir_test)

for w, i in Index_Dict.items():
    X_test[:, i] = X_test[:, i] * IDF[w]


I am working on the file "Python/6 Classification/Spam-Detection-SVM.ipynb".
The first task is:
We convert the training set into a feature matrix.
Can you write Python code for it?

%%time
X_train, Y_train = feature_matrix(spam_dir_train, ham__dir_train)


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
According to our computation, the probability that a spam email contains the word 'consonant' is about $0.28\%$, while the probability that this word occurs in a ham email is $2.55\%$.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
Common_Words is a list of the 2500 most common words found in all of our emails.
Can you write Python code for it?

N            = 2500             # number of the most common words to use
Word_Counter = read_all_files()
Word_Counter

Common_Words = { w for w, _ in Word_Counter.most_common(N) }
Common_Words


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
Computing the Conditional Probabilities
Having computed the most common words, we are now ready to compute the conditional probability that a given word occurs in a spam email.
The function $\texttt{get_common_words}(\texttt{fn})$ takes a filename $\texttt{fn}$
as its argument.  It reads the file and returns the set of all words in Common_Words that are found in the given file.
Can you write Python code for it?

def get_common_words(fn):
    return get_words(fn) & Common_Words


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
Deciding whether an Email is Spam
Given a file name fn, this function returns the probability that the message contained in the given file is spam.
When implementing the formula
$$\arg\max\limits_{C \in \mathcal{C}}  \left(\prod\limits_{i=1}^m P(f_i \;|\; C)\right) \cdot P(C) $$
we have to be careful, because a naive implementation will eveluate the product
$$\prod\limits_{i=1}^m P(f_i \;|\; C)$$
as the number $0$ due to numerical underflow.  The trick to compute this product is to remember that
$$ \ln(a \cdot b) = \ln(a) + \ln(b) $$
and therefore transform the product into a sum of logarithms:
$$ \prod\limits_{i=1}^m P(f_i \;|\; C) = \exp\left(\alpha + \sum\limits_{i=1}^m \ln\bigl(P(f_i \;|\; C)\bigr) \right) \cdot \exp(-\alpha)$$
Here, the constant $\alpha$ has to be chosen such that the application of the function exp to the value
$$ \alpha + \sum\limits_{i=1}^m \ln\bigl(P(f_i \;|\; C)\bigr) $$
does not lead to an underflow error.
As we want to compute a probability, we have to be aware that the term
$$ \left(\prod\limits_{i=1}^m P(f_i \;|\; C)\right) \cdot P(C) $$
is not the probability that the object is of class $C$ but rather is only proportional to this probability.  The fact that the probability
of an email being spam + the probability that the email is ham must be $1$ enables us to compute the probability.
Can you write Python code for it?

def spam_probability(fn):
    log_p_spam = 0.0
    log_p_ham  = 0.0
    words = get_common_words(fn)
    for w in Common_Words:
        if w in words:
            log_p_spam += math.log(Spam_Probability[w])
            log_p_ham  += math.log(Ham__Probability[w])
        else:
            log_p_spam += math.log(1.0 - Spam_Probability[w])
            log_p_ham  += math.log(1.0 - Ham__Probability[w])
    alpha  = abs(max(log_p_spam, log_p_ham))
    p_spam = math.exp(log_p_spam + alpha) * spam_prior
    p_ham  = math.exp(log_p_ham  + alpha) * ham__prior
    return p_spam / (p_spam + p_ham)


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
I have checked that the proportion of spam and ham emails in the test directory is also $1:1$.  If the proportion of spam and ham emails in life is different from $1:1$, then we would have to use this proportion in the spam filter to be developed.
The function $\texttt{get_words}(\texttt{fn})$ takes a filename $\texttt{fn}$ as its argument.  It reads the file and returns a set of all words that are found in this file.  The words are transformed to lower case.
Can you write Python code for it?

def get_words(fn):
    file = open(fn)
    text = file.read()
    text = text.lower()
    return set(re.findall(r"[\w']+", text))


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
In order to compute the prior probability that an email is ham or spam we need to count the number of spam and ham emails.
Can you write Python code for it?

no_spam    = len(os.listdir(spam_dir_train))
no_ham     = len(os.listdir(ham__dir_train))
spam_prior = no_spam / (no_spam + no_ham)
ham__prior = no_ham  / (no_spam + no_ham)
spam_prior, ham__prior


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
Let us test this function with a small example mail.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
Let us test this with a ham email.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
Next, we compute dictionaries that store the number of occurrences in emails for every common word.
Can you write Python code for it?

Spam_Counter = count_commmon_words(spam_dir_train)
Spam_Counter

Ham__Counter = count_commmon_words(ham__dir_train)
Ham__Counter


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
The function count_common_words takes a string specifying a directory.  It returns a
Counter that counts how often the words in Common_Words occur in any of the files in directory.
Can you write Python code for it?

def count_commmon_words(directory):
    Words = Counter()
    for file_name in os.listdir(directory):
        Words.update(get_common_words(directory + file_name))
    return Words


I am working on the file "Python/6 Classification/Spam-Detection.ipynb".
The first task is:
The function read_all_files reads all files contained in those directories that are stored in the list Directories.
It returns a Counter.  For every word $w$ this counter contains the number of files that contain $w$.
Can you write Python code for it?

def read_all_files():
    Words = Counter()
    for directory in Directories:
        for file_name in os.listdir(directory):
            Words.update(get_words(directory + file_name))
    return Words


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
Support Vector Machines
This notebook discusses support vector machines.  In order to understand why we need support vector machines (abbreviated as SVMs), we
will first demonstrate that classifiers constructed with logistic regression sometimes behave unintuitively.
The Problem with Logistic Regression
In this section of the notebook we discuss an example that demonstrates that logistic regression is not necessarily the best classifier we can get.
Can you write Python code for it?

import numpy                as np
import matplotlib.pyplot    as plt
import seaborn              as sns
import sklearn.linear_model as lm


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
The decision boundary separates the data perfectly because it maximizes the distance of the data from the boundary.
Can you write Python code for it?

X = np.array([[1.00, 2.00],
              [2.00, 1.00],
              [3.50, 3.50]])
Y = np.array([0,
              0,
              1])

plot_data_and_boundary(X, Y, M, 'three points')

import pandas as pd


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
The following function is used for plotting.
Can you write Python code for it?

def plot_data_and_boundary(X, Y, M, title):
    Corner = np.array([[0.0, 5.0], [5.0, 0.0]])
    X0, X1 = X[:, 0], X[:, 1]
    XX, YY = np.meshgrid(np.arange(0, 5, 0.005), np.arange(0, 5, 0.005))
    Z = M.predict(np.c_[XX.ravel(), YY.ravel()])
    Z = Z.reshape(XX.shape)
    plt.figure(figsize=(10, 10))
    sns.set(style='darkgrid')
    plt.contour(XX, YY, Z)
    plt.scatter(Corner[:,0], Corner[:,1], color='black', marker='.')
    plt.scatter(X0, X1, c=Y, edgecolors='k')
    plt.xlim(XX.min(), XX.max())
    plt.ylim(YY.min(), YY.max())
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.xticks()
    plt.yticks()
    plt.title(title)

plot_data_and_boundary(X, Y, M, 'some data')


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
The function $\texttt{train_and_plot}(X, Y)$ takes a design matrix $X$ and a vector $Y$ containing zeros and ones.  It builds a regression model and plots the data together with the decision boundary.
Can you write Python code for it?

def train_and_plot(X, Y):
    M = lm.LogisticRegression(C=1, solver='lbfgs')
    M.fit(X, Y)
    ϑ0     = M.intercept_[0]
    ϑ1, ϑ2 = M.coef_[0]
    plot_data_and_boundary(X, Y, ϑ0, ϑ1, ϑ2)

train_and_plot(X, Y)


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
To proceed, we will plot the data points using a scatter plot.  Furthermore, we plot a green line that intuitively make the best decision boundary.
Can you write Python code for it?

plt.figure(figsize=(12, 12))
Corner = np.array([[0.0, 5.0], [5.0, 0.0]])
X_pass = X[Y == 1]
X_fail = X[Y == 0]
sns.set(style='darkgrid')
plt.title('A Simple Classification Problem')
plt.axvline(x=0.0, c='k')
plt.axhline(y=0.0, c='k')
plt.xlabel('x1')
plt.ylabel('x2')
plt.xticks(np.arange(0.0, 5.1, step=0.5))
plt.yticks(np.arange(0.0, 5.1, step=0.5))
X1 = np.arange(0, 5.05, 0.05)
X2 = 5 - X1
plt.plot(X1, X2, color='green', linestyle='-')
X1 = np.arange(0, 3.05, 0.05)
X2 = 3 - X1
plt.plot(X1, X2, color='cyan', linestyle=':')
X1 = np.arange(2.0, 5.05, 0.05)
X2 = 7 - X1
plt.plot(X1, X2, color='cyan', linestyle=':')
plt.scatter(Corner[:,0], Corner[:,1], color='white', marker='.')
plt.scatter(X_pass[:,0], X_pass[:,1], color='b', marker='o') # class 1 is blue
plt.scatter(X_fail[:,0], X_fail[:,1], color='r', marker='x') # class 2 is red


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
We decision boundary is closer to the blue data point than to the red data points.  This is not optimal.
The function $\texttt{gen_X_Y}(n)$ take a natural number $n$ and generates additional data.  The number $n$ is the number of blue data points.
Concretely, it will add $n-1$ data points to the right of the blue dot shown above.  This should not really change the decision boundary as the data
do not provide any new information.  After all, these data are to the right of the first blue dot and hence should share the class of this data point.
Can you write Python code for it?


I am working on the file "Python/6 Classification/Support-Vector-Machines.ipynb".
The first task is:
When we test logistic regression with this data set, we see that the slope of the decision boundary is much steeper now and the separation of the blue dots from the red crosses is far worse than it needs to be, had the optimal decision boundary been computed.
Let us see how support vector machines deal with these data.
Can you write Python code for it?


I am working on the file "Python/60_Registration_Introduction.ipynb".
The first task is:
If we are satisfied with the results, save them to file.
Can you write Python code for it?

sitk.WriteImage(
    moving_resampled, os.path.join(OUTPUT_DIR, "RIRE_training_001_mr_T1_resampled.mha")
)
sitk.WriteTransform(
    final_transform, os.path.join(OUTPUT_DIR, "RIRE_training_001_CT_2_mr_T1.tfm")
)


I am working on the file "Python/60_Registration_Introduction.ipynb".
The first task is:
Initial Alignment
Use the CenteredTransformInitializer to align the centers of the two volumes and set the center of rotation to the center of the fixed image.
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(
    fixed_image,
    moving_image,
    sitk.Euler3DTransform(),
    sitk.CenteredTransformInitializerFilter.GEOMETRY,
)

moving_resampled = sitk.Resample(
    moving_image,
    fixed_image,
    initial_transform,
    sitk.sitkLinear,
    0.0,
    moving_image.GetPixelID(),
)

interact(
    display_images_with_alpha,
    image_z=(0, fixed_image.GetSize()[2] - 1),
    alpha=(0.0, 1.0, 0.05),
    fixed=fixed(fixed_image),
    moving=fixed(moving_resampled),
);


I am working on the file "Python/60_Registration_Introduction.ipynb".
The first task is:
Now visually inspect the results.
Can you write Python code for it?

moving_resampled = sitk.Resample(
    moving_image,
    fixed_image,
    final_transform,
    sitk.sitkLinear,
    0.0,
    moving_image.GetPixelID(),
)

interact(
    display_images_with_alpha,
    image_z=(0, fixed_image.GetSize()[2] - 1),
    alpha=(0.0, 1.0, 0.05),
    fixed=fixed(fixed_image),
    moving=fixed(moving_resampled),
);


I am working on the file "Python/60_Registration_Introduction.ipynb".
The first task is:
Post registration analysis
Query the registration method to see the metric value and the reason the optimization terminated.
The metric value allows us to compare multiple registration runs as there is a probabilistic aspect to our registration, we are using random sampling to estimate the similarity metric.
Always remember to query why the optimizer terminated. This will help you understand whether termination is too early, either due to thresholds being too tight, early termination due to small number of iterations - numberOfIterations, or too loose, early termination due to large value for minimal change in similarity measure - convergenceMinimumValue)
Can you write Python code for it?

print(f"Final metric value: {registration_method.GetMetricValue()}")
print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)


I am working on the file "Python/60_Registration_Introduction.ipynb".
The first task is:
Registration
The specific registration task at hand estimates a 3D rigid transformation between images of different modalities. There are multiple components from each group (optimizers, similarity metrics, interpolators) that are appropriate for the task. Note that each component selection requires setting some parameter values. We have made the following choices:

Similarity metric, mutual information (Mattes MI):

Number of histogram bins, 50.

Sampling strategy, random.

Sampling percentage, 1%.

Interpolator, sitkLinear.

Optimizer, gradient descent:

Learning rate, step size along traversal direction in parameter space, 1.0 .

Number of iterations, maximal number of iterations, 100.

Convergence minimum value, value used for convergence checking in conjunction with the energy profile of the similarity metric that is estimated in the given window size, 1e-6.

Convergence window size, number of values of the similarity metric which are used to estimate the energy profile of the similarity metric, 10.

Perform registration using the settings given above, and take advantage of the built in multi-resolution framework, use a three tier pyramid.
In this example we plot the similarity metric's value during registration. Note that the change of scales in the multi-resolution framework is readily visible.
Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()

# Similarity metric settings.
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)

registration_method.SetInterpolator(sitk.sitkLinear)

# Optimizer settings.
registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0,
    numberOfIterations=100,
    convergenceMinimumValue=1e-6,
    convergenceWindowSize=10,
)
registration_method.SetOptimizerScalesFromPhysicalShift()

# Setup for the multi-resolution framework.
registration_method.SetShrinkFactorsPerLevel(shrinkFactors=[4, 2, 1])
registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2, 1, 0])
registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

# Don't optimize in-place, we would possibly like to run this cell multiple times.
registration_method.SetInitialTransform(initial_transform, inPlace=False)

# Connect all of the observers so that we can perform plotting during registration.
registration_method.AddCommand(sitk.sitkStartEvent, start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, end_plot)
registration_method.AddCommand(
    sitk.sitkMultiResolutionIterationEvent, update_multires_iterations
)
registration_method.AddCommand(
    sitk.sitkIterationEvent, lambda: plot_values(registration_method)
)

final_transform = registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)


I am working on the file "Python/60_Registration_Introduction.ipynb".
The first task is:
Utility functions
A number of utility callback functions for image display and for plotting the similarity metric during registration.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

from ipywidgets import interact, fixed
from IPython.display import clear_output

# Callback invoked by the interact IPython method for scrolling through the image stacks of
# the two images (moving and fixed).
def display_images(fixed_image_z, moving_image_z, fixed_npa, moving_npa):
    # Create a figure with two subplots and the specified size.
    plt.subplots(1, 2, figsize=(10, 8))

    # Draw the fixed image in the first subplot.
    plt.subplot(1, 2, 1)
    plt.imshow(fixed_npa[fixed_image_z, :, :], cmap=plt.cm.Greys_r)
    plt.title("fixed image")
    plt.axis("off")

    # Draw the moving image in the second subplot.
    plt.subplot(1, 2, 2)
    plt.imshow(moving_npa[moving_image_z, :, :], cmap=plt.cm.Greys_r)
    plt.title("moving image")
    plt.axis("off")

    plt.show()

# Callback invoked by the IPython interact method for scrolling and modifying the alpha blending
# of an image stack of two images that occupy the same physical space.
def display_images_with_alpha(image_z, alpha, fixed, moving):
    img = (1.0 - alpha) * fixed[:, :, image_z] + alpha * moving[:, :, image_z]
    plt.imshow(sitk.GetArrayViewFromImage(img), cmap=plt.cm.Greys_r)
    plt.axis("off")
    plt.show()

# Callback invoked when the StartEvent happens, sets up our new data.
def start_plot():
    global metric_values, multires_iterations

    metric_values = []
    multires_iterations = []

# Callback invoked when the EndEvent happens, do cleanup of data and figure.
def end_plot():
    global metric_values, multires_iterations

    del metric_values
    del multires_iterations
    # Close figure, we don't want to get a duplicate of the plot latter on.
    plt.close()

# Callback invoked when the IterationEvent happens, update our data and display new figure.
def plot_values(registration_method):
    global metric_values, multires_iterations

    metric_values.append(registration_method.GetMetricValue())
    # Clear the output area (wait=True, to reduce flickering), and...


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Final registration
Version 1

 Single scale (not using image pyramid).

 Initial transformation is not modified in place.

Illustrate the need for scaling the step size differently for each parameter:

 SetOptimizerScalesFromIndexShift - estimated from maximum shift of voxel indexes (only use if data is isotropic).

 SetOptimizerScalesFromPhysicalShift - estimated from maximum shift of physical locations of voxels.

 SetOptimizerScalesFromJacobian - estimated from the averaged squared norm of the Jacobian w.r.t. parameters.

Look at the optimizer's stopping condition to ensure we have not terminated prematurely.

Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()

registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)

registration_method.SetInterpolator(sitk.sitkLinear)

# The learningRate parameter is always required. Using the default
# configuration this parameter is ignored because it is overridden
# by the default setting of the estimateLearningRate parameter which
# is sitk.ImageRegistrationMethod.Once. For the user selected
# learningRate to take effect you need to also set the
# estimateLearningRate parameter to sitk.ImageRegistrationMethod.Never
registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0, numberOfIterations=100
)
# Scale the step size differently for each parameter, this is critical!!!
registration_method.SetOptimizerScalesFromPhysicalShift()

registration_method.SetInitialTransform(initial_transform, inPlace=False)

registration_method.AddCommand(
    sitk.sitkStartEvent, registration_callbacks.metric_start_plot
)
registration_method.AddCommand(
    sitk.sitkEndEvent, registration_callbacks.metric_end_plot
)
registration_method.AddCommand(
    sitk.sitkIterationEvent,
    lambda: registration_callbacks.metric_plot_values(registration_method),
)

final_transform_v1 = registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)

print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)
print(f"Final metric value: {registration_method.GetMetricValue()}")

# Save moving image after registration and view overlap using external viewer.
save_transform_and_image(
    final_transform_v1,
    fixed_image,
    moving_image,
    os.path.join(OUTPUT_DIR, "finalAlignment-v1"),
)


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Loading Data
In this notebook we will work with CT and MR scans of the CIRS 057A multi-modality abdominal phantom. The scans are multi-slice DICOM images. The data is stored in a zip archive which is automatically retrieved and extracted when we request a file which is part of the archive.
Can you write Python code for it?

data_directory = os.path.dirname(fdata("CIRS057A_MR_CT_DICOM/readme.txt"))

# 'selected_series_moving/fixed' will be updated by the interact function.
selected_series_fixed = ""
selected_series_moving = ""

# Directory contains multiple DICOM studies/series, store the file names
# in dictionary with the key being the series ID.
reader = sitk.ImageSeriesReader()
series_file_names = {}
series_IDs = list(reader.GetGDCMSeriesIDs(data_directory))  # list of all series

if series_IDs:  # check that we have at least one series
    for series in series_IDs:
        series_file_names[series] = reader.GetGDCMSeriesFileNames(
            data_directory, series
        )
    interact(
        DICOM_series_dropdown_callback,
        fixed_image=series_IDs,
        moving_image=series_IDs,
        series_dictionary=fixed(series_file_names),
    )
else:
    print("This is surprising, data directory does not contain any DICOM series.")

# Actually read the data based on the user's selection.
fixed_image = sitk.ReadImage(series_file_names[selected_series_fixed])
moving_image = sitk.ReadImage(series_file_names[selected_series_moving])

# Save images to file and view overlap using external viewer.
sitk.WriteImage(fixed_image, os.path.join(OUTPUT_DIR, "fixedImage.mha"))
sitk.WriteImage(moving_image, os.path.join(OUTPUT_DIR, "preAlignment.mha"))


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Look at the final transformation, what type is it?
Can you write Python code for it?


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Look at the final transformation, what type is it? Why is it different from the previous example?
Can you write Python code for it?


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Utility functions
A number of utility functions, saving a transform and corresponding resampled image, callback for selecting a
DICOM series from several series found in the same directory.
Can you write Python code for it?

def save_transform_and_image(transform, fixed_image, moving_image, outputfile_prefix):
    """
    Write the given transformation to file, resample the moving_image onto the fixed_images grid and save the
    result to file.

    Args:
        transform (SimpleITK Transform): transform that maps points from the fixed image coordinate system to the moving.
        fixed_image (SimpleITK Image): resample onto the spatial grid defined by this image.
        moving_image (SimpleITK Image): resample this image.
        outputfile_prefix (string): transform is written to outputfile_prefix.tfm and resampled image is written to
                                    outputfile_prefix.mha.
    """
    resample = sitk.ResampleImageFilter()
    resample.SetReferenceImage(fixed_image)

    # SimpleITK supports several interpolation options, we go with the simplest that gives reasonable results.
    resample.SetInterpolator(sitk.sitkLinear)
    resample.SetTransform(transform)
    sitk.WriteImage(resample.Execute(moving_image), outputfile_prefix + ".mha")
    sitk.WriteTransform(transform, outputfile_prefix + ".tfm")

def DICOM_series_dropdown_callback(fixed_image, moving_image, series_dictionary):
    """
    Callback from dropbox which selects the two series which will be used for registration.
    The callback prints out some information about each of the series from the meta-data dictionary.
    For a list of all meta-dictionary tags and their human readable names see DICOM standard part 6,
    Data Dictionary (http://medical.nema.org/medical/dicom/current/output/pdf/part06.pdf)
    """
    # The callback will update these global variables with the user selection.
    global selected_series_fixed
    global selected_series_moving

    img_fixed = sitk.ReadImage(series_dictionary[fixed_image][0])
    img_moving = sitk.ReadImage(series_dictionary[moving_image][0])

    # There are many interesting tags in the DICOM data dictionary, display a selected few.
    tags_to_print = {
        "0010|0010": "Patient name: ",
      ...


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Version 1.1
The previous example illustrated the use of the ITK v4 registration framework in an ITK v3 manner. We only referred to a single transformation which was what we optimized.
In ITK v4 the registration method accepts three transformations (if you look at the diagram above you will only see two transformations, Moving transform represents $T_{opt} \circ T_m$):

SetInitialTransform, $T_{opt}$ - composed with the moving initial transform, maps points from the virtual image domain to the moving image domain, modified during optimization.

SetFixedInitialTransform $T_f$- maps points from the virtual image domain to the fixed image domain, never modified.

SetMovingInitialTransform $T_m$- maps points from the virtual image domain to the moving image domain, never modified.

The transformation that maps points from the fixed to moving image domains is thus: $^M\mathbf{p}  = T_{opt}(T_m(T_f^{-1}(^F\mathbf{p})))$
We now modify the previous example to use $T_{opt}$ and $T_m$.
Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(sitk.sitkLinear)
registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0, numberOfIterations=100
)
registration_method.SetOptimizerScalesFromPhysicalShift()

# Set the initial moving and optimized transforms.
optimized_transform = sitk.Euler3DTransform()
registration_method.SetMovingInitialTransform(initial_transform)
registration_method.SetInitialTransform(optimized_transform)

registration_method.AddCommand(
    sitk.sitkStartEvent, registration_callbacks.metric_start_plot
)
registration_method.AddCommand(
    sitk.sitkEndEvent, registration_callbacks.metric_end_plot
)
registration_method.AddCommand(
    sitk.sitkIterationEvent,
    lambda: registration_callbacks.metric_plot_values(registration_method),
)

registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)

# Need to compose the transformations after registration.
final_transform_v11 = sitk.CompositeTransform(optimized_transform)
final_transform_v11.AddTransform(initial_transform)

print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)
print(f"Final metric value: {registration_method.GetMetricValue()}")

# Save moving image after registration and view overlap using external viewer.
save_transform_and_image(
    final_transform_v11,
    fixed_image,
    moving_image,
    os.path.join(OUTPUT_DIR, "finalAlignment-v1.1"),
)


I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Version 2

 Multi scale - specify both scale, and how much to smooth with respect to original image.

 Initial transformation modified in place, so in the end we have the same type of transformation in hand.

Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()

registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)

registration_method.SetInterpolator(sitk.sitkLinear)

registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0, numberOfIterations=100
)  # , estimateLearningRate=registration_method.EachIteration)
registration_method.SetOptimizerScalesFromPhysicalShift()

final_transform = sitk.Euler3DTransform(initial_transform)
registration_method.SetInitialTransform(final_transform)
registration_method.SetShrinkFactorsPerLevel(shrinkFactors=[4, 2, 1])
registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2, 1, 0])
registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

registration_method.AddCommand(
    sitk.sitkStartEvent, registration_callbacks.metric_start_plot
)
registration_method.AddCommand(
    sitk.sitkEndEvent, registration_callbacks.metric_end_plot
)
registration_method.AddCommand(
    sitk.sitkMultiResolutionIterationEvent,
    registration_callbacks.metric_update_multires_iterations,
)
registration_method.AddCommand(
    sitk.sitkIterationEvent,
    lambda: registration_callbacks.metric_plot_values(registration_method),
)

registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)

print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)
print(f"Final metric value: {registration_method.GetMetricValue()}")

# Save moving image after registration and view overlap using external viewer.
save_transform_and_image(
    final_transform,
    fixed_image,
    moving_image,
    os.path.join(OUTPUT_DIR, "finalAlignment-v2"),
)


I am working on the file "Python/62_Registration_Tuning.ipynb".
The first task is:
Initial Alignment
We use the CenteredTransformInitializer. Should we use the GEOMETRY based version or the MOMENTS based one?
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(
    sitk.Cast(fixed_image, moving_image.GetPixelID()),
    moving_image,
    sitk.Euler3DTransform(),
    sitk.CenteredTransformInitializerFilter.GEOMETRY,
)

(
    initial_errors_mean,
    initial_errors_std,
    initial_errors_min,
    initial_errors_max,
    _,
) = ru.registration_errors(
    initial_transform,
    fixed_points,
    moving_points,
    min_err=pre_errors_min,
    max_err=pre_errors_max,
    display_errors=True,
)
print(
    f"After initialization, errors (TRE) in millimeters, mean(std): {initial_errors_mean:.2f}({initial_errors_std:.2f}), max: {initial_errors_max:.2f}"
)


I am working on the file "Python/62_Registration_Tuning.ipynb".
The first task is:
Now using the built in multi-resolution framework
Perform registration using the same settings as above, but take advantage of the multi-resolution framework which provides a significant speedup with minimal effort (3 lines of code).
It should be noted that when using this framework the similarity metric value will not necessarily decrease between resolutions, we are only ensured that it decreases per resolution. This is not an issue, as we are actually observing the values of a different function at each resolution.
The example below shows that registration is improving even though the similarity value increases when changing resolution levels.
Can you write Python code for it?

#%%timeit -r1 -n1
# the arguments to the timeit magic specify that this cell should only be run once. running it multiple
# times to get performance statistics is also possible, but takes time. if you want to analyze the accuracy
# results from multiple runs you will have to modify the code to save them instead of just printing them out.

registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.1)
registration_method.SetInterpolator(sitk.sitkLinear)  # 2. Replace with sitkLinear
registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0, numberOfIterations=100
)
registration_method.SetOptimizerScalesFromPhysicalShift()

# Don't optimize in-place, we would like to run this cell multiple times
registration_method.SetInitialTransform(initial_transform, inPlace=False)

# Add callbacks which will display the similarity measure value and the reference data during the registration process
registration_method.AddCommand(sitk.sitkStartEvent, rc.metric_and_reference_start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, rc.metric_and_reference_end_plot)
registration_method.AddCommand(
    sitk.sitkIterationEvent,
    lambda: rc.metric_and_reference_plot_values(
        registration_method, fixed_points, moving_points
    ),
)

registration_method.SetShrinkFactorsPerLevel(shrinkFactors=[4, 2, 1])
registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2, 1, 0])
registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

final_transform = registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)

print(f"Final metric value: {registration_method.GetMetricValue()}")
print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)
final_errors_mean, final_errors_std, _, final_errors_max,...


I am working on the file "Python/62_Registration_Tuning.ipynb".
The first task is:
Registration
Possible choices for simple rigid multi-modality registration framework (300 component combinations, in addition to parameter settings for each of the components):

Similarity metric, 2 options (Mattes MI, JointHistogram MI):

Number of histogram bins.

Sampling strategy, 3 options (NONE, REGULAR, RANDOM)

Sampling percentage.

Interpolator, 10 options (sitkNearestNeighbor, sitkLinear, sitkGaussian, sitkBSpline,...)

Optimizer, 5 options (GradientDescent, GradientDescentLineSearch, RegularStepGradientDescent...):

Number of iterations.

learning rate (step size along parameter space traversal direction).

In this example we will plot the similarity metric's value and more importantly the TREs for our reference data. A good choice for the former should be reflected by the later. That is, the TREs should go down as the similarity measure value goes down (not necessarily at the same rates).
Finally, we are also interested in timing our registration. IPython allows us to do this with minimal effort using the timeit cell magic (IPython has a set of predefined functions that use a command line syntax, and are referred to as magic functions).
Can you write Python code for it?

#%%timeit -r1 -n1
# to time this cell uncomment the line above
# the arguments to the timeit magic specify that this cell should only be run once. running it multiple
# times to get performance statistics is also possible, but takes time. if you want to analyze the accuracy
# results from multiple runs you will have to modify the code to save them instead of just printing them out.

registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(
    sitk.sitkNearestNeighbor
)  # 2. Replace with sitkLinear
registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0, numberOfIterations=100
)  # 1. Increase to 1000
registration_method.SetOptimizerScalesFromPhysicalShift()

# Don't optimize in-place, we would like to run this cell multiple times
registration_method.SetInitialTransform(initial_transform, inPlace=False)

# Add callbacks which will display the similarity measure value and the reference data during the registration process
registration_method.AddCommand(sitk.sitkStartEvent, rc.metric_and_reference_start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, rc.metric_and_reference_end_plot)
registration_method.AddCommand(
    sitk.sitkIterationEvent,
    lambda: rc.metric_and_reference_plot_values(
        registration_method, fixed_points, moving_points
    ),
)

final_transform_single_scale = registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)

print(f"Final metric value: {registration_method.GetMetricValue()}")
print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)
final_errors_mean, final_errors_std, _, final_errors_max, _ = ru.registration_errors(
    final_transform_single_scale,
    fixed_points,
    moving_points,
   ...


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Exhaustive optimizer
The exhaustive optimizer evaluates the similarity metric on a grid in parameter space centered on the parameters of the initial transform. This grid is defined using three elements:
1. numberOfSteps.
2. stepLength.
3. optimizer scales.
The similarity metric is evaluated on the resulting parameter grid:
initial_parameters ± numberOfSteps × stepLength × optimizerScales
Example:
1. numberOfSteps=[1,0,2,0,0,0]
2. stepLength = np.pi
3. optimizer scales = [1,1,0.5,1,1,1]
Will perform 15 metric evaluations ($\displaystyle\prod_i (2*numberOfSteps[i] + 1)$).
The parameter values for the second parameter and the last three parameters are the initial parameter values. The parameter values for the first parameter are $v_{init}-\pi, v_{init}, v_{init}+\pi$ and the parameter values for the third parameter are $v_{init}-\pi, v_{init}-\pi/2, v_{init}, v_{init}+\pi/2, v_{init}+\pi$.
The transformation corresponding to the lowest similarity metric is returned.
Using this approach we have superfluous evaluations, due to the symmetry of the grid in parameter space. On the other hand this method is often faster than evaluating the metric using the MetricEvaluate method (due to the setup and tear down time).
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(
    fixed_image,
    moving_image,
    sitk.Euler3DTransform(),
    sitk.CenteredTransformInitializerFilter.GEOMETRY,
)
registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(sitk.sitkLinear)
# The order of parameters for the Euler3DTransform is [angle_x, angle_y, angle_z, t_x, t_y, t_z]. The parameter
# sampling grid is centered on the initial_transform parameter values, that are all zero for the rotations. Given
# the number of steps and their length and optimizer scales we have:
# angle_x = 0
# angle_y = -pi, 0, pi
# angle_z = -pi, 0, pi
registration_method.SetOptimizerAsExhaustive(
    numberOfSteps=[0, 1, 1, 0, 0, 0], stepLength=np.pi
)
registration_method.SetOptimizerScales([1, 1, 1, 1, 1, 1])

# Perform the registration in-place so that the initial_transform is modified.
registration_method.SetInitialTransform(initial_transform, inPlace=True)
registration_method.Execute(fixed_image, moving_image)

print("best initial transformation is: " + str(initial_transform.GetParameters()))


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Find an initial translation for the image by scrolling through the moving image volume.
Can you write Python code for it?

slice_index_interface = gui.MultiImageDisplay(
    [fixed_image_slice, moving_image],
    title_list=["fixed image", "moving image"],
    figure_size=(8, 4),
    window_level_list=[ct_window_level, mr_window_level],
    intensity_slider_range_percentile=[0, 100],
);

# Align the center of the fixed image slice to the center of the user selected moving image slice
selected_slice_index = slice_index_interface.slider_list[1].value
# FOR TESTING: next line sets a value for the selected slice, comment out when you actually want to work.
selected_slice_index = moving_image.GetDepth() // 2
moving_image_slice_center = moving_image.TransformContinuousIndexToPhysicalPoint(
    [
        (moving_image.GetWidth() - 1) / 2,
        (moving_image.GetHeight() - 1) / 2,
        selected_slice_index,
    ]
)
initial_transform.SetTranslation(
    [m - f for m, f in zip(moving_image_slice_center, fixed_image_center)]
)

gui.RegistrationPointDataAquisition(
    fixed_image_slice,
    moving_image,
    figure_size=(8, 4),
    known_transformation=initial_transform,
    fixed_window_level=ct_window_level,
    moving_window_level=mr_window_level,
);


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Register using centered transform initializer  (assumes orientation is similar)
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(
    fixed_image,
    moving_image,
    sitk.Euler3DTransform(),
    sitk.CenteredTransformInitializerFilter.GEOMETRY,
)

final_transform, _ = multires_registration(fixed_image, moving_image, initial_transform)


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Register using manual initialization - slice to volume
In some cases, initialization may be more critical than others. For example, when registering a 2D slice to a 3D image. This requires careful initialization because the potential for converging to a local minimum is much larger than when we register two corresponding volumes. Note that SimpleITK does not support 2D/3D registration. A slice to volume registration is a 3D/3D registration. The slice is a "very thin" 3D image, one of the axes has a size of one.
In the next cells we explore the basic slice to volume registration scenario.
Can you write Python code for it?

fixed_image_slice = fixed_image[
    :, :, fixed_image.GetDepth() // 2 : fixed_image.GetDepth() // 2 + 1
]
# We know our image is rotated by 180 degrees around the y axis, and the rotation center
# should be the center of the image.
fixed_image_center = fixed_image_slice.TransformContinuousIndexToPhysicalPoint(
    [(sz - 1) / 2 for sz in fixed_image_slice.GetSize()]
)
initial_transform = sitk.Euler3DTransform(fixed_image_center, 0, 3.141592653589793, 0)


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Run the registration and visually evaluate the results:
Can you write Python code for it?

final_transform, _ = multires_registration(fixed_image, moving_image, initial_transform)
gui.RegistrationPointDataAquisition(
    fixed_image,
    moving_image,
    figure_size=(8, 4),
    known_transformation=final_transform,
    fixed_window_level=ct_window_level,
    moving_window_level=mr_window_level,
);


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Visually evaluate the registration results:
Can you write Python code for it?

gui.RegistrationPointDataAquisition(
    fixed_image,
    moving_image,
    figure_size=(8, 4),
    known_transformation=final_transform,
    fixed_window_level=ct_window_level,
    moving_window_level=mr_window_level,
);


I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Visually evaluate the results using a linked cursor approach, a mouse click in one image will create the "corresponding" point in the other image. Don't be fooled by clicking on the "ribs" (symmetry is the bane of registration).
Can you write Python code for it?