I am working on the file "10. 기초 확률론3 - 확률 분포 모형/04. 가우시안 정규 분포 (파이썬 버전).ipynb".
The first task is:
중심 극한 정리
실세계에서 발생하는 현상 중 많은 것들이 정규 분포로 모형화 가능하다. 그 이유 중의 하나는 다음과 같은 중심 극한 정리(Central Limit Theorem)이다. 중심 극한 정리는 어떤 분포를 따르는 확류 변수든 간에 해당 확률 변수가 복수인 경우 그 합은 정규 분포와 비슥한 분포를 이루는 현상을 말한다.
좀 더 수학적인 용어로 쓰면 다음과 같다.
$X_1, X_2, \ldots, X_n$가 기댓값이 $\mu$이고 분산이 $\sigma^2$으로 동일한 분포이며 서로 독립인 확률 변수들이라고 하자.
이 값들의 합
$$ S_n = X_1+\cdots+X_n $$
도 마찬가지로 확률 변수이다. 이 확률 변수 $S_n$의 분포는 $n$이 증가할 수록 다음과 같은 정규 분포에 수렴한다.
$$ \dfrac{S_n}{\sqrt{n}} \xrightarrow{d}\ N(\mu,\;\sigma^2) $$
시뮬레이션을 사용하여 중심 극한 정리가 성립하는지 살펴보도록 하자.
Can you write Python code for it?

xx = np.linspace(-2, 2, 100)
plt.figure(figsize=(6,9))
for i, N in enumerate([1, 2, 10]):
    X = np.random.rand(1000, N) - 0.5
    S = X.sum(axis=1)/np.sqrt(N)
    plt.subplot(3, 2, 2*i+1)
    sns.distplot(S, bins=10, kde=False, norm_hist=True)
    plt.xlim(-2, 2)
    plt.yticks([])
    plt.subplot(3, 2, 2*i+2)
    sp.stats.probplot(S, plot=plt)

plt.tight_layout()
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/12. 디리클레 분포.ipynb".
The first task is:
$\alpha$가 $(1,1,1)$이 아닌 경우에는 다음과 같이 특정 위치에 분포가 집중되도록 할 수 있다. 이 특성을 이용하면 다항 분포의 모수를 추정하는 베이지안 추정 문제에 응용할 수 있다.
Can you write Python code for it?

def project(x):
    n1 = np.array([1, 0, 0])
    n2 = np.array([0, 1, 0])
    n3 = np.array([0, 0, 1])
    n12 = (n1 + n2)/2
    m1 = np.array([1, -1, 0])
    m2 = n3 - n12
    m1 = m1/np.linalg.norm(m1)
    m2 = m2/np.linalg.norm(m2)
    return np.dstack([(x-n12).dot(m1), (x-n12).dot(m2)])[0]

def project_reverse(x):
    n1 = np.array([1, 0, 0])
    n2 = np.array([0, 1, 0])
    n3 = np.array([0, 0, 1])
    n12 = (n1 + n2)/2
    m1 = np.array([1, -1, 0])
    m2 = n3 - n12
    m1 = m1/np.linalg.norm(m1)
    m2 = m2/np.linalg.norm(m2)
    return x[:,0][:, np.newaxis] * m1 + x[:,1][:, np.newaxis] * m2 + n12

eps = np.finfo(float).eps * 10
X = project([[1-eps,0,0], [0,1-eps,0], [0,0,1-eps]])

import matplotlib.tri as mtri
triang = mtri.Triangulation(X[:,0], X[:,1], [[0, 1, 2]])
refiner = mtri.UniformTriRefiner(triang)
triang2 = refiner.refine_triangulation(subdiv=6)
XYZ = project_reverse(np.dstack([triang2.x, triang2.y, 1-triang2.x-triang2.y])[0])

pdf = sp.stats.dirichlet((1,1,1)).pdf(XYZ.T)
plt.tricontourf(triang2, pdf)
plt.axis("equal")
plt.show()

pdf = sp.stats.dirichlet((3,4,2)).pdf(XYZ.T)
plt.tricontourf(triang2, pdf)
plt.axis("equal")
plt.show()

pdf = sp.stats.dirichlet((16,24,14)).pdf(XYZ.T)
plt.tricontourf(triang2, pdf)
plt.axis("equal")
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/12. 디리클레 분포.ipynb".
The first task is:
다음 함수는 생성된 점들을 2차원 삼각형 위에서 볼 수 있도록 그려주는 함수이다.
Can you write Python code for it?

def plot_triangle(X, kind):
    n1 = np.array([1, 0, 0])
    n2 = np.array([0, 1, 0])
    n3 = np.array([0, 0, 1])
    n12 = (n1 + n2)/2
    m1 = np.array([1, -1, 0])
    m2 = n3 - n12
    m1 = m1/np.linalg.norm(m1)
    m2 = m2/np.linalg.norm(m2)

    X1 = (X-n12).dot(m1)
    X2 = (X-n12).dot(m2)

    g = sns.jointplot(X1, X2, kind=kind, xlim=(-0.8,0.8), ylim=(-0.45,0.9))
    g.ax_joint.axis("equal")
    plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/12. 디리클레 분포.ipynb".
The first task is:
만약 이 문제를 단순하게 생각하여 서로 독립인 0과 1사이의 유니폼 확률 변수를 3개 생성하고 이들의 합이 1이 되도록 크기를 정규화(normalize)하면 다음 그림과 같이 삼각형의 중앙 근처에 많은 확률 분포가 집중된다. 즉, 확률 변수가 골고루 분포되지 않는다.
Can you write Python code for it?

X1 = np.random.rand(1000, 3)
X1 = X1/X1.sum(axis=1)[:, np.newaxis]
plot_triangle(X1, kind="scatter")

plot_triangle(X1, kind="hex")


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/13. 다변수 가우시안 정규 분포.ipynb".
The first task is:
경우 2
만약
$$
\mu = \begin{bmatrix}2 \ 3 \end{bmatrix}. \;\;\;
\Sigma = \begin{bmatrix}2 & 3 \ 3 & 7 \end{bmatrix}
$$
이면
$$
;   \Sigma ;  = 5,\;\;\;
\Sigma^{-1} = \begin{bmatrix}1.4 & -0.6 \ -0.6 & 0.4 \end{bmatrix}
$$
$$
(x-\mu)^T \Sigma^{-1} (x-\mu) =
\begin{bmatrix}x_1 - 2 & x_2 - 3 \end{bmatrix}
\begin{bmatrix}1.4 & -0.6 \ -0.6 & 0.4\end{bmatrix}
\begin{bmatrix}x_1 - 2 \ x_2 - 3 \end{bmatrix}
=
\dfrac{1}{10}\left(14(x_1 - 2)^2 - 12(x_1 - 2)(x_2 - 3) + 4(x_2 - 3)^2\right)
$$
$$
\mathcal{N}(x_1, x_2) = \dfrac{1}{20\pi}
\exp \left( -\dfrac{1}{10}\left(7(x_1 - 2)^2 - 6(x_1 - 2)(x_2 - 3) + 2(x_2 - 3)^2\right) \right)
$$
이 확률 밀도 함수의 모양은 다음과 같다.
Can you write Python code for it?

mu = [2, 3]
cov = [[2, 3],[3, 7]]
rv = sp.stats.multivariate_normal(mu, cov)
xx = np.linspace(0, 4, 120)
yy = np.linspace(1, 5, 150)
XX, YY = np.meshgrid(xx, yy)
plt.grid(False)
plt.contourf(XX, YY, rv.pdf(np.dstack([XX, YY])))
plt.axis("equal")
plt.show()


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
2. Assigning X and Y. No resampling.
3. Resampling.

As we mentioned earlier, there are several ways to resample skewed data. Apart from under and over sampling, there is a very popular approach called SMOTE (Synthetic Minority Over-Sampling Technique), which is a combination of oversampling and undersampling, but the oversampling approach is not by replicating minority class but constructing new minority class data instance via an algorithm.

In this notebook, we will use traditional UNDER-sampling. I will probably try to implement SMOTE in future versions of the code, but for now I will use traditional undersamplig.

The way we will under sample the dataset will be by creating a 50/50 ratio. This will be done by randomly selecting "x" amount of sample from the majority class, being "x" the total number of records with the minority class.
Can you write Python code for it?

X = data.ix[:, data.columns != 'Class']
y = data.ix[:, data.columns == 'Class']

# Number of data points in the minority class
number_records_fraud = len(data[data.Class == 1])
fraud_indices = np.array(data[data.Class == 1].index)

# Picking the indices of the normal classes
normal_indices = data[data.Class == 0].index

# Out of the indices we picked, randomly select "x" number (number_records_fraud)
random_normal_indices = np.random.choice(normal_indices, number_records_fraud, replace = False)
random_normal_indices = np.array(random_normal_indices)

# Appending the 2 indices
under_sample_indices = np.concatenate([fraud_indices,random_normal_indices])

# Under sample dataset
under_sample_data = data.iloc[under_sample_indices,:]

X_undersample = under_sample_data.ix[:, under_sample_data.columns != 'Class']
y_undersample = under_sample_data.ix[:, under_sample_data.columns == 'Class']

# Showing ratio
print("Percentage of normal transactions: ", len(under_sample_data[under_sample_data.Class == 0])/len(under_sample_data))
print("Percentage of fraud transactions: ", len(under_sample_data[under_sample_data.Class == 1])/len(under_sample_data))
print("Total number of transactions in resampled data: ", len(under_sample_data))


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
An additional comment that would be interesting to do is to initialise multiple undersampled datasets and repeat the process in loop. Remember that, to create an undersample data, we randomly got records from the majority class. Even though this is a valid technique, is doesn't represent the real population, so it would be interesting to repeat the process with different undersample configurations and check if the previous chosen parameters are still the most effective. In the end, the idea is to use a wider random representation of the whole dataset and rely on the averaged best parameters.
Logistic regression classifier - Skewed data
Having tested our previous approach, I find really interesting to test the same process on the skewed data. Our intuition is that skewness will introduce issues difficult to capture, and therefore, provide a less effective algorithm.

To be fair, taking into account the fact that the train and test datasets are substantially bigger than the undersampled ones, I believe a K-fold cross validation is necessary. I guess that by splitting the data with 60% in training set, 20% cross validation and 20% test should be enough... but let's take the same approach as before (no harm on this, it's just that K-fold is computationally more expensive)
Can you write Python code for it?

best_c = printing_Kfold_scores(X_train,y_train)

# Use this C_parameter to build the final model with the whole training dataset and predict the classes in the test
# dataset
lr = LogisticRegression(C = best_c, penalty = 'l1')
lr.fit(X_train,y_train.values.ravel())
y_pred_undersample = lr.predict(X_test.values)

# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test,y_pred_undersample)
np.set_printoptions(precision=2)

print("Recall metric in the testing dataset: ", cnf_matrix[1,1]/(cnf_matrix[1,0]+cnf_matrix[1,1]))

# Plot non-normalized confusion matrix
class_names = [0,1]
plt.figure()
plot_confusion_matrix(cnf_matrix
                      , classes=class_names
                      , title='Confusion matrix')
plt.show()


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Approach

We are not going to perform feature engineering in first instance. The dataset has been downgraded in order to contain 30 features (28 anonamised + time + amount).
We will then compare what happens when using resampling and when not using it. We will test this approach using a simple logistic regression classifier.
We will evaluate the models by using some of the performance metrics mentioned above.
We will repeat the best resampling/not resampling method, by tuning the parameters in the logistic regression classifier.
We will finally perform classifications model using other classification algorithms.

Setting our input and target variables + resampling.
1. Normalising the amount column. The amount column is not in line with the anonimised features.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

data['normAmount'] = StandardScaler().fit_transform(data['Amount'].reshape(-1, 1))
data = data.drop(['Time','Amount'],axis=1)
data.head()


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Before continuing... changing classification threshold.
We have seen that by undersampling the data, our algorithm does a much better job at detecting fraud. I wanted also to show how can we tweak our final classification by changing the thresold.

Initially, you build the classification model and then you predict unseen data using it.
We previously used the "predict()" method to decided whether a record should belong to "1" or "0".
There is another method "predict_proba()".
This method returns the probabilities for each class. The idea is that by changing the threshold to assign a record to class 1, we can control precision and recall.

Let's check this using the undersampled data (best C_param = 0.01)
Can you write Python code for it?

lr = LogisticRegression(C = 0.01, penalty = 'l1')
lr.fit(X_train_undersample,y_train_undersample.values.ravel())
y_pred_undersample_proba = lr.predict_proba(X_test_undersample.values)

thresholds = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]

plt.figure(figsize=(10,10))

j = 1
for i in thresholds:
    y_test_predictions_high_recall = y_pred_undersample_proba[:,1] > i

    plt.subplot(3,3,j)
    j += 1

    # Compute confusion matrix
    cnf_matrix = confusion_matrix(y_test_undersample,y_test_predictions_high_recall)
    np.set_printoptions(precision=2)

    print("Recall metric in the testing dataset: ", cnf_matrix[1,1]/(cnf_matrix[1,0]+cnf_matrix[1,1]))

    # Plot non-normalized confusion matrix
    class_names = [0,1]
    plot_confusion_matrix(cnf_matrix
                          , classes=class_names
                          , title='Threshold >= %s'%i)


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Create a function to plot a fancy confusion matrix
Can you write Python code for it?

import itertools

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=0)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        #print("Normalized confusion matrix")
    else:
        1#print('Confusion matrix, without normalization')

    #print(cm)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Loading the dataset
Can you write Python code for it?

data = pd.read_csv("data/creditcard.csv")
data.head()


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Logistic regression classifier - Undersampled data
We are very interested in the recall score, because that is the metric that will help us try to capture the most fraudulent transactions. If you think how Accuracy, Precision and Recall work for a confusion matrix, recall would be the most interesting:

Accuracy = (TP+TN)/total
Precision = TP/(TP+FP)
Recall = TP/(TP+FN)

As we know, due to the imbalacing of the data, many observations could be predicted as False Negatives, being, that we predict a normal transaction, but it is in fact a fraudulent one. Recall captures this.

Obviously, trying to increase recall, tends to come with a decrease of precision. However, in our case, if we predict that a transaction is fraudulent and turns out not to be, is not a massive problem compared to the opposite.
We could even apply a cost function when having FN and FP with different weights for each type of error, but let's leave that aside for now.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import KFold, cross_val_score
from sklearn.metrics import confusion_matrix,precision_recall_curve,auc,roc_auc_score,roc_curve,recall_score,classification_report


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
No missing values, that makes things a little easier.
Let's see how time compares across fraudulent and normal transactions.
Can you write Python code for it?

print ("Fraud")
print (data.Time[data.Class == 1].describe())
print ()
print ("Normal")
print (data.Time[data.Class == 0].describe())

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(12,4))

bins = 50

ax1.hist(data.Time[data.Class == 1], bins = bins)
ax1.set_title('Fraud')

ax2.hist(data.Time[data.Class == 0], bins = bins)
ax2.set_title('Normal')

plt.xlabel('Time (in Seconds)')
plt.ylabel('Number of Transactions')
plt.show()


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
So, the model is offering an 93.2% recall accuracy on the generalised unseen data (test set). Not a bad percentage to be the first try. However, recall this is a 93.2% recall accuracy measure on the undersampled test set.
Being happy with this result, let's apply the model we fitted and test it on the whole data.
Can you write Python code for it?

# Use this C_parameter to build the final model with the whole training dataset and predict the classes in the test
# dataset
lr = LogisticRegression(C = best_c, penalty = 'l1')
lr.fit(X_train_undersample,y_train_undersample.values.ravel())
y_pred = lr.predict(X_test.values)

# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test,y_pred)
np.set_printoptions(precision=2)

print("Recall metric in the testing dataset: ", cnf_matrix[1,1]/(cnf_matrix[1,0]+cnf_matrix[1,1]))

# Plot non-normalized confusion matrix
class_names = [0,1]
plt.figure()
plot_confusion_matrix(cnf_matrix
                      , classes=class_names
                      , title='Confusion matrix')
plt.show()


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Splitting data into train and test set. Cross validation will be used when calculating accuracies.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

# Whole dataset
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.3, random_state = 0)

print("Number transactions train dataset: ", len(X_train))
print("Number transactions test dataset: ", len(X_test))
print("Total number of transactions: ", len(X_train)+len(X_test))

# Undersampled dataset
X_train_undersample, X_test_undersample, y_train_undersample, y_test_undersample = train_test_split(X_undersample
                                                                                ,y_undersample
                                                                                ,test_size = 0.3
                                                                                ,random_state = 0)
print("")
print("Number transactions train dataset: ", len(X_train_undersample))
print("Number transactions test dataset: ", len(X_test_undersample))
print("Total number of transactions: ", len(X_train_undersample)+len(X_test_undersample))


I am working on the file "10.Fraud_Detection.ipynb".
The first task is:
Very ad-hoc function to print K_fold_scores
Can you write Python code for it?

def printing_Kfold_scores(x_train_data,y_train_data):
    fold = KFold(len(y_train_data),5,shuffle=False)

    # Different C parameters
    c_param_range = [0.01,0.1,1,10,100]

    results_table = pd.DataFrame(index = range(len(c_param_range),2), columns = ['C_parameter','Mean recall score'])
    results_table['C_parameter'] = c_param_range

    # the k-fold will give 2 lists: train_indices = indices[0], test_indices = indices[1]
    j = 0
    for c_param in c_param_range:
        print('-------------------------------------------')
        print('C parameter: ', c_param)
        print('-------------------------------------------')
        print('')

        recall_accs = []
        for iteration, indices in enumerate(fold,start=1):

            # Call the logistic regression model with a certain C parameter
            lr = LogisticRegression(C = c_param, penalty = 'l1')

            # Use the training data to fit the model. In this case, we use the portion of the fold to train the model
            # with indices[0]. We then predict on the portion assigned as the 'test cross validation' with indices[1]
            lr.fit(x_train_data.iloc[indices[0],:],y_train_data.iloc[indices[0],:].values.ravel())

            # Predict values using the test indices in the training data
            y_pred_undersample = lr.predict(x_train_data.iloc[indices[1],:].values)

            # Calculate the recall score and append it to a list for recall scores representing the current c_parameter
            recall_acc = recall_score(y_train_data.iloc[indices[1],:].values,y_pred_undersample)
            recall_accs.append(recall_acc)
            print('Iteration ', iteration,': recall score = ', recall_acc)

        # The mean value of those recall scores is the metric we want to save and get hold of.
        results_table.ix[j,'Mean recall score'] = np.mean(recall_accs)
        j += 1
        print('')
        print('Mean recall score ', np.mean(recall_accs))
        print('')

    best_c = results_table.loc[results_table['Mean...


I am working on the file "10.Visualise_This.ipynb".
The first task is:
Another Scatterplot Example
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
plt.rcParams["figure.figsize"] = (15,7)

x = np.random.randn(1, 500)
y = np.random.randn(1,500)
plt.scatter(x, y, color='b', s=50) # s = size of the point
plt.xlabel('X axis')
plt.ylabel('Y axis')
plt.title('Scatter Plot')


I am working on the file "10.Visualise_This.ipynb".
The first task is:
But this is too small. Let's specify the size of the plot. Note that you set it once at the very top, right after you import your libraries, or keep varying it every time you want to plot a graph.
Can you write Python code for it?

plt.rcParams["figure.figsize"] = (15,7)

# Plot the data
plt.plot(x, y, label='Sales')

# Add a legend
plt.legend()

# Add more information
plt.xlabel('Adwords Spending (ZIM $)')
plt.ylabel('Monthly Sales (Oranges)')
plt.title('Effect of Adwords Spending on Monthly Sales')


I am working on the file "10.Visualise_This.ipynb".
The first task is:
Histograms
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline
plt.rcParams["figure.figsize"] = (15,7)

Y = []
for x in range(0,1000000):
    Y.append(np.random.randn())

# Here 50 is the bin size. Try playing around with 10,100,200 etc and see how it effects the shape of the graph
plt.hist(Y, 500)
plt.title('Distribution of Random Numbers')


I am working on the file "10.Visualise_This.ipynb".
The first task is:
Scatterplots
Can you write Python code for it?

radius = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]
# We import the math library.
# This can also be done as from math import pi
# Then instead of math.pi, we simply use pi
import math
import matplotlib.pyplot as plt
% matplotlib inline
plt.rcParams["figure.figsize"] = (15,7)

# How awesome is list comprehension!!
area = [round((r**2)*math.pi,2) for r in radius]
print(area)

plt.xlabel('Radius')
plt.ylabel('Area')
plt.title('Radius of Circle v Area')
plt.scatter(radius, area, color='g', s=30)


I am working on the file "10.Visualise_This.ipynb".
The first task is:
So as you see, the convention plt can save you from typing chuck_norris every single time.
Back to business though. Let's reimport matplotlib.
Another Line Plot
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]
y = [1, 4, 9, 16, 25]
# We have two lists, or more in mathematical terms, arrays, x and y

plt.plot(x, y)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
10. Find indices of non-zero elements from [1,2,0,0,4,0] (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
100. Compute bootstrapped 95% confidence intervals for the mean of a 1D array X (i.e., resample the elements of an array with replacement N times, compute the mean of each sample, and then compute percentiles over the means). (★★★)
Can you write Python code for it?

# Author: Jessica B. Hamrick

X = np.random.randn(100) # random 1D array
N = 1000 # number of bootstrap samples
idx = np.random.randint(0, X.size, (N, X.size))
means = X[idx].mean(axis=1)
confint = np.percentile(means, [2.5, 97.5])
print(confint)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
11. Create a 3x3 identity matrix (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
12. Create a 3x3x3 array with random values (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
13. Create a 10x10 array with random values and find the minimum and maximum values (★☆☆)
Can you write Python code for it?

Z = np.random.random((10,10))
Zmin, Zmax = Z.min(), Z.max()
print(Zmin, Zmax)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
16. How to add a border (filled with 0's) around an existing array? (★☆☆)
Can you write Python code for it?

Z = np.ones((5,5))
Z = np.pad(Z, pad_width=1, mode='constant', constant_values=0)
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
17. What is the result of the following expression? (★☆☆)
Can you write Python code for it?

print(0 * np.nan)
print(np.nan == np.nan)
print(np.inf > np.nan)
print(np.nan - np.nan)
print(0.3 == 3 * 0.1)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
19. Create a 8x8 matrix and fill it with a checkerboard pattern (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
2. Print the numpy version and the configuration (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
20. Consider a (6,7,8) shape array, what is the index (x,y,z) of the 100th element?
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
21. Create a checkerboard 8x8 matrix using the tile function (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
22. Normalize a 5x5 random matrix (★☆☆)
Can you write Python code for it?

Z = np.random.random((5,5))
Zmax, Zmin = Z.max(), Z.min()
Z = (Z - Zmin)/(Zmax - Zmin)
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
23. Create a custom dtype that describes a color as four unisgned bytes (RGBA) (★☆☆)
Can you write Python code for it?

color = np.dtype([("r", np.ubyte, 1),
                  ("g", np.ubyte, 1),
                  ("b", np.ubyte, 1),
                  ("a", np.ubyte, 1)])


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
26. What is the output of the following script? (★☆☆)
Can you write Python code for it?

# Author: Jake VanderPlas

print(sum(range(5),-1))
from numpy import *
print(sum(range(5),-1))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
27. Consider an integer vector Z, which of these expressions are legal? (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
28. What are the result of the following expressions?
Can you write Python code for it?

print(np.array(0) / np.array(0))
print(np.array(0) // np.array(0))
print(np.array([np.nan]).astype(int).astype(float))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
29. How to round away from zero a float array ? (★☆☆)
Can you write Python code for it?

# Author: Charles R Harris

Z = np.random.uniform(-10,+10,10)
print (np.copysign(np.ceil(np.abs(Z)), Z))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
3. Create a null vector of size 10 (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
30. How to find common values between two arrays? (★☆☆)
Can you write Python code for it?

Z1 = np.random.randint(0,10,10)
Z2 = np.random.randint(0,10,10)
print(np.intersect1d(Z1,Z2))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
31. How to ignore all numpy warnings (not recommended)? (★☆☆)
Can you write Python code for it?

# Suicide mode on
defaults = np.seterr(all="ignore")
Z = np.ones(1)/0

# Back to sanity
_ = np.seterr(**defaults)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
32. Is the following expressions true? (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
34. How to get all the dates corresponding to the month of July 2016? (★★☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
35. How to compute ((A+B)*(-A/2)) in place (without copy)? (★★☆)
Can you write Python code for it?

A = np.ones(3)*1
B = np.ones(3)*2
C = np.ones(3)*3
np.add(A,B,out=B)
np.divide(A,2,out=A)
np.negative(A,out=A)
np.multiply(A,B,out=A)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
36. Extract the integer part of a random array using 5 different methods (★★☆)
Can you write Python code for it?

Z = np.random.uniform(0,10,10)

print (Z - Z%1)
print (np.floor(Z))
print (np.ceil(Z)-1)
print (Z.astype(int))
print (np.trunc(Z))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
37. Create a 5x5 matrix with row values ranging from 0 to 4 (★★☆)
Can you write Python code for it?

Z = np.zeros((5,5))
Z += np.arange(5)
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
4.  How to find the memory size of any array (★☆☆)
Can you write Python code for it?

Z = np.zeros((10,10))
print("%d bytes" % (Z.size * Z.itemsize))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
40. Create a random vector of size 10 and sort it (★★☆)
Can you write Python code for it?

Z = np.random.random(10)
Z.sort()
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
41. How to sum a small array faster than np.sum? (★★☆)
Can you write Python code for it?

# Author: Evgeni Burovski

Z = np.arange(10)
np.add.reduce(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
46. Create a structured array with x and y coordinates covering the [0,1]x[0,1] area (★★☆)
Can you write Python code for it?

Z = np.zeros((5,5), [('x',float),('y',float)])
Z['x'], Z['y'] = np.meshgrid(np.linspace(0,1,5),
                             np.linspace(0,1,5))
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
48. Print the minimum and maximum representable value for each numpy scalar type (★★☆)
Can you write Python code for it?

for dtype in [np.int8, np.int32, np.int64]:
   print(np.iinfo(dtype).min)
   print(np.iinfo(dtype).max)
for dtype in [np.float32, np.float64]:
   print(np.finfo(dtype).min)
   print(np.finfo(dtype).max)
   print(np.finfo(dtype).eps)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
5.  How to get the documentation of the numpy add function from the command line? (★☆☆)
Can you write Python code for it?

%run `python -c "import numpy; numpy.info(numpy.add)"`


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
51. Create a structured array representing a position (x,y) and a color (r,g,b) (★★☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
54. How to read the following file? (★★☆)
Can you write Python code for it?

from io import StringIO

# Fake file
s = StringIO("""1, 2, 3, 4, 5\n
                6,  ,  , 7, 8\n
                 ,  , 9,10,11\n""")
Z = np.genfromtxt(s, delimiter=",", dtype=np.int)
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
57. How to randomly place p elements in a 2D array? (★★☆)
Can you write Python code for it?

# Author: Divakar

n = 10
p = 3
Z = np.zeros((n,n))
np.put(Z, np.random.choice(range(n*n), p, replace=False),1)
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
59. How to I sort an array by the nth column? (★★☆)
Can you write Python code for it?

# Author: Steve Tjoa

Z = np.random.randint(0,10,(3,3))
print(Z)
print(Z[Z[:,1].argsort()])


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
6.  Create a null vector of size 10 but the fifth value which is 1 (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
60. How to tell if a given 2D array has null columns? (★★☆)
Can you write Python code for it?

# Author: Warren Weckesser

Z = np.random.randint(0,3,(3,10))
print((~Z.any(axis=0)).any())


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
61. Find the nearest value from a given value in an array (★★☆)
Can you write Python code for it?

Z = np.random.uniform(0,1,10)
z = 0.5
m = Z.flat[np.abs(Z - z).argmin()]
print(m)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
63. Create an array class that has a name attribute (★★☆)
Can you write Python code for it?

class NamedArray(np.ndarray):
    def __new__(cls, array, name="no name"):
        obj = np.asarray(array).view(cls)
        obj.name = name
        return obj
    def __array_finalize__(self, obj):
        if obj is None: return
        self.info = getattr(obj, 'name', "no name")

Z = NamedArray(np.arange(10), "range_10")
print (Z.name)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
64. Consider a given vector, how to add 1 to each element indexed by a second vector (be careful with repeated indices)? (★★★)
Can you write Python code for it?

# Author: Brett Olsen

Z = np.ones(10)
I = np.random.randint(0,len(Z),20)
Z += np.bincount(I, minlength=len(Z))
print(Z)

# Another solution
# Author: Bartosz Telenczuk
np.add.at(Z, I, 1)
print(Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
66. Considering a (w,h,3) image of (dtype=ubyte), compute the number of unique colors (★★★)
Can you write Python code for it?

# Author: Nadav Horesh

w,h = 16,16
I = np.random.randint(0,2,(h,w,3)).astype(np.ubyte)
F = I[...,0]*256*256 + I[...,1]*256 +I[...,2]
n = len(np.unique(F))
print(np.unique(I))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
68. Considering a one-dimensional vector D, how to compute means of subsets of D using a vector S of same size describing subset  indices? (★★★)
Can you write Python code for it?

# Author: Jaime Fernández del Río

D = np.random.uniform(0,1,100)
S = np.random.randint(0,10,100)
D_sums = np.bincount(S, weights=D)
D_counts = np.bincount(S)
D_means = D_sums / D_counts
print(D_means)

# Pandas solution as a reference due to more intuitive code
import pandas as pd
print(pd.Series(D).groupby(S).mean())


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
69. How to get the diagonal of a dot product? (★★★)
Can you write Python code for it?

# Author: Mathieu Blondel

A = np.random.uniform(0,1,(5,5))
B = np.random.uniform(0,1,(5,5))

# Slow version
np.diag(np.dot(A, B))

# Fast version
np.sum(A * B.T, axis=1)

# Faster version
np.einsum("ij,ji->i", A, B)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
70. Consider the vector [1, 2, 3, 4, 5], how to build a new vector with 3 consecutive zeros interleaved between each value? (★★★)
Can you write Python code for it?

# Author: Warren Weckesser

Z = np.array([1,2,3,4,5])
nz = 3
Z0 = np.zeros(len(Z) + (len(Z)-1)*(nz))
Z0[::nz+1] = Z
print(Z0)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
73. Consider a set of 10 triplets describing 10 triangles (with shared vertices), find the set of unique line segments composing all the  triangles (★★★)
Can you write Python code for it?

# Author: Nicolas P. Rougier

faces = np.random.randint(0,100,(10,3))
F = np.roll(faces.repeat(2,axis=1),-1,axis=1)
F = F.reshape(len(F)*3,2)
F = np.sort(F,axis=1)
G = F.view( dtype=[('p0',F.dtype),('p1',F.dtype)] )
G = np.unique(G)
print(G)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
75. How to compute averages using a sliding window over an array? (★★★)
Can you write Python code for it?

# Author: Jaime Fernández del Río

def moving_average(a, n=3) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret[n - 1:] / n
Z = np.arange(20)
print(moving_average(Z, n=3))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
77. How to negate a boolean, or to change the sign of a float inplace? (★★★)
Can you write Python code for it?

# Author: Nathaniel J. Smith

Z = np.random.randint(0,2,100)
np.logical_not(Z, out=Z)

Z = np.random.uniform(-1.0,1.0,100)
np.negative(Z, out=Z)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
78. Consider 2 sets of points P0,P1 describing lines (2d) and a point p, how to compute distance from p to each line i  (P0[i],P1[i])? (★★★)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
79. Consider 2 sets of points P0,P1 describing lines (2d) and a set of points P, how to compute distance from each point j (P[j]) to each line i (P0[i],P1[i])? (★★★)
Can you write Python code for it?

# Author: Italmassov Kuanysh

# based on distance function from previous question
P0 = np.random.uniform(-10, 10, (10,2))
P1 = np.random.uniform(-10,10,(10,2))
p = np.random.uniform(-10, 10, (10,2))
print(np.array([distance(P0,P1,p_i) for p_i in p]))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
8.  Reverse a vector (first element becomes last) (★☆☆)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
81. Consider an array Z = [1,2,3,4,5,6,7,8,9,10,11,12,13,14], how to generate an array R = [[1,2,3,4], [2,3,4,5], [3,4,5,6], ..., [11,12,13,14]]? (★★★)
Can you write Python code for it?

# Author: Stefan van der Walt

Z = np.arange(1,15,dtype=np.uint32)
R = stride_tricks.as_strided(Z,(11,4),(4,4))
print(R)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
82. Compute a matrix rank (★★★)
Can you write Python code for it?

# Author: Stefan van der Walt

Z = np.random.uniform(0,1,(10,10))
U, S, V = np.linalg.svd(Z) # Singular Value Decomposition
rank = np.sum(S > 1e-10)
print(rank)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
83. How to find the most frequent value in an array?
Can you write Python code for it?

Z = np.random.randint(0,10,50)
print(np.bincount(Z).argmax())


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
84. Extract all the contiguous 3x3 blocks from a random 10x10 matrix (★★★)
Can you write Python code for it?

# Author: Chris Barker

Z = np.random.randint(0,5,(10,10))
n = 3
i = 1 + (Z.shape[0]-3)
j = 1 + (Z.shape[1]-3)
C = stride_tricks.as_strided(Z, shape=(i, j, n, n), strides=Z.strides + Z.strides)
print(C)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
85. Create a 2D array subclass such that Z[i,j] == Z[j,i] (★★★)
Can you write Python code for it?

# Author: Eric O. Lebigot
# Note: only works for 2d array and value setting using indices

class Symetric(np.ndarray):
    def __setitem__(self, index, value):
        i,j = index
        super(Symetric, self).__setitem__((i,j), value)
        super(Symetric, self).__setitem__((j,i), value)

def symetric(Z):
    return np.asarray(Z + Z.T - np.diag(Z.diagonal())).view(Symetric)

S = symetric(np.random.randint(0,10,(5,5)))
S[2,3] = 42
print(S)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
87. Consider a 16x16 array, how to get the block-sum (block size is 4x4)? (★★★)
Can you write Python code for it?

# Author: Robert Kern

Z = np.ones((16,16))
k = 4
S = np.add.reduceat(np.add.reduceat(Z, np.arange(0, Z.shape[0], k), axis=0),
                                       np.arange(0, Z.shape[1], k), axis=1)
print(S)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
88. How to implement the Game of Life using numpy arrays? (★★★)
Can you write Python code for it?


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
89. How to get the n largest values of an array (★★★)
Can you write Python code for it?

Z = np.arange(10000)
np.random.shuffle(Z)
n = 5

# Slow
print (Z[np.argsort(Z)[-n:]])

# Fast
print (Z[np.argpartition(-Z,n)[:n]])


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
93. Consider two arrays A and B of shape (8,3) and (2,2). How to find rows of A that contain elements of each row of B regardless of the order of the elements in B? (★★★)
Can you write Python code for it?

# Author: Gabe Schwartz

A = np.random.randint(0,5,(8,3))
B = np.random.randint(0,5,(2,2))

C = (A[..., np.newaxis, np.newaxis] == B)
rows = np.where(C.any((3,1)).all(1))[0]
print(rows)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
94. Considering a 10x3 matrix, extract rows with unequal values (e.g. [2,2,3]) (★★★)
Can you write Python code for it?

# Author: Robert Kern

Z = np.random.randint(0,5,(10,3))
print(Z)
U = Z[~(Z[:,0,None]==Z[:,1:]).all(axis=1)]
print(U)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
95. Convert a vector of ints into a matrix binary representation (★★★)
Can you write Python code for it?

# Author: Warren Weckesser

I = np.array([0, 1, 2, 3, 15, 16, 32, 64, 128])
B = ((I.reshape(-1,1) & (2**np.arange(8))) != 0).astype(int)
print(B[:,::-1])

# Author: Daniel T. McDonald

I = np.array([0, 1, 2, 3, 15, 16, 32, 64, 128], dtype=np.uint8)
print(np.unpackbits(I[:, np.newaxis], axis=1))


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
96. Given a two dimensional array, how to extract unique rows? (★★★)
Can you write Python code for it?

# Author: Jaime Fernández del Río

Z = np.random.randint(0,2,(6,3))
T = np.ascontiguousarray(Z).view(np.dtype((np.void, Z.dtype.itemsize * Z.shape[1])))
_, idx = np.unique(T, return_index=True)
uZ = Z[idx]
print(uZ)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
97. Considering 2 vectors A & B, write the einsum equivalent of inner, outer, sum, and mul function (★★★)
Can you write Python code for it?

# Author: Alex Riley
# Make sure to read: http://ajcr.net/Basic-guide-to-einsum/

A = np.random.uniform(0,1,10)
B = np.random.uniform(0,1,10)

np.einsum('i->', A)       # np.sum(A)
np.einsum('i,i->i', A, B) # A * B
np.einsum('i,i', A, B)    # np.inner(A, B)
np.einsum('i,j->ij', A, B)    # np.outer(A, B)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
98. Considering a path described by two vectors (X,Y), how to sample it using equidistant samples (★★★)?
Can you write Python code for it?

# Author: Bas Swinckels

phi = np.arange(0, 10*np.pi, 0.1)
a = 1
x = a*phi*np.cos(phi)
y = a*phi*np.sin(phi)

dr = (np.diff(x)**2 + np.diff(y)**2)**.5 # segment lengths
r = np.zeros_like(x)
r[1:] = np.cumsum(dr)                # integrate path
r_int = np.linspace(0, r.max(), 200) # regular spaced path
x_int = np.interp(r_int, r, x)       # integrate path
y_int = np.interp(r_int, r, y)


I am working on the file "100 Numpy exercises.ipynb".
The first task is:
99. Given an integer n and a 2D array X, select from X the rows which can be interpreted as draws from a multinomial distribution with n degrees, i.e., the rows which only contain integers and which sum to n. (★★★)
Can you write Python code for it?


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
$\newcommand{\bPhi}{\mathbf{\Phi}}$
$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bm}{\mathbf{m}}$
$\newcommand{\bS}{\mathbf{S}}$
$\newcommand{\bI}{\mathbf{I}}$
Part 1: Polynomial Regression
1.1. Generate periodic data (5 points)
Write a method gen_cosine(N) that generates toy data like in fig 1.2 of Bishop's book. The method should have a parameter $N$, and should return $N$-dimensional vectors $\bx$ and $\bt$, where $\bx$ contains evenly spaced values from 0 to (including) 2$\pi$, and the elements $t_i$ of $\bt$ are distributed according to:
$$t_i \sim \mathcal{N}(\mu_i, \sigma^2)$$
where $x_i$ is the $i$-th elements of $\bf{x}$, the mean $\mu_i = \cos(x_i)$ and the standard deviation $\sigma = 0.2$.
Can you write Python code for it?

def gen_cosine(n):

    x = np.array(np.linspace(0, 2 * np.pi, n))
    t = np.random.normal(np.cos(x), 0.2)

    return x, t

def gen_real_cosine(n):
    real_x = np.array(np.linspace(0, 2 * np.pi, n))
    real_y = np.cos(real_x)
    return real_x,real_y

### Test your function
np.random.seed(5)
N = 10
x, t = gen_cosine(N)

assert x.shape == (N,), "the shape of x is incorrect"
assert t.shape == (N,), "the shape of t is incorrect"


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
1.2 Polynomial regression (10 points)
Write a method fit_polynomial(x, t, M) that finds the maximum-likelihood solution of an unregularized $M$-th order polynomial for some dataset x. The error function to minimize w.r.t. $\bw$ is:
$E(\bw) = \frac{1}{2} (\bPhi\bw - \bt)^T(\bPhi\bw - \bt)$
where $\bPhi$ is the feature matrix (or design matrix) as explained in Bishop's book at section 3.1.1, $\bt$ is the vector of target values. Your method should return a vector $\bw$ with the maximum-likelihood parameter estimates, as well as the feature matrix $\bPhi$.
Can you write Python code for it?

def designmatrix(x, M): # it is highly recommended to write a helper function that computes Phi
    design_matrix = []
    for i in range(M+1):
         design_matrix.append([data ** i for data in x])
    design_matrix = np.matrix(design_matrix).transpose()
    return design_matrix

def LSE(phi, t):
    phi_squared_inv = np.linalg.inv(np.matmul(phi.transpose(), phi))
    mp_pseudo_inv = np.matmul(phi_squared_inv, phi.transpose())
    return np.reshape(np.array(np.matmul(mp_pseudo_inv, t)), phi.shape[1])

def fit_polynomial(x, t, M):
    Phi = designmatrix(x, M)
    w_ml = LSE(Phi, t)
    return w_ml, Phi

### Test your function
N = 10
x = np.square((np.linspace(-1, 1, N)))
t = 0.5*x + 1.5
m = 2
w, Phi = fit_polynomial(x,t,m)

assert w.shape == (m+1,), "The shape of w is incorrect"
assert Phi.shape == (N, m+1), "The shape of Phi is incorrect"


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
1.3 Plot (5 points)
Sample a dataset with $N=10$, and fit four polynomials with $M \in (0, 2, 4, 8)$.
For each value of $M$, plot the prediction function, along with the data and the original cosine function. The resulting figure should look similar to fig 1.4 of the Bishop's book. Note that you can use matplotlib's plt.pyplot(.) functionality for creating grids of figures.
Can you write Python code for it?

N = 10

x,t = gen_cosine(N)
real_x , real_y = gen_real_cosine(100)
w_ml = []
Phi = []
M = [0, 2, 4, 8]
for m in M:
    w,phi = fit_polynomial(x,t,m)
    w_ml.append(w)
    Phi.append(phi)

predictions = []
for i in range(len(M)):
    predictions.append(np.squeeze(np.asarray(np.matmul(Phi[i],w_ml[i].transpose()))))

f, ((ax1, ax2), (ax4, ax8)) = plt.subplots(2, 2, sharex='col', sharey='row')
ax1.scatter(x,t)
ax1.plot(real_x,real_y)
ax1.plot(x,predictions[0])
ax1.set_title("M = "+ str(M[0]))

ax2.scatter(x,t)
ax2.plot(real_x,real_y)
ax2.plot(x,predictions[1])
ax2.set_title("M = "+ str(M[1]))

ax4.scatter(x,t)
ax4.plot(real_x,real_y)
ax4.plot(x,predictions[2])
ax4.set_title("M = "+ str(M[2]))

ax8.scatter(x,t)
ax8.plot(real_x,real_y)
ax8.plot(x,predictions[3])
ax8.set_title("M = "+ str(M[3]))


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
1.7 Plot best cross-validated fit (5 points)
For some dataset with $N = 10$, plot the model with the optimal $M$ and $\lambda$ according to the cross-validation error, using the method you just wrote. In addition, the plot should show the dataset itself and the function that we try to approximate. Let the plot make clear which $M$ and $\lambda$ were found.
Can you write Python code for it?

N=20
x, t = gen_cosine(N)
real_x,real_y = gen_real_cosine(100)
M,lamb = find_best_m_and_lamb(x,t)
w, Phi = fit_polynomial(x,t,M)
res = np.matmul(Phi,w.transpose())
res = np.asarray(res).reshape(-1)
plt.plot(real_x,real_y)
plt.scatter(x,t)
plt.plot(x,res)
plt.figtext(0.6, 0.8, "M = "+ str(M),  size='xx-large')
plt.figtext(0.6, 0.77, "Lambda = exp("+ str(math.log(lamb))+" )", size='xx-large')
plt.show()


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
2.2 Compute Posterior (15 points)
You're going to implement a Bayesian linear regression model, and fit it to the periodic data. Your regression model has a zero-mean isotropic Gaussian prior over the parameters, governed by a single (scalar) precision parameter $\alpha$, i.e.:
$$p(\bw \;|\; \alpha) = \mathcal{N}(\bw \;|\; 0, \alpha^{-1} \bI)$$
The covariance and mean of the posterior are given by:
$$\bS_N= \left( \alpha \bI + \beta \bPhi^T \bPhi \right)^{-1} $$
$$\bm_N = \beta\; \bS_N \bPhi^T \bt$$
where $\alpha$ is the precision of the predictive distribution, and $\beta$ is the noise precision.
See MLPR chapter 3.3 for background.
Write a method fit_polynomial_bayes(x, t, M, alpha, beta) that returns the mean $\bm_N$ and covariance $\bS_N$ of the posterior for a $M$-th order polynomial. In addition it should return the design matrix $\bPhi$. The arguments x, t and M have the same meaning as in question 1.2.
Can you write Python code for it?

def bayesian_LR(phi, t, alpha, beta, M):

    phi_squared = np.matmul(phi.transpose(), phi)
    Alpha = np.multiply(alpha, np.identity(M + 1))
    cov = np.linalg.inv(np.add(Alpha, np.multiply(beta, phi_squared)))
    phi_t = np.matmul(phi.transpose(), t).transpose()
    mean = np.multiply(beta, np.matmul(cov, phi_t))
    mean = np.reshape(np.array(mean), M+1)

    return cov, mean

def fit_polynomial_bayes(x, t, M, alpha, beta):

    Phi = designmatrix(x, M)
    S, m = bayesian_LR(Phi, t, alpha, beta, M)

    return m, S, Phi

### Test your function
N = 10
x = np.linspace(-1, 1, N)
t = 0.5*np.square(x) + 1.5
M = 2
alpha = 0.5
beta = 25
m, S, Phi = fit_polynomial_bayes(x, t, M, alpha, beta)

assert m.shape == (M+1,), "the shape of m is incorrect"
assert S.shape == (M+1, M+1), "the shape of S is incorrect"
assert Phi.shape == (N, M+1), "the shape of Phi is incorrect"


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
2.4 Plot predictive distribution (10 points)
a) (5 points) Generate 10 datapoints with gen_cosine2(10). Compute the posterior mean and covariance for a Bayesian polynomial regression model with $M=4$, $\alpha=\frac{1}{2}$ and $\beta=\frac{1}{0.2^2}$.
Plot the Bayesian predictive distribution, where you plot (for $x$ between 0 and $2 \pi$) $t$'s predictive mean and a 1-sigma predictive variance using plt.fill_between(..., alpha=0.1) (the alpha argument induces transparency).
Include the datapoints in your plot.
Can you write Python code for it?

np.random.seed(101)
real_x, real_y = gen_real_cosine(100)
N = 10
M = 4
alpha = 0.5
x, t = gen_cosine2(N)
perm = x.argsort()
beta = 25
m, S, Phi = fit_polynomial_bayes(x, t, M, alpha, beta)

plt.plot(real_x, real_y)
plt.scatter(x[perm], t[perm])
mean_pt, var, Phi = predict_polynomial_bayes(x, m, S, beta)

plt.plot(x[perm], mean_pt[perm])
plt.fill_between(x[perm],mean_pt[perm] - var[perm],mean_pt[perm] + var[perm], alpha=0.1)
plt.show()


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
Lab 1: Linear Regression and Overfitting
Machine Learning 1, September 2017
Notes on implementation:

You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
Please write your answers right below the questions.
Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
Refer to last week's lab notes, i.e. http://docs.scipy.org/doc/, if you are unsure about what function to use. There are different correct ways to implement each problem!
For this lab, your regression solutions should be in closed form, i.e., should not perform iterative gradient-based optimization but find the exact optimum directly.
use the provided test boxes to check if your answers are correct
Can you write Python code for it?

%pylab inline
plt.rcParams["figure.figsize"] = [20,10]


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
Part 2: Bayesian Linear (Polynomial) Regression
2.1 Cosine 2 (5 points)
Write a function gen_cosine2(N) that behaves identically to gen_cosine(N) except that the generated values $x_i$ are not linearly spaced, but drawn from a uniform distribution between $0$ and $2 \pi$.
Can you write Python code for it?

def gen_cosine2(n):

    x = np.array(np.random.uniform(0, 2 * np.pi, n))
    t = np.random.normal(np.cos(x), 0.2)
    return x, t

### Test your function
np.random.seed(5)
N = 10
x, t = gen_cosine2(N)

assert x.shape == (N,), "the shape of x is incorrect"
assert t.shape == (N,), "the shape of t is incorrect"


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
b) (10 points) Now write a method find_best_m_and_lamb(x, t) that finds the best values for $M$ and $\lambda$. The method should return the best $M$ and $\lambda$. To get you started, here is a method you can use to generate indices of cross-validation folds.
Can you write Python code for it?

def kfold_indices(N, k):
    all_indices = np.arange(N,dtype=int)
    np.random.shuffle(all_indices)
    idx = [int(i) for i in np.floor(np.linspace(0,N,k+1))]
    train_folds = []
    valid_folds = []
    for fold in range(k):
        valid_indices = all_indices[idx[fold]:idx[fold+1]]
        valid_folds.append(valid_indices)
        train_folds.append(np.setdiff1d(all_indices, valid_indices))
    return train_folds, valid_folds

def find_best_m_and_lamb(x, t):
    folds = 5
    train_idx, valid_idx = kfold_indices(len(x),folds)
    M_best = 2
    lamb_best = 1
    err_best = 1000000000
    for m in range(11):
        for exponent in range(0,11):
            total_err = 0
            for i in range(folds):
                x_train = np.take(x, train_idx[i])
                t_train = np.take(t, train_idx[i])
                x_valid = np.take(x, valid_idx[i])
                t_valid = np.take(t, valid_idx[i])
                total_err =total_err + pred_error(x_train, x_valid, t_train, t_valid, m, math.exp(-exponent))
#               print(m,exponent,err)
            if total_err < err_best:
                err_best = total_err
                M_best = m
                lamb_best = math.exp(-exponent)
    return M_best, lamb_best

### If you want you can write your own test here

np.random.seed(9)
N=10
x, t = gen_cosine(N)
print(find_best_m_and_lamb(x,t))


I am working on the file "10158006_10185453_11896493_lab1.ipynb".
The first task is:
b) (5 points) For a second plot, draw 100 samples from the parameters' posterior distribution. Each of these samples is a certain choice of parameters for 4-th order polynomial regression.
Display each of these 100 polynomials.
Can you write Python code for it?

M = 4
alpha = 0.5
beta = 25
real_x, real_y = gen_real_cosine(100)
perm = x.argsort()
plt.plot(real_x, real_y)
for i in range(1, 100):
    m, S, Phi = fit_polynomial_bayes(x, t, M, alpha, i)
    mean, var, phi = predict_polynomial_bayes(x, m, S, i)
    plt.plot(x[perm], mean[perm])

plt.show()


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
1.1.3 Stochastic gradient descent (10 points)
Write a function sgd_iter(x_train, t_train, w, b) that performs one iteration of stochastic gradient descent (SGD), and returns the new weights. It should go through the trainingset once in randomized order, call logreg_gradient(x, t, w, b) for each datapoint to get the gradients, and update the parameters using a small learning rate of 1E-6. Note that in this case we're maximizing the likelihood function, so we should actually performing gradient ascent... For more information about SGD, see Bishop 5.2.4 or an online source (i.e. https://en.wikipedia.org/wiki/Stochastic_gradient_descent)
Can you write Python code for it?

def sgd_iter(x_train, t_train, W, b):

    # get vector with random indices and intialize vector for logpt
    l = len(t_train)
    w_old = W
    b_old = b
    rand_list = [i for i in range(l)]
    shuffle(rand_list)
    logp_train = [0 for i in range(l)]

    # get gradient
    logpt, grad_w, grad_b = logreg_gradient(x_train[rand_list[0]:rand_list[0]+1,:], t_train[rand_list[0]:rand_list[0]+1], W, b)
    logp_train[0] = np.asscalar(logpt)
    learning_rate = 1 * 10**(-6)
    w_new = w_old + learning_rate * grad_w
    b_new = b_old + learning_rate * grad_b

    # go through dataset in randomized order
    for i in range(1, l):
        logpt, grad_w, grad_b = logreg_gradient(x_train[rand_list[i]:rand_list[i] + 1,:], t_train[rand_list[i]:rand_list[i] + 1],
                                                w_new, b_new)
        w_new = w_new + learning_rate * grad_w
        b_new = b_new + learning_rate * grad_b
        logp_train[i] = np.asscalar(logpt)

    logp_train_mean = mean(logp_train)

    return logp_train_mean, w_new, b_new

# Sanity check:
np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)

logp_train, W, b = sgd_iter(x_train[:5], t_train[:5], w, b)


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
1.2. Train
1.2.1 Train (10 points)
Perform 10 SGD iterations through the trainingset. Plot (in one graph) the conditional log-probability of the trainingset and validation set after each iteration.
Can you write Python code for it?

iterations = 10
def test_sgd(x_train, t_train, w, b):
    iterations = 10
    w_old = w
    logp_iter = [0 for i in range(iterations)]
    logp_iter_valid = [0 for i in range(iterations)]
    w_valid = w_old
    b_valid = b
    for i in range(iterations):
        print('iteration: ', i)

        # training set
        logp_train, w, b = sgd_iter(x_train, t_train, w, b)
        logp_iter[i] = logp_train

        # validation est
        logp_valid_train, w_valid, b_valid = sgd_iter(x_valid, t_valid, w_valid, b_valid)
        logp_iter_valid[i] = logp_valid_train

    return logp_iter, logp_iter_valid, w, b, b_valid

np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)

logp_iter, logp_iter_valid, w, b, b_valid = test_sgd(x_train, t_train, w, b)
plt.scatter(range(iterations), logp_iter)
plt.scatter(range(iterations), logp_iter_valid)
plt.show()


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
1.2.2 Visualize weights (10 points)
Visualize the resulting parameters $\bW$ after a few iterations through the training set, by treating each column of $\bW$ as an image. If you want, you can use or edit the plot_digits(...) above.
Can you write Python code for it?

tars = [i for i in range(10)]
plot_digits(w.transpose(), num_cols=5, targets=tars)


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
2.3.1. Explain the weights (5 points)
In less than 80 words, explain how and why the weights of the hidden layer of the MLP differ from the logistic regression model, and relate this to the stronger performance of the MLP.
The weight are more noisy but they represent different structure than in case of lienar regression.
2.3.1. Less than 250 misclassifications on the test set (10 bonus points)
You receive an additional 10 bonus points if you manage to train a model with very high accuracy: at most 2.5% misclasified digits on the test set. Note that the test set contains 10000 digits, so you model should misclassify at most 250 digits. This should be achievable with a MLP model with one hidden layer. See results of various models at : http://yann.lecun.com/exdb/mnist/index.html. To reach such a low accuracy, you probably need to have a very high $L$ (many hidden units), probably $L > 200$, and apply a strong Gaussian prior on the weights. In this case you are allowed to use the validation set for training.
You are allowed to add additional layers, and use convolutional networks, although that is probably not required to reach 2.5% misclassifications.
Can you write Python code for it?

predict_test = np.zeros(len(t_test))
# Fill predict_test with the predicted targets from your model, don't cheat :-).
# YOUR CODE HERE
raise NotImplementedError()

assert predict_test.shape == t_test.shape
n_errors = np.sum(predict_test != t_test)
print('Test errors: %d' % n_errors)


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
MNIST consists of small 28 by 28 pixel images of written digits (0-9). We split the dataset into a training, validation and testing arrays. The variables x_train, x_valid and x_test are $N \times M$ matrices, where $N$ is the number of datapoints in the respective set, and $M = 28^2 = 784$ is the dimensionality of the data. The second set of variables t_train, t_valid and t_test contain the corresponding $N$-dimensional vector of integers, containing the true class labels.
Here's a visualisation of the first 8 digits of the trainingset:
Can you write Python code for it?

def plot_digits(data, num_cols, targets=None, shape=(28,28)):
    num_digits = data.shape[0]
    num_rows = int(num_digits/num_cols)
    for i in range(num_digits):
        plt.subplot(num_rows, num_cols, i+1)
        plt.imshow(data[i].reshape(shape), interpolation='none', cmap='Greys')
        if targets is not None:
            plt.title(int(targets[i]))
        plt.colorbar()
        plt.axis('off')
    plt.tight_layout()
    plt.show()

plot_digits(x_train[0:40000:5000], num_cols=4, targets=t_train[0:40000:5000])


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
Part 1. Multiclass logistic regression
Scenario: you have a friend with one big problem: she's completely blind. You decided to help her: she has a special smartphone for blind people, and you are going to develop a mobile phone app that can do machine vision using the mobile camera: converting a picture (from the camera) to the meaning of the image. You decide to start with an app that can read handwritten digits, i.e. convert an image of handwritten digits to text (e.g. it would enable her to read precious handwritten phone numbers).
A key building block for such an app would be a function predict_digit(x) that returns the digit class of an image patch $\bx$. Since hand-coding this function is highly non-trivial, you decide to solve this problem using machine learning, such that the internal parameters of this function are automatically learned using machine learning techniques.
The dataset you're going to use for this is the MNIST handwritten digits dataset (http://yann.lecun.com/exdb/mnist/). You can download the data with scikit learn, and load it as follows:
Can you write Python code for it?

from sklearn.datasets import fetch_mldata
# Fetch the data
mnist = fetch_mldata('MNIST original')
data, target = mnist.data, mnist.target.astype('int')
# Shuffle
indices = np.arange(len(data))
np.random.seed(123)
np.random.shuffle(indices)
data, target = data[indices].astype('float32'), target[indices]

# Normalize the data between 0.0 and 1.0:
data /= 255.

# Split
x_train, x_valid, x_test = data[:50000], data[50000:60000], data[60000: 70000]
t_train, t_valid, t_test = target[:50000], target[50000:60000], target[60000: 70000]


I am working on the file "10158006_10185453_11896493_lab2.ipynb".
The first task is:
Part 2. Multilayer perceptron
You discover that the predictions by the logistic regression classifier are not good enough for your application: the model is too simple. You want to increase the accuracy of your predictions by using a better model. For this purpose, you're going to use a multilayer perceptron (MLP), a simple kind of neural network. The perceptron wil have a single hidden layer $\bh$ with $L$ elements. The parameters of the model are $\bV$ (connections between input $\bx$ and hidden layer $\bh$), $\ba$ (the biases/intercepts of $\bh$), $\bW$ (connections between $\bh$ and $\log q$) and $\bb$ (the biases/intercepts of $\log q$.
The conditional probability of the class label $j$ is given by:
$\log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$
where $q_j$ are again the unnormalized probabilities per class, and $Z = \sum_j q_j$ is again the probability normalizing factor. Each $q_j$ is computed using:
$\log q_j = \bw_j^T \bh + b_j$
where $\bh$ is a $L \times 1$ vector with the hidden layer activations (of a hidden layer with size $L$), and $\bw_j$ is the $j$-th column of $\bW$ (a $L \times 10$ matrix). Each element of the hidden layer is computed from the input vector $\bx$ using:
$h_j = \sigma(\bv_j^T \bx + a_j)$
where $\bv_j$ is the $j$-th column of $\bV$ (a $784 \times L$ matrix), $a_j$ is the $j$-th element of $\ba$, and $\sigma(.)$ is the so-called sigmoid activation function, defined by:
$\sigma(x) = \frac{1}{1 + \exp(-x)}$
Note that this model is almost equal to the multiclass logistic regression model, but with an extra 'hidden layer' $\bh$. The activations of this hidden layer can be viewed as features computed from the input, where the feature transformation ($\bV$ and $\ba$) is learned.
2.1 Derive gradient equations (20 points)
State (shortly) why $\nabla_{\bb} \mathcal{L}^{(n)}$ is equal to the earlier (multiclass logistic regression) case, and why $\nabla_{\bw_j} \mathcal{L}^{(n)}$ is almost equal to the earlier case.
Like in multiclass logistic regression, you should use...
Can you write Python code for it?

import math

def sigmoid(z):
    return 1.0/(1.0+np.exp(np.negative(z)))

def sigmoid_prime(z):
    return sigmoid(z)*(1-sigmoid(z))

class Network(object):
    def  __init__(self,sizes):
        self.num_layers = len(sizes)-1
        self.sizes = sizes
        self.biases = [np.random.randn(1,j) for j in self.sizes[1:]]
        self.b_grads = [np.zeros((1,j)) for j in self.sizes[1:]]

        self.weights = [(np.random.randn(y, x)/np.sqrt(x)).transpose()
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]
        self.w_grads = [np.zeros((y, x)).transpose()
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]

        self.activations = [np.zeros((1,j)) for j in self.sizes]
#         print('V',self.weights[0].shape,'a',self.biases[0].shape,'activ:',self.activations[0].shape)
#         print('W',self.weights[1].shape,'b',self.biases[1].shape,'activ2:',self.activations[1].shape)

    def feedForward(self,x):
        self.activations[0] =  x.reshape((1,len(x)))
        for i,b,w in zip(range(1,self.num_layers),self.biases[:-1],self.weights[:-1]):
            self.activations[i]= sigmoid(self.activations[i-1].dot(w) + b)
        # last layer activation no activation function
        self.activations[-1] = self.activations[-2].dot(self.weights[-1]) + self.biases[-1]
        return self.activations[-1]

    def SGD(self,x,t,learn):
        idx = list(range(len(x_train)))
        random.shuffle(idx)
        log_train = 0
        for i in idx:
            self.feedForward(x[i,:])
            err = self.backprop(t[i])
            for i in range(self.num_layers):
                self.biases[i] += learn * self.b_grads[i]
                self.weights[i] += learn * self.w_grads[i]
            log_train +=err
        return log_train

    def backprop(self,t):
        max_activ = np.max(self.activations[-1])
        logZ = max_activ + np.log(np.sum(np.exp(self.activations[-1] - max_activ)))
        logp = self.activations[-1]...


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
1. Sampling from the Gaussian process prior (30 points)
We will implement Gaussian process regression using the kernel function in Bishop Eqn. 6.63.
1.1 k_n_m( xn, xm, thetas ) (5 points)
To start, implement function k_n_m(xn, xm, thetas) that takes scalars $x_n$ and $x_m$, and a vector of $4$ thetas, and computes the kernel function Bishop Eqn. 6.63 (10 points).  NB: usually the kernel function will take $D$ by $1$ vectors, but since we are using a univariate problem, this makes things easier.
Can you write Python code for it?

def k_n_m(xn, xm, thetas):
    dist = (xn - xm)**2
    k = thetas[0] * exp(-(thetas[1] / 2) * dist) + thetas[2] + thetas[3] * xn * xm

    return k


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
1.3 Plot function samples (15 points)
Now sample mean functions at the x_test locations for the theta values in Bishop Figure 6.5, make a figure with a 2 by 3 subplot and make sure the title reflects the theta values (make sure everything is legible).  In other words, sample $\by_i \sim \mathcal{N}(0, \mathbf{K}_{\theta})$.  Make use of numpy.random.multivariate_normal().  On your plots include the expected value of $\by$ with a dashed line and fill_between 2 standard deviations of the uncertainty due to $\mathbf{K}$ (the diagonal of $\mathbf{K}$ is the variance of the model uncertainty) (15 points).
Can you write Python code for it?

all_thetas = [[1, 4, 0, 0], [9, 4, 0, 0], [1, 64, 0, 0], [1,0.25,0, 0], [1, 4, 10, 0], [1, 4, 0, 5]]

N_test = 100

x_test = np.linspace(-1, 1, N_test)
mu_test = np.zeros(N_test)

n_lines = 5
y = []
cov_tests = []
st_devs = []
for thetas in all_thetas:
    cov_test = computeK(x_test, x_test, thetas)
    cov_tests.append(cov_test)

    st_dev = np.diag(cov_test)
    st_devs.append(st_dev)
    h = []
    for i in range(n_lines):
        h.append(np.random.multivariate_normal(mu_test, cov_test))
    y.append(h)

n = 2
m = 3
f, axarr = plt.subplots(n, m)
for i in range(n):
    for j in range(m):
        for k in range(n_lines):
            axarr[i, j].plot(x_test, y[j + i*3][k])
        axarr[i, j].plot(x_test, mu_test, "b--")
        axarr[i, j].fill_between(x_test, mu_test - np.sqrt(st_devs[j + i*3]), mu_test + np.sqrt(st_devs[j + i*3]), alpha=0.1)
        axarr[i, j].set_title("Values of theta: " + str(all_thetas[j + i*3]))


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
2. Predictive distribution (35 points)
So far we have sampled mean functions from the prior.  We can draw actual data $\bt$ two ways.  The first way is generatively, by first sampling $\by | \mathbf{K}$, then sampling $\bt | \by, \beta$ (Eqns 6.60 followed by 6.59).  The second way is to integrate over $\by$ (the mean draw) and directly sample $\bt | \mathbf{K}, \beta$ using Eqn 6.61.    This is the generative process for $\bt$.  Note that we have not specified a distribution over inputs $\bx$;  this is because Gaussian processes are conditional models.  Because of this we are free to generate locations $\bx$ when playing around with the GP; obviously a dataset will give us input-output pairs.
Once we have data, we are interested in the predictive distribution (note: the prior is the predictive distribution when there is no data).  Consider the joint distribution for $N+1$ targets, given by Eqn 6.64.  Its covariance matrix is composed of block components $C_N$, $\mathbf{k}$, and $c$.  The covariance matrix $C_N$ for $\bt_N$ is $C_N = \mathbf{K}_N + \beta^{-1}\mathbf{I}_N$.  We have just made explicit the size $N$ of the matrix; $N$ is the number of training points.  The kernel vector $\mathbf{k}$ is a $N$ by $1$ vector of kernel function evaluations between the training input data and the test input vector.  The scalar $c$ is a kernel evaluation at the test input.
2.1 gp_predictive_distribution(...) (10 points)
Write a function gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None) that computes  Eqns 6.66 and 6.67, except allow for an arbitrary number of test points (not just one) and now the kernel matrix is for training data.  By having C as an optional parameter, we can avoid computing it more than once (for this problem it is unimportant, but for real problems this is an issue).  The function should compute $\mathbf{C}$, $\mathbf{k}$, and return the mean, variance and $\mathbf{C}$.  Do not forget: the computeK function computes $\mathbf{K}$, not $\mathbf{C}$! (10 points)
Can you write Python code for it?

def gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None):

    K = computeK(x_train, x_train, theta)
    if not any(C):
        C = K + beta**(-1) * np.identity(len(x_train))
    k = computeK(x_train, x_test, theta)
    c = np.diag(computeK(x_test, x_test, theta) + beta**(-1))
    C_inv = np.linalg.inv(C)
    mean_test = k.T @ C_inv @ t_train
    var_test = c - k.T @ C_inv @ k

    return mean_test, var_test, C

### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2*train_x
test_N = 3
test_x = np.linspace(-1, 1, test_N)
theta = [1, 2, 3, 4]
beta = 25
test_mean, test_var, C = gp_predictive_distribution(train_x, train_t, test_x, theta, beta, C=None)

assert test_mean.shape == (test_N,), "the shape of mean is incorrect"
assert test_var.shape == (test_N, test_N), "the shape of var is incorrect"
assert C.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, _, C_out = gp_predictive_distribution(train_x, train_t, test_x, theta, beta, C=C_in)

assert np.allclose(C_in, C_out), "C is not reused!"


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
2.2 gp_log_likelihood(...) (10 points)
To learn the hyperparameters, we would need to compute the log-likelihood of the of the training data.  Implicitly, this is conditioned on the value setting for $\mathbf{\theta}$.  Write a function gp_log_likelihood(x_train, t_train, theta, C=None, invC=None, beta=None), where C and invC can be stored and reused. It should return the log-likelihood, C and invC  (10 points)
Can you write Python code for it?

def gp_log_likelihood(x_train, t_train, theta, beta, C=None, invC=None):
    K = computeK(x_train, x_train, theta)
    if not any(C):
        C = K + beta**(-1) * np.identity(len(x_train))
    if not any(invC):
        invC = np.linalg.inv(C)
    detC = np.linalg.det(C)
    lp = -1/2 * log(detC) - 1/2 * t_train.T @ invC @ t_train - len(t_train)/2 * log(2 * np.pi)

    return lp, C, invC

### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2 * train_x
theta = [1, 2, 3, 4]
beta = 25
lp, C, invC = gp_log_likelihood(train_x, train_t, theta, beta, C=None, invC=None)

assert lp < 0, "the log-likelihood should smaller than 0"
assert C.shape == (N, N), "the shape of var is incorrect"
assert invC.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, C_out, _ = gp_log_likelihood(train_x, train_t, theta, beta, C=C_in, invC=None)

assert np.allclose(C_in, C_out), "C is not reused!"

invC_in = np.array([[1.26260453, 0.15416407], [0.15416407, 1.26260453]])
_, _, invC_out = gp_log_likelihood(train_x, train_t, theta, beta, C=None, invC=invC_in)

assert np.allclose(invC_in, invC_out), "invC is not reused!"


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
2.3 Plot support vectors (5 points)
Now that we have obtained the lagrangian multipliers $\ba$, we use them to find our support vectors. Repeat the plot from 2.1, this time use a third color to indicate which samples are the support vectors.
Can you write Python code for it?

X,t = create_X_and_t(X1, X2)
a  = compute_multipliers(X, t)
idx = np.array([], dtype=int)
for i in range(len(a)):
    if a[i] > 0.00001:
         idx  = append(idx,i)
plt.scatter(X1[:,0], X1[:,1], color='Green')
plt.scatter(X2[:,0], X2[:,1], color='Blue')
plt.scatter(X[idx,0], X[idx,1],s=200, edgecolors='red', facecolors = 'none', lw=2)
plt.show()


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
2.3 Plotting (10 points)
Repeat the 6 plots above, but this time conditioned on the training points.  Use the periodic data generator to create 2 training points where x is sampled uniformly between $-1$ and $1$.  For these plots, feel free to use the provided function "gp_plot".  Make sure you put the parameters in the title and this time also the log-likelihood. Try to understand the two types of uncertainty!  If you do not use gp_plot(...), please add a fill between for the model and target noise. (10 points)
Can you write Python code for it?

def gp_plot( x_test, y_test, mean_test, var_test, x_train, t_train, theta, beta ):
    # x_test:
    # y_test:    the true function at x_test
    # mean_test: predictive mean at x_test
    # var_test:  predictive covariance at x_test
    # t_train:   the training values
    # theta:     the kernel parameters
    # beta:      the precision (known)

    # the reason for the manipulation is to allow plots separating model and data stddevs.
    std_total = np.sqrt(np.diag(var_test))       # includes all uncertainty, model and target noise
    std_model = np.sqrt(std_total**2 - 1.0/beta) # remove data noise to get model uncertainty in stddev
    std_combo = std_model + np.sqrt(1.0/beta)    # add stddev (note: not the same as full)

    plt.plot(x_test, y_test, 'b', lw=3)
    plt.plot(x_test, mean_test, 'k--', lw=2)
    plt.fill_between(x_test, mean_test+2*std_combo,mean_test-2*std_combo, color='k', alpha=0.25)
    plt.fill_between(x_test, mean_test+2*std_model,mean_test-2*std_model, color='r', alpha=0.25)
    plt.plot(x_train, t_train, 'ro', ms=10)

def plot_subplots(N):
    n = 2
    m = 3
    sigma = 0.2
    beta  = 1.0 / pow(sigma, 2)
    N_test = 100
    x_train = np.random.uniform(-1, 1, N)
    y_train = true_mean_function(x_train)
    t_train = add_noise(y_train, sigma)
    f, axarr = plt.subplots(n, m)
    x_test = np.linspace(-1, 1, N_test)
    mu_test = np.zeros(N_test)
    y_test = true_mean_function(x_test)
    for i in range(n):
        for j in range(m):
            mu_test, var_test, C = gp_predictive_distribution(x_train, t_train, x_test, all_thetas[j + i*3], beta, C=None)
            loglikeli, C, invC = gp_log_likelihood(x_train, t_train, all_thetas[j + i*3], beta, C=None, invC=None)
            axarr[i, j].set_title("Values of theta: " + str(all_thetas[j + i*3]))
            plt.subplot(331 + j + i*3)
            gp_plot(x_test, y_test, mu_test, var_test, x_train, t_train, all_thetas[j + i*3], beta)
            plt.title("Thetas: " + str(all_thetas[j + i*3]) + " Log...


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
2.4 More plotting (5 points)
Repeat the 6 plots above, but this time conditioned a new set of 10 training points. (5 points)
Can you write Python code for it?


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
2.4 Plot the decision boundary (10 Points)
The decision boundary is fully specified by a (usually very small) subset of training samples, the support vectors. Make use of
\begin{align}
\bw &= \sum_{n=1}^N a_n t_n \mathbf{\phi}(\bx_n)\
b &= \frac{1}{N_S}\sum_{n \in S} (t_n - \sum_{m \in S} a_m t_m k(\bx_n, \bx_m)),
\end{align}
where $S$ denotes the set of indices of the support vectors, to calculate the slope and intercept of the decision boundary. Generate a last plot that contains the two subsets, support vectors and decision boundary.
Can you write Python code for it?

def boundary(x, w, b, c):
    w  = w.reshape(-1,1)
    return (w[0] * x + b - c) / -w[1]

X,t = create_X_and_t(X1, X2)
a  = compute_multipliers(X, t)
idx = np.array([], dtype=int)
for i in range(len(a)):
    if a[i] > 0.00001:
         idx  = append(idx,i)

w = t[idx].reshape(1,-1)*a[idx].reshape(1,-1) @ X[idx]
sumall = 0
for n in idx:
    sum2 = 0
    for m in idx:
        sum2+=a[m] * t[m] * (X[n] @ X[m].T)
    sumall = sumall + ( t[n] - sum2)

b = (1/len(idx) * sumall)[0]

x = np.linspace(np.min(X[:,0]), np.max(X[:,0]))
y=boundary(x,w,b,0)
plt.scatter(X1[:,0], X1[:,1], color='Green')
plt.scatter(X2[:,0], X2[:,1], color='Blue')
plt.scatter(X[idx,0], X[idx,1],s=200, edgecolors='red', facecolors = 'none', lw=2)
plt.plot(x,boundary(x,w,b,0),'r')
plt.plot(x,boundary(x,w,b,-1),'r--')
plt.plot(x,boundary(x,w,b,1),'r--')
plt.show()


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
Next, we will rewrite the dual representation so that we can make use of computationally efficient vector-matrix multiplication. The objective becomes
\begin{align}
\min_{\ba} \frac{1}{2} \ba^T K' \ba - 1^T\ba,
\end{align}
subject to the constraints
\begin{align}
a_n &\geq 0, n=1,...,N,\
\bt^T\ba &= 0.
\end{align}
Where
\begin{align}
K'{nm} = t_n t_m k(\bx_n, \bx_m),
\end{align}
and in the special case of a linear kernel function,
\begin{align}
K'{nm} = t_n t_m k(\bx_n, \bx_m) = k(t_n \bx_n, t_m \bx_m).
\end{align}
To solve the quadratic programming problem we will use a python module called cvxopt. You first have to install the module in your virtual environment (you have to activate it first), using the following command:
conda install -c conda-forge cvxopt
The quadratic programming solver can be called as
cvxopt.solvers.qp(P, q[, G, h[, A, b[, solver[, initvals]]]])
This solves the following problem,
\begin{align}
\min_{\bx} \frac{1}{2} \bx^T P \bx + q^T\bx,
\end{align}
subject to the constraints,
\begin{align}
G\bx &\leq h,\
A\bx &= b.
\end{align}
All we need to do is to map our formulation to the cvxopt interface.
b) (10 points) Write a function compute_multipliers(X, t) that solves the quadratic programming problem using the cvxopt module and returns the lagrangian multiplier for every sample in the dataset.
Can you write Python code for it?

import cvxopt

def compute_multipliers(X, t):
    K = computeK(X * t.reshape(-1,1))
    P = cvxopt.matrix(K)
    A =  cvxopt.matrix(t.reshape(1,-1))
    b = cvxopt.matrix(np.zeros((1,1)))
    q = cvxopt.matrix(-1 *np.ones((len(t),1)))
    h = cvxopt.matrix(np.zeros((len(t),1)))
    G = cvxopt.matrix(-1 * np.identity(len(t)))
    sol = cvxopt.solvers.qp(P, q, G, h, A, b)
    a = np.array(sol['x'])
    return a

### Test your function
dim = 2
N_test = 3
X_test = np.arange(6).reshape((N_test, dim))
t_test = np.array([-1., 1., 1.])
a_test = compute_multipliers(X_test, t_test)

assert a_test.shape == (N_test, 1)


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
Part 1: Gaussian Processes
For part 1 we will be refer to Bishop sections 6.4.2 and 6.4.3. You may also want to refer to Rasmussen's Gaussian Process text which is available online at http://www.gaussianprocess.org/gpml/chapters/ and especially to the project found at https://www.automaticstatistician.com/index/ by Ghahramani for some intuition in GP.  To understand Gaussian processes, it is highly recommended understand how marginal, partitioned Gaussian distributions can be converted into conditional Gaussian distributions.  This is covered in Bishop 2.3 and summarized in Eqns 2.94-2.98.
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\ba}{\mathbf{a}}$
Periodic Data
We will use the same data generating function that we used previously for regression.
Can you write Python code for it?

def true_mean_function(x):
    return np.cos(2*pi*(x+1))

def add_noise(y, sigma):
    return y + sigma*np.random.randn(len(y))

def generate_t(x, sigma):
    return add_noise(true_mean_function(x), sigma)

sigma = 0.2
beta  = 1.0 / pow(sigma, 2)
N_test = 100

x_test = np.linspace(-1, 1, N_test)
mu_test = np.zeros(N_test)
y_test = true_mean_function(x_test)
t_test = add_noise(y_test, sigma)

plt.plot( x_test, y_test, 'b-', lw=2)
plt.plot( x_test, t_test, 'go')
plt.show()


I am working on the file "10158006_10185453_11896493_lab3.ipynb".
The first task is:
Part 2: Support Vector Machines (45 points)
As seen in Part 1: Gaussian Processes, one of the significant limitations of many such algorithms is that the kernel function $k(\bx_n , \bx_m)$ must be evaluated for all possible pairs $\bx_n$ and $\bx_m$ of training points, which can be computationally infeasible during training and can lead to excessive computation times when making predictions for new data points.
In Part 2: Support Vector Machines, we shall look at kernel-based algorithms that have sparse solutions, so that predictions for new inputs depend only on the kernel function evaluated at a subset of the training data points.
2.1 Generating a linearly separable dataset (15 points)
a) (5 points) First of all, we are going to create our own 2D toy dataset $X$. The dataset will consists of two i.i.d. subsets $X_1$ and $X_2$, each of the subsets will be sampled from a multivariate Gaussian distribution,
\begin{align}
X_1 \sim &\mathcal{N}(\mu_1, \Sigma_1)\
&\text{ and }\
X_2 \sim &\mathcal{N}(\mu_2, \Sigma_2).
\end{align}
In the following, $X_1$ will have $N_1=20$ samples and a mean $\mu_1=(1,1)$. $X_2$ will have $N_2=30$ samples and a mean $\mu_2=(3,3)$.
Plot the two subsets in one figure, choose two colors to indicate which sample belongs to which subset. In addition you should choose, $\Sigma_1$ and $\Sigma_2$ in a way that the two subsets become linearly separable. (Hint: Which form has the covariance matrix for a i.i.d. dataset?)
Can you write Python code for it?

np.random.seed(101)
cov1 = np.identity(2) /8
cov2 = np.identity(2) /5

X1 = np.random.multivariate_normal(mean=[1,1],cov=cov1,size=20)
X2 = np.random.multivariate_normal(mean=[3,3],cov=cov2,size=30)

plt.scatter(X1[:,0], X1[:,1])
plt.scatter(X2[:,0], X2[:,1])
plt.show()


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
DataFrame取数时，如果不设置字段名称，就会取所有字段 (与 SQL 的 * 等效)
WHERE
sql
SELECT *
FROM tips
WHERE time = 'Dinner'
LIMIT 5;
DataFrames过滤方法很多，最常用的是布尔索引（boolean indexing）
Can you write Python code for it?


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
FULL JOIN
pandas支持FULL JOINs, 绝大部分RDBMS都支持，HIVE支持，MySQL不支持，
sql
SELECT *
FROM df1
FULL OUTER JOIN df2
  ON df1.key = df2.key;
Can you write Python code for it?

pd.merge(df1, df2, on='key', how='outer')


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
INNER JOIN
sql
SELECT *
FROM df1
INNER JOIN df2
  ON df1.key = df2.key;
Can you write Python code for it?

# 默认是INNER JOIN
pd.merge(df1, df2, on='key')


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
JOIN
JOIN在pandas是join()或merge()方法。join()方法默认是按照DataFrames的索引值(indices)连接，两种方法都支持四种连接(LEFT, RIGHT, INNER, FULL)，也可以按字段连接。
Can you write Python code for it?

df1 = pd.DataFrame({'key': ['A', 'B', 'C', 'D'],
                     'value': np.random.randn(4)})

df2 = pd.DataFrame({'key': ['B', 'D', 'D', 'E'],
                    'value': np.random.randn(4)})


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
LEFT OUTER JOIN
sql
SELECT *
FROM df1
LEFT OUTER JOIN df2
  ON df1.key = df2.key;
Can you write Python code for it?

pd.merge(df1, df2, on='key', how='left')


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
SELECT
sql
SELECT total_bill, tip, smoker, time
FROM tips
LIMIT 5;
pandas在DataFrame直接取字段名称即可
Can you write Python code for it?

tips[['total_bill', 'tip', 'smoker', 'time']].head(5)


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
SQL的UNION与UNION ALL类似，不过UNION会剔除重复行
SELECT city, rank
FROM df1
UNION
SELECT city, rank
FROM df2;
/*
         city  rank
      Chicago     1
San Francisco     2
New York City     3
       Boston     4
  Los Angeles     5
*/
pandas需要用concat()+drop_duplicates()方法实现
Can you write Python code for it?


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
UNION
SQL的UNION ALL在pandas中是concat()
Can you write Python code for it?

df1 = pd.DataFrame({'city': ['Chicago', 'San Francisco', 'New York City'],
                     'rank': range(1, 4)})

df2 = pd.DataFrame({'city': ['Chicago', 'Boston', 'Los Angeles'],
                     'rank': [1, 4, 5]})


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
merge()也可以连接一个表的字段与另一个表的索引
Can you write Python code for it?

indexed_df2 = df2.set_index('key')

pd.merge(df1, indexed_df2, left_on='key', right_index=True)


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
sql
SELECT smoker, day, COUNT(*), AVG(tip)
FROM tips
GROUP BY smoker, day;
/*
smoker day
No     Fri      4  2.812500
       Sat     45  3.102889
       Sun     57  3.167895
       Thur    45  2.673778
Yes    Fri     15  2.714000
       Sat     42  2.875476
       Sun     19  3.516842
       Thur    17  3.030000
*/
Can you write Python code for it?

tips.groupby(['smoker', 'day']).agg({'tip': [np.size, np.mean]})


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
与SQL的OR/AND类似，DataFrame 用 | (OR) 与 & (AND)
sql
SELECT *
FROM tips
WHERE time = 'Dinner' AND tip > 5.00;
Can you write Python code for it?

tips[(tips['time'] == 'Dinner') & (tips['tip'] > 5.00)]


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
也可以对一个字段用count()方法，实现同样效果：
Can you write Python code for it?


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
找出每组消费低于2美金(tips < 2)的前两名(rank < 3)，这里用rank(method='min')
mysql
SELECT * FROM (
  SELECT
    t.*,
    RANK() OVER(PARTITION BY sex ORDER BY tip) AS rnk
  FROM tips t
  WHERE tip < 2
)
WHERE rnk < 3
ORDER BY sex, rnk;
Can you write Python code for it?

tips[tips['tip'] < 2].assign(rnk_min=tips.groupby(['sex'])['tip'].rank(
    method='min')).query('rnk_min < 3').sort_values(['sex', 'rnk_min'])


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
每组Top N 问题
mysql
SELECT * FROM (
  SELECT
    t.*,
    ROW_NUMBER() OVER(PARTITION BY day ORDER BY total_bill DESC) AS rn
  FROM tips t
)
WHERE rn < 3
ORDER BY day, rn;
Can you write Python code for it?

tips.assign(rn=tips.sort_values(['total_bill'], ascending=False).groupby(
    ['day']).cumcount() + 1).query('rn < 3').sort_values(['day', 'rn'])


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
用 notna() and isna() 方法检测NULL
Can you write Python code for it?

frame = pd.DataFrame({
    'col1': ['A', 'B', np.NaN, 'C', 'D'],
    'col2': ['F', np.NaN, 'G', 'H', 'I']
})

frame


I am working on the file "101basic/001_Pandas_vs_SQL.ipynb".
The first task is:
这行代码是向DataFrame传递一个True/False对象的Series，返回所有带True的行。
Can you write Python code for it?

is_dinner = tips['time'] == 'Dinner'

is_dinner.value_counts()

tips[is_dinner].head(5)


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
Part 2: Further Data Exploration
I do some slightly more sophisticated visualization with the data, using matplotlib and pandas.
Relevant Resources:

Pivot Tables Section from the Python Data Science Handbook
Can you write Python code for it?

data.resample('W').sum().plot();

data.resample('D').sum().rolling(365).sum().plot();

ax = data.resample('D').sum().rolling(365).sum().plot();
ax.set_ylim(0, None);

data.groupby(data.index.time).mean().plot();

pivoted = data.pivot_table('Total', index=data.index.time, columns=data.index.date)
pivoted.iloc[:5, :3]

pivoted.plot(legend=False, alpha=0.01);


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
Part 9: Further Data Exploration: PCA and GMM
Apply unsupervised learning techniques to the data to explore what we can learn from it
对一周不同时段（工作日、节假日和双休日）的骑车人数进行聚类分析
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-dark')

import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
from sklearn.mixture import GaussianMixture


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
下图是双休日与节假日的自行车车数
Can you write Python code for it?

plt.scatter(X2[:, 0], X2[:, 1], c=dayofweek, cmap='rainbow');
plt.colorbar();


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
主成分分析
Can you write Python code for it?

X = pivoted.fillna(0).T.values
X.shape

X2 = PCA(2, svd_solver='full').fit_transform(X)

X2.shape

plt.scatter(X2[:, 0], X2[:, 1]);


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
工作日 VS 节假日
Can you write Python code for it?

dayofweek = pd.DatetimeIndex(pivoted.columns).dayofweek
dayofweek


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
无监督分类
Can you write Python code for it?

gmm = GaussianMixture(2).fit(X)

labels = gmm.predict(X)
labels

plt.scatter(X2[:, 0], X2[:, 1], c=labels, cmap='rainbow');
plt.colorbar();

fig, ax = plt.subplots(1, 2, figsize=(14, 4))

pivoted.T[labels == 0].T.plot(legend=False, alpha=0.1, ax=ax[0]);
pivoted.T[labels == 1].T.plot(legend=False, alpha=0.1, ax=ax[1]);

ax[0].set_title('Pupple Cluster')
ax[1].set_title('Red Cluster');


I am working on the file "103bestpractice/JupyterWorkflow.ipynb".
The first task is:
获取数据
Can you write Python code for it?

from jupyterworkflow.data import get_fremont_data

data = get_fremont_data()
pivoted = data.pivot_table('Total', index=data.index.time, columns=data.index.date)
pivoted.plot(legend=False, alpha=0.01);


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
An epoch normally means one full processing of the training-set. But the data-generator that we created above, will produce batches of training-data for eternity. So we need to define the number of steps we want to run for each "epoch" and this number gets multiplied by the batch-size defined above. In this case we have 100 steps per epoch and a batch-size of 20, so the "epoch" consists of 2000 random images from the training-set. We run 20 such "epochs".
The reason these particular numbers were chosen, was because they seemed to be sufficient for training with this particular model and dataset, and it didn't take too much time, and resulted in 20 data-points (one for each "epoch") which can be plotted afterwards.
Can you write Python code for it?


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Because the data-generators will loop for eternity, we need to specify the number of steps to perform during evaluation and prediction on the test-set. Because our test-set contains 530 images and the batch-size is set to 20, the number of steps is 26.5 for one full processing of the test-set. This is why we need to reset the data-generator's counter in the example_errors() function above, so it always starts processing from the beginning of the test-set.
This is another slightly awkward aspect of the Keras API which could perhaps be improved.
Can you write Python code for it?

steps_test = generator_test.n / batch_size
steps_test


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
By default all the layers of the VGG16 model are trainable.
Can you write Python code for it?


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Class Weights
The Knifey-Spoony dataset is quite imbalanced because it has few images of forks, more images of knives, and many more images of spoons. This can cause a problem during training because the neural network will be shown many more examples of spoons than forks, so it might become better at recognizing spoons.
Here we use scikit-learn to calculate weights that will properly balance the dataset. These weights are applied to the gradient for each image in the batch during training, so as to scale their influence on the overall gradient for the batch.
Can you write Python code for it?

from sklearn.utils.class_weight import compute_class_weight

class_weight = compute_class_weight(class_weight='balanced',
                                    classes=np.unique(cls_train),
                                    y=cls_train)


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Dataset: Knifey-Spoony
The Knifey-Spoony dataset was introduced in Tutorial #09. It was generated from video-files by taking individual frames and converting them to images.
Can you write Python code for it?


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Download and extract the dataset if it hasn't already been done. It is about 22 MB.
Can you write Python code for it?


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Example Predictions
Here we will show a few examples of using the pre-trained VGG16 model for prediction.
We need a helper-function for loading and resizing an image so it can be input to the VGG16 model, as well as doing the actual prediction and showing the result.
Can you write Python code for it?

def predict(image_path):
    # Load and resize the image using PIL.
    img = PIL.Image.open(image_path)
    img_resized = img.resize(input_shape, PIL.Image.LANCZOS)

    # Plot the image.
    plt.imshow(img_resized)
    plt.show()

    # Convert the PIL image to a numpy-array with the proper shape.
    img_array = np.expand_dims(np.array(img_resized), axis=0)

    # Use the VGG16 model to make a prediction.
    # This outputs an array with 1000 numbers corresponding to
    # the classes of the ImageNet-dataset.
    pred = model.predict(img_array)

    # Decode the output of the VGG16 model.
    pred_decoded = decode_predictions(pred)[0]

    # Print the predictions.
    for code, name, score in pred_decoded:
        print("{0:>6.2%} : {1}".format(score, name))


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Fine-Tuning
In Transfer Learning the original pre-trained model is locked or frozen during training of the new classifier. This ensures that the weights of the original VGG16 model will not change. One advantage of this, is that the training of the new classifier will not propagate large gradients back through the VGG16 model that may either distort its weights or cause overfitting to the new dataset.
But once the new classifier has been trained we can try and gently fine-tune some of the deeper layers in the VGG16 model as well. We call this Fine-Tuning.
It is a bit unclear whether Keras uses the trainable boolean in each layer of the original VGG16 model or if it is overrided by the trainable boolean in the "meta-layer" we call conv_layer. So we will enable the trainable boolean for both conv_layer and all the relevant layers in the original VGG16 model.
Can you write Python code for it?


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Function for calculating the predicted classes of the entire test-set and calling the above function to plot a few examples of mis-classified images.
Can you write Python code for it?

def example_errors():
    # The Keras data-generator for the test-set must be reset
    # before processing. This is because the generator will loop
    # infinitely and keep an internal index into the dataset.
    # So it might start in the middle of the test-set if we do
    # not reset it first. This makes it impossible to match the
    # predicted classes with the input images.
    # If we reset the generator, then it always starts at the
    # beginning so we know exactly which input-images were used.
    generator_test.reset()

    # Predict the classes for all images in the test-set.
    y_pred = new_model.predict(generator_test, steps=steps_test)

    # Convert the predicted classes from arrays to integers.
    cls_pred = np.argmax(y_pred,axis=1)

    # Plot examples of mis-classified images.
    plot_example_errors(cls_pred)

    # Print the confusion matrix.
    print_confusion_matrix(cls_pred)


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Get the class-numbers for all the images in the training- and test-sets.
Can you write Python code for it?

cls_train = generator_train.classes
cls_test = generator_test.classes


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Get the file-paths for all the images in the training- and test-sets.
Can you write Python code for it?

image_paths_train = path_join(train_dir, generator_train.filenames)
image_paths_test = path_join(test_dir, generator_test.filenames)


I am working on the file "10_Fine-Tuning.ipynb".
The first task is:
Get the number of classes for the dataset.
Can you write Python code for it?

num_classes = generator_train.num_classes
num_classes


num_classes = generator_train.num_class
num_classes