I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
So our data have 1797 samples in 64 dimensions.
Unsupervised Learning: Dimensionality Reduction
We'd like to visualize our points within the 64-dimensional parameter space, but it's difficult to plot points in 64 dimensions!
Instead we'll reduce the dimensions to 2, using an unsupervised method.
Here, we'll make use of a manifold learning algorithm called Isomap, and transform the data to two dimensions.
Can you write Python code for it?

from sklearn.manifold import Isomap

iso = Isomap(n_components=2)
data_projected = iso.fit_transform(digits.data)

data_projected.shape

plt.scatter(data_projected[:, 0], data_projected[:, 1], c=digits.target,
            edgecolor='none', alpha=0.5, cmap=plt.cm.get_cmap('nipy_spectral', 10));
plt.colorbar(label='digit label', ticks=range(10))
plt.clim(-0.5, 9.5)


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
The Scikit-learn Estimator Object
Every algorithm is exposed in scikit-learn via an ''Estimator'' object. For instance a linear regression is implemented as so:
Can you write Python code for it?

from sklearn.linear_model import LinearRegression


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
The model found a line with a slope 2 and intercept 1, as we'd expect.
Supervised Learning: Classification and Regression
In Supervised Learning, we have a dataset consisting of both features and labels.
The task is to construct an estimator which is able to predict the label of an object
given the set of features. A relatively simple example is predicting the species of
iris given a set of measurements of its flower. This is a relatively simple task.
Some more complicated examples are:

given a multicolor image of an object through a telescope, determine
  whether that object is a star, a quasar, or a galaxy.
given a photograph of a person, identify the person in the photo.
given a list of movies a person has watched and their personal rating
  of the movie, recommend a list of movies they would like
  (So-called recommender systems: a famous example is the Netflix Prize).

What these tasks have in common is that there is one or more unknown
quantities associated with the object which needs to be determined from other
observed quantities.
Supervised learning is further broken down into two categories, classification and regression.
In classification, the label is discrete, while in regression, the label is continuous. For example,
in astronomy, the task of determining whether an object is a star, a galaxy, or a quasar is a
classification problem: the label is from three distinct categories. On the other hand, we might
wish to estimate the age of an object based on such observations: this would be a regression problem,
because the label (age) is a continuous quantity.
Classification Example
K nearest neighbors (kNN) is one of the simplest learning strategies: given a new, unknown observation, look up in your reference database which ones have the closest features and assign the predominant class.
Let's try it out on our iris classification problem:
Can you write Python code for it?

from sklearn import neighbors, datasets

iris = datasets.load_iris()
X, y = iris.data, iris.target

# create the model
knn = neighbors.KNeighborsClassifier(n_neighbors=5)

# fit the model
knn.fit(X, y)

# What kind of iris has 3cm x 5cm sepal and 4cm x 2cm petal?
# call the "predict" method:
result = knn.predict([[5, 3, 4, 2],])

print(iris.target_names[result])


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
This paints a better picture of the true performance of our classifier: apparently there is some confusion between the second and third species, which we might anticipate given what we've seen of the data above.
This is why it's extremely important to use a train/test split when evaluating your models.  We'll go into more depth on model evaluation later in this tutorial.
Flow Chart: How to Choose your Estimator
This is a flow chart created by scikit-learn super-contributor Andreas Mueller which gives a nice summary of which algorithms to choose in various situations. Keep it around as a handy reference!
Can you write Python code for it?

from IPython.display import Image
Image("http://scikit-learn.org/dev/_static/ml_map.png")


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
This single number doesn't tell us where we've gone wrong: one nice way to do this is to use the confusion matrix
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
print(confusion_matrix(ytest, ypred))

plt.imshow(np.log(confusion_matrix(ytest, ypred)),
           cmap='Blues', interpolation='nearest')
plt.grid(False)
plt.ylabel('true')
plt.xlabel('predicted');


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
We might also take a look at some of the outputs along with their predicted labels. We'll make the bad labels red:
Can you write Python code for it?

fig, axes = plt.subplots(10, 10, figsize=(8, 8))
fig.subplots_adjust(hspace=0.1, wspace=0.1)

for i, ax in enumerate(axes.flat):
    ax.imshow(Xtest[i].reshape(8, 8), cmap='binary')
    ax.text(0.05, 0.05, str(ypred[i]),
            transform=ax.transAxes,
            color='green' if (ytest[i] == ypred[i]) else 'red')
    ax.set_xticks([])
    ax.set_yticks([])


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
We see here that the digits are fairly well-separated in the parameter space; this tells us that a supervised classification algorithm should perform fairly well. Let's give it a try.
Classification on Digits
Let's try a classification task on the digits. The first thing we'll want to do is split the digits into a training and testing sample:
Can you write Python code for it?

from sklearn.model_selection import train_test_split
Xtrain, Xtest, ytrain, ytest = train_test_split(digits.data, digits.target,
                                                random_state=2)
print(Xtrain.shape, Xtest.shape)


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Whether either of these is a "good" fit or not depends on a number of things; we'll discuss details of how to choose a model later in the tutorial.

Exercise
Explore the RandomForestRegressor object using IPython's help features (i.e. put a question mark after the object).
What arguments are available to RandomForestRegressor?
How does the above plot change if you change these arguments?
These class-level arguments are known as hyperparameters, and we will discuss later how you to select hyperparameters in the model validation section.

Unsupervised Learning: Dimensionality Reduction and Clustering
Unsupervised Learning addresses a different sort of problem. Here the data has no labels,
and we are interested in finding similarities between the objects in question. In a sense,
you can think of unsupervised learning as a means of discovering labels from the data itself.
Unsupervised learning comprises tasks such as dimensionality reduction, clustering, and
density estimation. For example, in the iris data discussed above, we can used unsupervised
methods to determine combinations of the measurements which best display the structure of the
data. As we'll see below, such a projection of the data can be used to visualize the
four-dimensional dataset in two dimensions. Some more involved unsupervised learning problems are:

given detailed observations of distant galaxies, determine which features or combinations of
  features best summarize the information.
given a mixture of two sound sources (for example, a person talking over some music),
  separate the two (this is called the blind source separation problem).
given a video, isolate a moving object and categorize in relation to other moving objects which have been seen.

Sometimes the two may even be combined: e.g. Unsupervised learning can be used to find useful
features in heterogeneous data, and then these features can be used within a supervised
framework.
Dimensionality Reduction: PCA
Principle Component Analysis (PCA) is a dimension reduction technique...
Can you write Python code for it?

X, y = iris.data, iris.target

from sklearn.decomposition import PCA
pca = PCA(n_components=0.95)
pca.fit(X)
X_reduced = pca.transform(X)
print("Reduced dataset shape:", X_reduced.shape)

import pylab as plt
plt.scatter(X_reduced[:, 0], X_reduced[:, 1], #c=y,
           cmap='RdYlBu')

print("Meaning of the 2 components:")
for component in pca.components_:
    print(" + ".join("%.3f x %s" % (value, name)
                     for value, name in zip(component,
                                            iris.feature_names)))


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
You can also do probabilistic predictions:
Can you write Python code for it?

iris.target_names

knn.predict_proba([[5, 3, 4, 2],])

from fig_code import plot_iris_knn
plot_iris_knn()


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
As you can see, the non-parametric random forest model is flexible enough to fit the multi-period data, without us even specifying a multi-period model!
Example: Random Forest for Classifying Digits
We previously saw the hand-written digits data. Let's use that here to test the efficacy of the SVM and Random Forest classifiers.
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()
digits.keys()

X = digits.data
y = digits.target
print(X.shape)
print(y.shape)


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
Exercise

Repeat this classification task with sklearn.ensemble.RandomForestClassifier.  How does the max_depth, max_features, and n_estimators affect the results?
Try this classification with sklearn.svm.SVC, adjusting kernel, C, and gamma. Which classifier performs optimally?
Try a few sets of parameters for each model and check the F1 score (sklearn.metrics.f1_score) on your results. What's the best F1 score you can reach?
Can you write Python code for it?

clf = RandomForestClassifier(max_depth=100)
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)
metrics.accuracy_score(ypred, ytest)


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
Motivating Random Forests: Decision Trees
Random forests are an example of an ensemble learner built on decision trees.
For this reason we'll start by discussing decision trees themselves.
Decision trees are extremely intuitive ways to classify or label objects: you simply ask a series of questions designed to zero-in on the classification:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
Notice that at each increase in depth, every node is split in two except those nodes which contain only a single class.
The result is a very fast non-parametric classification, and can be extremely useful in practice.
Question: Do you see any problems with this?
Decision Trees and over-fitting
One issue with decision trees is that it is very easy to create trees which over-fit the data. That is, they are flexible enough that they can learn the structure of the noise in the data rather than the signal! For example, take a look at two trees built on two subsets of this dataset:
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier()

plt.figure()
visualize_tree(clf, X[:200], y[:200], boundaries=False)
plt.figure()
visualize_tree(clf, X[-200:], y[-200:], boundaries=False)


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
The binary splitting makes this extremely efficient.
As always, though, the trick is to ask the right questions.
This is where the algorithmic process comes in: in training a decision tree classifier, the algorithm looks at the features and decides which questions (or "splits") contain the most information.
Creating a Decision Tree
Here's an example of a decision tree classifier in scikit-learn. We'll start by defining some two-dimensional labeled data:
Can you write Python code for it?

from sklearn.datasets import make_blobs

X, y = make_blobs(n_samples=300, centers=4,
                  random_state=0, cluster_std=1.0)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='rainbow');


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
The details of the classifications are completely different! That is an indication of over-fitting: when you predict the value for a new point, the result is more reflective of the noise in the model rather than the signal.
Ensembles of Estimators: Random Forests
One possible way to address over-fitting is to use an Ensemble Method: this is a meta-estimator which essentially averages the results of many individual estimators which over-fit the data. Somewhat surprisingly, the resulting estimates are much more robust and accurate than the individual estimates which make them up!
One of the most common ensemble methods is the Random Forest, in which the ensemble is made up of many decision trees which are in some way perturbed.
There are volumes of theory and precedent about how to randomize these trees, but as an example, let's imagine an ensemble of estimators fit on subsets of the data. We can get an idea of what these might look like as follows:
Can you write Python code for it?

def fit_randomized_tree(random_state=0):
    X, y = make_blobs(n_samples=300, centers=4,
                      random_state=0, cluster_std=2.0)
    clf = DecisionTreeClassifier(max_depth=15)

    rng = np.random.RandomState(random_state)
    i = np.arange(len(y))
    rng.shuffle(i)
    visualize_tree(clf, X[i[:250]], y[i[:250]], boundaries=False,
                   xlim=(X[:, 0].min(), X[:, 0].max()),
                   ylim=(X[:, 1].min(), X[:, 1].max()))

from IPython.html.widgets import interact
interact(fit_randomized_tree, random_state=[0, 100]);


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
To remind us what we're looking at, we'll visualize the first few data points:
Can you write Python code for it?

# set up the figure
fig = plt.figure(figsize=(6, 6))  # figure size in inches
fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)

# plot the digits: each image is 8x8 pixels
for i in range(64):
    ax = fig.add_subplot(8, 8, i + 1, xticks=[], yticks=[])
    ax.imshow(digits.images[i], cmap=plt.cm.binary, interpolation='nearest')

    # label the image with the target value
    ax.text(0, 7, str(digits.target[i]))


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
We can quickly classify the digits using a decision tree as follows:
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from sklearn import metrics

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=0)
clf = DecisionTreeClassifier(max_depth=11)
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)


I am working on the file "Day_1_Scientific_Python/scikit-learn/03.2-Regression-Forests.ipynb".
The first task is:
and for good measure, plot the confusion matrix:
Can you write Python code for it?

plt.imshow(metrics.confusion_matrix(ypred, ytest),
           interpolation='nearest', cmap=plt.cm.binary)
plt.grid(False)
plt.colorbar()
plt.xlabel("predicted label")
plt.ylabel("true label");


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
But the pixel-wise representation is not the only choice. We can also use other basis functions, and write something like
$$
image(x) = {\rm mean} + x_1 \cdot{\rm (basis~1)} + x_2 \cdot{\rm (basis~2)} + x_3 \cdot{\rm (basis~3)} \cdots
$$
What PCA does is to choose optimal basis functions so that only a few are needed to get a reasonable approximation.
The low-dimensional representation of our data is the coefficients of this series, and the approximate reconstruction is the result of the sum:
Can you write Python code for it?

from fig_code.figures import plot_pca_interactive
plot_pca_interactive(digits.data)


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
By specifying that we want to throw away 5% of the variance, the data is now compressed by a factor of 50%! Let's see what the data look like after this compression:
Can you write Python code for it?

X_new = clf.inverse_transform(X_trans)
plt.plot(X[:, 0], X[:, 1], 'o', alpha=0.2)
plt.plot(X_new[:, 0], X_new[:, 1], 'ob', alpha=0.8)
plt.axis('equal');


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
Here we see that our two-dimensional projection loses a lot of information (as measured by the explained variance) and that we'd need about 20 components to retain 90% of the variance.  Looking at this plot for a high-dimensional dataset can help you understand the level of redundancy present in multiple observations.
PCA as data compression
As we mentioned, PCA can be used for is a sort of data compression. Using a small n_components allows you to represent a high dimensional point as a sum of just a few principal vectors.
Here's what a single digit looks like as you change the number of components:
Can you write Python code for it?

fig, axes = plt.subplots(8, 8, figsize=(8, 8))
fig.subplots_adjust(hspace=0.1, wspace=0.1)

for i, ax in enumerate(axes.flat):
    pca = PCA(i + 1).fit(X)
    im = pca.inverse_transform(pca.transform(X[20:21]))

    ax.imshow(im.reshape((8, 8)), cmap='binary')
    ax.text(0.95, 0.05, 'n = {0}'.format(i + 1), ha='right',
            transform=ax.transAxes, color='green')
    ax.set_xticks([])
    ax.set_yticks([])


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
Introducing Principal Component Analysis
Principal Component Analysis is a very powerful unsupervised method for dimensionality reduction in data.  It's easiest to visualize by looking at a two-dimensional dataset:
Can you write Python code for it?

np.random.seed(1)
X = np.dot(np.random.random(size=(2, 2)), np.random.normal(size=(2, 200))).T
plt.plot(X[:, 0], X[:, 1], 'o')
plt.axis('equal');


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
Let's take another look at this by using IPython's interact functionality to view the reconstruction of several images at once:
Can you write Python code for it?

from IPython.html.widgets import interact

def plot_digits(n_components):
    fig = plt.figure(figsize=(8, 8))
    plt.subplot(1, 1, 1, frameon=False, xticks=[], yticks=[])
    nside = 10

    pca = PCA(n_components).fit(X)
    Xproj = pca.inverse_transform(pca.transform(X[:nside ** 2]))
    Xproj = np.reshape(Xproj, (nside, nside, 8, 8))
    total_var = pca.explained_variance_ratio_.sum()

    im = np.vstack([np.hstack([Xproj[i, j] for j in range(nside)])
                    for i in range(nside)])
    plt.imshow(im)
    plt.grid(False)
    plt.title("n = {0}, variance = {1:.2f}".format(n_components, total_var),
                 size=18)
    plt.clim(0, 16)

interact(plot_digits, n_components=[1, 64], nside=[1, 8]);


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
Notice that one vector is longer than the other. In a sense, this tells us that that direction in the data is somehow more "important" than the other direction.
The explained variance quantifies this measure of "importance" in direction.
Another way to think of it is that the second principal component could be completely ignored without much loss of information! Let's see what our data look like if we only keep 95% of the variance:
Can you write Python code for it?

clf = PCA(0.95) # keep 95% of variance
X_trans = clf.fit_transform(X)
print(X.shape)
print(X_trans.shape)


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
The light points are the original data, while the dark points are the projected version.  We see that after truncating 5% of the variance of this dataset and then reprojecting it, the "most important" features of the data are maintained, and we've compressed the data by 50%!
This is the sense in which "dimensionality reduction" works: if you can approximate a data set in a lower dimension, you can often have an easier time visualizing it or fitting complicated models to the data.
Application of PCA to Digits
The dimensionality reduction might seem a bit abstract in two dimensions, but the projection and dimensionality reduction can be extremely useful when visualizing high-dimensional data.  Let's take a quick look at the application of PCA to the digits data we looked at before:
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()
X = digits.data
y = digits.target

pca = PCA(2)  # project from 64 to 2 dimensions
Xproj = pca.fit_transform(X)
print(X.shape)
print(Xproj.shape)

plt.scatter(Xproj[:, 0], Xproj[:, 1], c=y, edgecolor='none', alpha=0.5,
            cmap=plt.cm.get_cmap('nipy_spectral', 10))
plt.colorbar();


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
This gives us an idea of the relationship between the digits. Essentially, we have found the optimal stretch and rotation in 64-dimensional space that allows us to see the layout of the digits, without reference to the labels.
What do the Components Mean?
PCA is a very useful dimensionality reduction algorithm, because it has a very intuitive interpretation via eigenvectors.
The input data is represented as a vector: in the case of the digits, our data is
$$
x = [x_1, x_2, x_3 \cdots]
$$
but what this really means is
$$
image(x) = x_1 \cdot{\rm (pixel~1)} + x_2 \cdot{\rm (pixel~2)} + x_3 \cdot{\rm (pixel~3)} \cdots
$$
If we reduce the dimensionality in the pixel space to (say) 6, we recover only a partial image:
Can you write Python code for it?

from fig_code.figures import plot_image_components

sns.set_style('white')
plot_image_components(digits.data[0])


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.1-Dimensionality-PCA.ipynb".
The first task is:
To see what these numbers mean, let's view them as vectors plotted on top of the data:
Can you write Python code for it?

plt.plot(X[:, 0], X[:, 1], 'o', alpha=0.5)
for length, vector in zip(pca.explained_variance_, pca.components_):
    v = vector * 3 * np.sqrt(length)
    plt.plot([0, v[0]], [0, v[1]], '-k', lw=3)
plt.axis('equal');


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
80% – not bad! Let's check-out the confusion matrix for this:
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
print(confusion_matrix(digits.target, labels))

plt.imshow(confusion_matrix(digits.target, labels),
           cmap='Blues', interpolation='nearest')
plt.colorbar()
plt.grid(False)
plt.ylabel('true')
plt.xlabel('predicted');


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
Again, this is an 80% classification accuracy for an entirely unsupervised estimator which knew nothing about the labels.
Example: KMeans for Color Compression
One interesting application of clustering is in color image compression. For example, imagine you have an image with millions of colors. In most images, a large number of the colors will be unused, and conversely a large number of pixels will have similar or identical colors.
Scikit-learn has a number of images that you can play with, accessed through the datasets module. For example:
Can you write Python code for it?

from sklearn.datasets import load_sample_image
china = load_sample_image("china.jpg")
plt.imshow(china)
plt.grid(False);


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
For good measure, let's use our PCA visualization and look at the true cluster labels and K-means cluster labels:
Can you write Python code for it?

from sklearn.decomposition import PCA

X = PCA(2).fit_transform(digits.data)

kwargs = dict(cmap = plt.cm.get_cmap('rainbow', 10),
              edgecolor='none', alpha=0.6)
fig, ax = plt.subplots(1, 2, figsize=(8, 4))
ax[0].scatter(X[:, 0], X[:, 1], c=labels, **kwargs)
ax[0].set_title('learned cluster labels')

ax[1].scatter(X[:, 0], X[:, 1], c=digits.target, **kwargs)
ax[1].set_title('true labels');


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
Just for kicks, let's see how accurate our K-Means classifier is with no label information:
Can you write Python code for it?

from sklearn.metrics import accuracy_score
accuracy_score(digits.target, labels)


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
The algorithm identifies the four clusters of points in a manner very similar to what we would do by eye!
The K-Means Algorithm: Expectation Maximization
K-Means is an example of an algorithm which uses an Expectation-Maximization approach to arrive at the solution.
Expectation-Maximization is a two-step approach which works as follows:

Guess some cluster centers
Repeat until converged
Assign points to the nearest cluster center
Set the cluster centers to the mean

Let's quickly visualize this process:
Can you write Python code for it?

from fig_code import plot_kmeans_interactive
plot_kmeans_interactive();


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
This algorithm will (often) converge to the optimal cluster centers.
KMeans Caveats
The convergence of this algorithm is not guaranteed; for that reason, scikit-learn by default uses a large number of random initializations and finds the best results.
Also, the number of clusters must be set beforehand... there are other clustering algorithms for which this requirement may be lifted.
Application of KMeans to Digits
For a closer-to-real-world example, let's again take a look at the digits data. Here we'll use KMeans to automatically cluster the data in 64 dimensions, and then look at the cluster centers to see what the algorithm has found.
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()

est = KMeans(n_clusters=10)
clusters = est.fit_predict(digits.data)
est.cluster_centers_.shape


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
We can envision this image as a cloud of points in a 3-dimensional color space. We'll rescale the colors so they lie between 0 and 1, then reshape the array to be a typical scikit-learn input:
Can you write Python code for it?

X = (china / 255.0).reshape(-1, 3)
print(X.shape)


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
We now have 273,280 points in 3 dimensions.
Our task is to use KMeans to compress the $256^3$ colors into a smaller number (say, 64 colors). Basically, we want to find $N_{color}$ clusters in the data, and create a new image where the true input color is replaced by the color of the closest cluster.
Here we'll use MiniBatchKMeans, a more sophisticated estimator that performs better for larger datasets:
Can you write Python code for it?

from sklearn.cluster import MiniBatchKMeans

# reduce the size of the image for speed
n_colors = 64

X = (china / 255.0).reshape(-1, 3)

model = MiniBatchKMeans(n_colors)
labels = model.fit_predict(X)
colors = model.cluster_centers_
new_image = colors[labels].reshape(china.shape)
new_image = (255 * new_image).astype(np.uint8)

# create and plot the new image
with sns.axes_style('white'):
    plt.figure()
    plt.imshow(china)
    plt.title('input: 16 million colors')

    plt.figure()
    plt.imshow(new_image)
    plt.title('{0} colors'.format(n_colors))


I am working on the file "Day_1_Scientific_Python/scikit-learn/04.2-Clustering-KMeans.ipynb".
The first task is:
We see that even without the labels, KMeans is able to find clusters whose means are recognizable digits (with apologies to the number 8)!
The cluster labels are permuted; let's fix this:
Can you write Python code for it?

from scipy.stats import mode

labels = np.zeros_like(clusters)
for i in range(10):
    mask = (clusters == i)
    labels[mask] = mode(digits.target[mask])[0]


I am working on the file "Day_1_Scientific_Python/scikit-learn/13-Cross-Validation.ipynb".
The first task is:
Another helpful cross-validation generator is ShuffleSplit. This generator simply splits of a random portion of the data repeatedly. This allows the user to specify the number of repetitions and the training set size independently:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/13-Cross-Validation.ipynb".
The first task is:
As you can see, the function uses three folds by default. You can change the number of folds using the cv argument:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/13-Cross-Validation.ipynb".
The first task is:
As you can see, there is a rather wide spectrum of scores from 90% correct to 100% correct. If we only did a single split, we might have gotten either answer.
As cross-validation is such a common pattern in machine learning, there are functions to do the above for you with much more flexibility and less code.
The sklearn.model_selection module has all functions related to cross validation. There easiest function is cross_val_score which takes an estimator and a dataset, and will do all of the splitting for you:
Can you write Python code for it?

from sklearn.model_selection import cross_val_score
scores = cross_val_score(classifier, X, y)
print(scores)
print(np.mean(scores))


I am working on the file "Day_1_Scientific_Python/scikit-learn/13-Cross-Validation.ipynb".
The first task is:
For comparison, again the standard KFold, that ignores the labels:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/13-Cross-Validation.ipynb".
The first task is:
Keep in mind that increasing the number of folds will give you a larger training dataset, but will lead to more repetitions, and therefore a slower evaluation:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/13-Cross-Validation.ipynb".
The first task is:
There are also helper objects in the cross-validation module that will generate indices for you for all kinds of different cross-validation methods, including k-fold:
Can you write Python code for it?

from sklearn.model_selection import KFold, StratifiedKFold, ShuffleSplit


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
As this is such a very common pattern, there is a built-in class for this in scikit-learn, GridSearchCV. GridSearchCV takes a dictionary that describes the parameters that should be tried and a model to train.
The grid of parameters is defined as a dictionary, where the keys are the parameters and the values are the settings to be tested.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10], 'gamma': [0.001, 0.01, 0.1, 1]}

grid = GridSearchCV(SVR(), param_grid=param_grid, cv=cv, verbose=3)


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
Some practitioners go for an easier scheme, splitting the data simply into three parts, training, validation and testing. This is a possible alternative if your training set is very large, or it is infeasible to train many models using cross-validation because training a model takes very long.
You can do this with scikit-learn for example by splitting of a test-set and then applying GridSearchCV with ShuffleSplit cross-validation with a single iteration:
<img src="figures/train_validation_test2.svg" width="100%">
Can you write Python code for it?

from sklearn.model_selection import train_test_split, ShuffleSplit

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

param_grid = {'C': [0.001, 0.01, 0.1, 1, 10], 'gamma': [0.001, 0.01, 0.1, 1]}
single_split_cv = ShuffleSplit(n_splits=1)

grid = GridSearchCV(SVR(), param_grid=param_grid, cv=single_split_cv, verbose=3)

grid.fit(X_train, y_train)
grid.score(X_test, y_test)


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
There is a function in scikit-learn, called validation_plot to reproduce the cartoon figure above. It plots one parameter, such as the number of neighbors, against training and validation error (using cross-validation):
Can you write Python code for it?

from sklearn.model_selection import validation_curve
n_neighbors = [1, 3, 5, 10, 20, 50]

train_errors, test_errors = validation_curve(KNeighborsRegressor(), X, y, param_name="n_neighbors",
                                             param_range=n_neighbors, cv=cv)
plt.plot(n_neighbors, train_errors.mean(axis=1), label="train error")
plt.plot(n_neighbors, test_errors.mean(axis=1), label="test error")
plt.legend(loc="best")


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
There is a problem with using this score for evaluation, however. You might be making what is called a multiple hypothesis testing error. If you try very many parameter settings, some of them will work better just by chance, and the score that you obtained might not reflect how your model would perform on new unseen data.
Therefore, it is good to split off a separate test-set before performing grid-search. This pattern can be seen as a training-validation-test split, and is common in machine learning:
<img src="figures/grid_search_cross_validation.svg" width="100%">
We can do this very easily by splitting of some test data using train_test_split, training GridSearchCV on the training set, and applying the score method to the test set:
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

param_grid = {'C': [0.001, 0.01, 0.1, 1, 10], 'gamma': [0.001, 0.01, 0.1, 1]}
cv = KFold(n_splits=10, shuffle=True)

grid = GridSearchCV(SVR(), param_grid=param_grid, cv=cv)

grid.fit(X_train, y_train)
grid.score(X_test, y_test)


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
This is much faster, but might result in worse hyperparameters and therefore worse results.
Can you write Python code for it?

clf = GridSearchCV(SVR(), param_grid=param_grid)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
We can also look at the parameters that were selected:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
What fit does is a bit more involved then what we did above. First, it runs the same loop with cross-validation, to find the best parameter combination.
Once it has the best combination, it runs fit again on all data passed to fit (without cross-validation), to built a single new model using the best parameter setting.
Then, as with all models, we can use predict or score:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/14-Model-Complexity-and-GridSearchCV.ipynb".
The first task is:
You can inspect the best parameters found by GridSearchCV in the best_params_ attribute, and the best score in the best_score_ attribute:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/scikit-learn/15-Pipelining-Estimators.ipynb".
The first task is:
As you can see, this makes the code much shorter and easier to handle. Behind the scenes, exactly the same as above is happening. When calling fit on the pipeline, it will call fit on each step in turn.
After the first step is fit, it will use the transform method of the first step to create a new representation.
This will then be fed to the fit of the next step, and so on.
Finally, on the last step, only fit is called.

If we call score, only transform will be called on each step - this could be the test set after all! Then, on the last step, score is called with the new representation. The same goes for predict.
Building pipelines not only simplifies the code, it is also important for model selection.
Say we want to grid-search C to tune our Logistic Regression above.
Let's say we do it like this:
Can you write Python code for it?

# this illustrates a common mistake. Don't use this code!
from sklearn.model_selection import GridSearchCV

pca = PCA(n_components=5)
pca.fit(X_train)

X_train_reduced = pca.transform(X_train)
X_test_reduced = pca.transform(X_test)

logistic = LogisticRegression()
grid = GridSearchCV(logistic, param_grid={'C': [.1, 1, 10, 100, 1000, 10000]}, cv=5)
grid.fit(X_train_reduced, y_train)

print(grid.score(X_test_reduced, y_test))
print(grid.best_params_)


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
A one-liner using np.where:
Can you write Python code for it?

result = np.where(condition, a**2, 1)
print(result)


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Array attributes
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
By default, the dtype of the created array is float64 but other dtypes can be used:
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Can you guess what the following slices are equal to? Print them to check your understanding.
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Create array with random numbers
Can you write Python code for it?

np.random.rand(4)       # uniform in [0, 1]

np.random.normal(0,1,4)      # Gaussian (mean,std dev, num samples)

np.random.gamma(1,1,(2,2))      # Gamma (shape, scale , num samples)


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Creating an array from a list
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Exercise 1
Create a 2D NumPy array from the following list and assign it to the variable "arr":
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Exercise 3

Create a "data" array of 30 linearly spaced numbers in the interval (1, 15)
Hint: use np.linspace or np.arange functions
Create a condition - a True/False (boolean) array, corresponding to the masked values
The data array should be masked where elements are
greater than 1
less or equal than 12
divisible by 2.

Mask the array depending on the condition
Hint: use np.where function
Can you write Python code for it?

# Your code:
# arr =

# Your code:
# condition =

# masked_arr = np.ma.masked_where(condition, arr)
# print(masked_arr)


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Fancy indexing
NumPy arrays can be indexed with slices, but also with boolean or
integer arrays (masks)
Can you write Python code for it?

a = np.random.randint(1, 100, 6) # array of 6 random integers between 1 and 100
a

mask = ( a % 3 == 0 ) # Where divisible by 3 (% is the modulus operator).
mask

a[mask]


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Functions for creating arrays
arange([start,] stop[, step,], dtype=None)
evenly spaced, defined by step
Can you write Python code for it?

np.arange(1, 9, 2)

# for integers, np.arange is same as range but returns an array insted of a list
np.array( range(1,9,2) )


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Grid generation

A common task is to generate a pair of arrays that represent data coordinates.
Useful for interpolation of mapping contours.
When orthogonal 1D coordinate arrays already exist, NumPy's meshgrid function is very useful:
Can you write Python code for it?

x = np.linspace(-5, 5, 3)
y = np.linspace(10, 40, 4)
x2d, y2d = np.meshgrid(x, y)
print(x2d)
print(y2d)


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Masked arrays - how to handle (propagating) missing values

All operations related to masked arrays live in numpy.ma submodule.
The simplest example of manual creation of a masked array:
Can you write Python code for it?

a = np.ma.masked_array(data=[1, 2, 3],
                       mask=[True, True, False],
                       fill_value=-999)
a


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Note that size is not equal to len(). The latter returns the length of the first dimension.
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Shape manipulation
Can you write Python code for it?

a = np.array([[1, 2, 3], [4, 5, 6]])

print('{} <-- array'.format(a))
print('{} <-- its shape'.format(a.shape))

a.flatten()

a.repeat(4)

a.reshape((3, 2))

print('Old shape: {}'.format(a.shape))
print('New shape: {}'.format(a.reshape((3, 2)).shape))


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Statistical methods of arrays
Can you write Python code for it?

print('array a1d                       :',a1d)
print('Minimum and maximum             :', a1d.min(), a1d.max())
print('Sum and product of all elements :', a1d.sum(), a1d.prod())
print('Mean and standard deviation     :', a1d.mean(), a1d.std())


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Suppose you want to return an array result, which has the squared value when an element in array a is greater than -90 and less than -40, and is 1 otherwise.

With a For loop it would look like this:
Can you write Python code for it?

result = np.zeros(a.shape, dtype=a.dtype)

for i in range(a.shape[0]):
    for j in range(a.shape[1]):
        if a[i, j] > -90 and a[i, j] < -40:
            result[i, j] = a[i, j]**2
        else:
            result[i, j] = 1

result


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
The core class of NumPy is the ndarray (homogeneous n-dimensional array).
To find methods or attributes:
<code>
a1d.   ->tab
</code>
Common mistakes
Can you write Python code for it?

try:
    a = np.array(1,2,3,4)   # WRONG, only 2 non-keyword arguments accepted
except ValueError as err:
    print(err)

a = np.array([1,2,3,4]) # RIGHT

np.ndarray([1,2,3,4]) #  ndarray is is a low level method. Use np.array() instead


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the rank, or number of dimensions, ndim.
ndarray.size
the total number of elements of the array
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Using indexes: how to calculate x[i]-x[i-1] without a loop?
Can you write Python code for it?

x = np.random.rand(6)
x

xs = np.sort(x)
xs

xs[1:] - xs[:-1]


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Vectorisation: operations on whole arrays
Can you write Python code for it?

np.exp(a/100.)/a

# Non-vectorised
r=np.zeros(a.shape)   # create empy array for results

for i in range(len(a)):
    r[i] = np.exp(a[i]/100.)/a[i]
r


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Vectorised

array2 = array1*k+c

Non-vectorised, here with a list

for index in range(len(list1)):
    list2[index] = list1[index]*k + c

Do we still need lists?

Lists can have different objects as elements. Arrays are homogenous.

example_list = [number, string, cat, dog]

Both arrays and lists can be nested

nested_list =
[[1,2],
[1,2,3],
[1]]

Looking for help?

Documentation: http://docs.scipy.org/doc/numpy/reference/

Use help function (tab will show options available)

    help(np.mean)

Interactive help: NumPy has an a built-in search engine
Can you write Python code for it?


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
Vectorization is generally faster than a for loop.
But it is not always possible or the most readable:
<code>
for i in range(len(a)):
    if isprime(i):
        r[i] = a[i]
    else
        r[i]= a[i]+a[i-1]
</code>
Array indexing

Indices begin at 0, like other Python sequences and C/C++. Note that many languages, such as Matlab, R and Fortran, start with 1
In 2D, the first dimension corresponds to rows, the second to columns.
The fastest varying dimension is the last dimension! The outer level of the hierarchy is the first dimension.
Can you write Python code for it?

a = np.arange(10, 100, 10)
a

a[2:9:3] # [start:end:step]

a[:3] # last is not included

a[-2] # negative index counts from the end


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None)
evenly spaced, defined by length
Can you write Python code for it?

np.linspace(0, 1, 10)   # start, end, num-points


I am working on the file "Day_2/14-Numpy-Intro.ipynb".
The first task is:
ndarray.ndim
the number of axes (dimensions) of the array. In NumPy, the number of dimensions is referred to as rank.
Can you write Python code for it?


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
2. numpy.genfromtxt()
Can you write Python code for it?

## Header is 6 lines and we are going to read in the data
skip_rows = 6

## Read 'file_name' and call it 'data'
## This is a simple .csv file. Go into the folder 'Data' and open it in Excel if you don't believe us!
## Set delimiter for csv, we will skip the headers, and names=True means that columns have names we will use.
data = np.genfromtxt(file_name, delimiter=',', skip_header=skip_rows, names=True, dtype=None)

## Name the data based on the column names
years = data['Year']
temperature = data['Annual_anomaly']


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
4. Save the figure
Can you write Python code for it?

# fig.savefig('awesome_figure.png', dpi=300, bbox_to_inches='tight')


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Example - Contour plots
Can you write Python code for it?

#### Create a function for generating visual data
def f(x,y):
    return (1-x/2+x**5+y**3)*np.exp(-x**2-y**2)

### Creates data
n = 256
x = np.linspace(-3,3,n)
y = np.linspace(-3,3,n)
X,Y = np.meshgrid(x,y)

### Object orientated
fig4 = plt.figure()
ax = fig4.add_subplot(111)

### Plot a contour plot
ax.contourf(X, Y, f(X,Y), 8, alpha=.75, cmap=plt.cm.hot)
C = ax.contour(X, Y, f(X,Y), 8, colors='black', linewidth=.5)

### Do you want values inline with the contours? If so, uncomment me below!
#ax.clabel(C, inline=1, fontsize=30)
print("Did you label the values inline?")


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Example of what else you can do:
Scatter Plot and colour the points by frequency
Can you write Python code for it?

  ### Create some random numbers
n = 100000
x = np.random.randn(n)
y = (1.5 * x) + np.random.randn(n)

    ### Plot the data without colouring the points by frequency
fig1 = plt.figure()
ax = fig1.add_subplot(111)

    ### Formatting
ax.plot(x,y,'.r')
ax.set_xlabel('Random X')
ax.set_ylabel('Random Y')

    ### Estimate the 2D histogram we are using a numpy function and binning the data.
    ### What happends if you play around with the value 200???
#nbins = 200
H, xedges, yedges = np.histogram2d(x,y,bins=nbins)

# H needs to be rotated and flipped
H = np.rot90(H)
H = np.flipud(H)

# Mask zeros
Hmasked = np.ma.masked_where(H==0,H) # Mask pixels with a value of zero

# Plot 2D histogram using pcolor
fig2 = plt.figure()
ax = fig2.add_subplot(111)

ax.pcolormesh(xedges,yedges,Hmasked)
ax.set_xlabel('Random X distribution')
ax.set_ylabel('Random Y distribution')


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Formatting:
Can you write Python code for it?

### How about adding some labels?
#ax.set_xlabel('Year')
#ax.set_ylabel('Temperature (oF)')
#ax.set_title('Annual Anomaly')


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Note: pandas.read_csv() function has a similar functionality and returns similar data structure.
3. Plot a line graph of Average Ocean Temperature from 1880-Present
Can you write Python code for it?

### Time to create our plot
### fig sets up a blank figure for us.
fig = plt.figure()

### We are now going to use object orientated apporoach and call it 'ax'
ax = fig.add_subplot(111)

### Plot X, Y using 'ax' our blank figure. Colour (c) = 'red'...try changing this!
#ax.plot(x, y, c='red')


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Now lets see if we can make some subplots
Can you write Python code for it?

# Here we are going to loop over the same dataset twice and product a subplot with a different colour.
# Two data arrays are created - uncomment 'data' to see what the output looks like.
data = [temperature for i in range(2)]

# Create a colour scheme up to the value of 12. A simple array of numbers.
colours = np.linspace(0.2,6,12)

### Trying playing with the value '6' what does it do and how does it effect the output?
### Why not try the other values too....remember to save your origional code! Use comments '#' to save it for later.

# Set a figure of 2 Vertical, by 1 Horizonal
fig, ax = plt.subplots(2, 1)

# Loop over the two data arrays,
for ix, arr, col in zip(ax.flat, data, colours):
    ix.plot(arr, color=str(col))


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Source: http://matplotlib.org/examples/shapes_and_collections/scatter_demo.html
Exercise 1
Now lets have a look at some real data!
Can you write Python code for it?

# First of all, lets read some data from a CSV file.
#"Figure 1. Average Global Sea Surface Temperature, 1880-2015",,,
#Source: EPA's Climate Change Indicators in the United States: www.epa.gov/climate-indicators,,,"Data source: NOAA, 2016",,,
# Units: temperature anomaly (°F),,,
# https://www.epa.gov/sites/production/files/2016-08/sea-surface-temp_fig-1.csv

# Import our libraries
import csv
import matplotlib.pyplot as plt
import os

# We first need to find our file.
file_name = os.path.join(os.path.pardir, 'data', 'sea-surface-temp_fig-1.csv')


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Task:
Try to produce three plots of the above data.
If you are feeling very adventureous, how about functionalising it!
The simple task is to produce a scatter plot for each variable above, changing the colour, and transparency of the points.
If you have time, how about using numpy.mean to mean the data via year, and then plot a line graph. If you get this far, try changing the 'linestyle' or 'ls' as you have done with colour and transparency.
Can you write Python code for it?

#fig = plt.figure()
#ax = fig.add_subplot(111)

#ax.???(???, ???, color=???, alpha=???)  # hint how about we try a scatter plot?
#ax.set_xlabel('???')
#ax.set_ylabel('???')
#ax.set_title('???')


I am working on the file "Day_2/15-Matplotlib.ipynb".
The first task is:
Try looking at the data from the 1960's to the present day. You can do this by just changing the 'x' and 'y' axis
Can you write Python code for it?

## This one sets the limits of the x and y axis.
#ax.set_xlim(..., ...)
#ax.set_ylim(..., ...)


I am working on the file "Day_2/16-Scipy-Intro.ipynb".
The first task is:
Aside
Just as in Fortran, you can use direct access way of reading unformatted binary files with a given length of records. For example:

python
nx = 123 # points in longitude
ny = 456 # points in latitude
with open(my_file_name, 'rb') as f:
    f.seek(4 * ny * nx)  # Move nx*ny float32 records forward (skip the first slice)
    arr = np.fromfile(f, dtype=np.float32, count=ny*nx) # Read the next slice of data

Check the type and length of bin_data. Is this what you expected?
How will you slice this array into coordinates and data values?
Can you write Python code for it?

# lons = bin_data[:nrec]
# lats =
# temp =


I am working on the file "Day_2/16-Scipy-Intro.ipynb".
The first task is:
Define a rectangular grid with constant step
Can you write Python code for it?

# lon_step =
# lat_step =
# lon1d = np.arange(-10, 60.1, lon_step)
# lat1d = np.arange(35, 70.1, lat_step)
# lon2d, lat2d = np.meshgrid(lon1d, lat1d)


I am working on the file "Day_2/16-Scipy-Intro.ipynb".
The first task is:
Plot results
Enough boring interpolation, time to plot what you got so far.

Create a grid of 4 subplots: one for the observations and 3 for the gridded data
Plot observations using scatter() function
Use contourf() or pcolormesh() to plot gridded data
For easier comparison, use the same colour limits in all 4 subplots
Attach a colorbar to each subplot, if you like
Set titles
Tweak/optimize the code as you want
Compare different interpolation methods
Can you write Python code for it?

# fig, axs = plt.subplots(figsize=(15, 10), nrows=2, ncols=2)  #
# print('2x2 subplot grid: {}'.format(axs.shape))
#
# h = axs[0, 0].scatter(lons[sub_idx], lats[sub_idx], c=temp[sub_idx],
#                       edgecolors='none', vmin=..., vmax=...)
# fig.colorbar(h, ax=axs[0, 0])
# axs[0, 0].set_title('Observations')
#
#
#
# h = axs[0, 1].pcolormesh(lon2d, lat2d, temp_grd_lin,
#                          vmin=..., vmax=...)
# fig.colorbar(h, ax=axs[0, 1])
# axs[0, 0].set_title('Linear')
#
#
#
# h = axs[1, 0] ... ( ... temp_grd_near)
#
#
# axs[1, 0].set_title('Nearest')
#
#
#
# axs[1, 1].set_title('Cubic')


I am working on the file "Day_2/16-Scipy-Intro.ipynb".
The first task is:
Read data from a binary file
Define a file name:
Can you write Python code for it?

## code is ready here, just uncomment
#
# file_name = os.path.join(os.path.pardir, 'data',
#                          'surface_synoptic_20161109_0000_lon_lat_temperature_17097.bin')


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
2. Plot the subset data

You can create figure and axis using matplotlib.pyplot
Or just use the plot() method of pandas DataFrame
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
3. Explore what rolling() method is

What does this method return?
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
4. Plot the original series and the smoothed series
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
A lot of information at once including memory usage:
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Basic visualisation
Exercise
Try calling plot() method of the air_quality object:
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Cleaning data

The truth about data science: cleaning your data is 90% of the work. Fitting the model is easy. Interpreting the results is the other 90%.
— Jake VanderPlas (@jakevdp) June 13, 2016

Dealing with dates and times
Can you write Python code for it?

# def dateparse(date_str, time_str):
#     diff = pd.to_timedelta((df['End Date'] == '24:00:00').astype(int), unit='d')
#     pd.datetime.strptime(x+y, '%Y-%m-%d%H:%M:%S')

air_quality = pd.read_csv(fname, header=4, skipfooter=4, na_values='No data', engine='python',
                          parse_dates={'Time': [0, 1]})


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Computing correlations
Both Series and DataFrames have a corr() method to compute the correlation coefficient.
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Convert numpy.datetime64 objects (which are the indices of our DataFrame) to matplotlib floating point numbers. These numbers represent the number of days (fraction part represents hours, minutes, seconds) since 0001-01-01 00:00:00 UTC (assuming Gregorian calendar).
Can you write Python code for it?

# mdt = mdates.date2num(df.index.astype(pd.datetime))


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Correlations and regressions

Is there correlations between the timeseries we loaded?
First, let's take a glance at the whole DataFrame using a fancy scatter_matrix function.
Can you write Python code for it?

from pandas.tools.plotting import scatter_matrix

# with plt.style.context('ggplot'):
#     scatter_matrix(air_quality, figsize=(7, 7))


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Creating DataFrames

DataFrame can also be created manually, by grouping several Series together.

Now just for fun we switch to another dataset

create 2 Series objects from 2 CSV files
create a DataFrame by combining the two Series

Data are monthly values of

Southern Oscillation Index (SOI)
Outgoing Longwave Radiation (OLR), which is a proxy for convective precipitation in the western equatorial Pacific

Data were downloaded from NOAA's website: https://www.ncdc.noaa.gov/teleconnections/
Can you write Python code for it?

soi_df = pd.read_csv('../data/soi.csv', skiprows=1, parse_dates=[0], index_col=0, na_values=-999.9,
                     date_parser=lambda x: pd.datetime.strptime(x, '%Y%m'))

olr_df = pd.read_csv('../data/olr.csv', skiprows=1, parse_dates=[0], index_col=0, na_values=-999.9,
                     date_parser=lambda x: pd.datetime.strptime(x, '%Y%m'))

df = pd.DataFrame({'OLR': olr_df.Value,
                   'SOI': soi_df.Value})

# df.describe()


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Deleting columns
Let us concentrate our attention on the first 4 chemical species, and remove cobalt data from our DataFrame:
Can you write Python code for it?

air_quality = air_quality.drop('Co', 1)
air_quality = air_quality.drop('Co_status', 1)

air_quality.head()


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Exercise: create a scatter plot

You can use df.plot function with the appropriate keywords
What happens if you use "c=" keyword?
Pass another parameter: edgecolors='none'
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Exercise: rolling functions
1. Subset data

Start by subsetting the SOI DataFrame
Use either numerical indices, or, even better, datetime indices
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Exercise: writing to CSV text files
Find a method to save DataFrames to a text file (or whatever format you like more).
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Get descriptors for the horizontal axis (axis=1):
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Get descriptors for the vertical axis (axis=0):
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
If you want to visualise this correlation matrix, uncomment the following code cell.
Can you write Python code for it?

# fig, ax = plt.subplots()
# p = ax.imshow(air_quality.corr(), interpolation="nearest", cmap='RdBu_r', vmin=-1, vmax=1)
# ax.set_xticks(np.arange(len(air_quality.corr().columns)))
# ax.set_yticks(np.arange(len(air_quality.corr().index)))
# ax.set_xticklabels(air_quality.corr().columns)
# ax.set_yticklabels(air_quality.corr().index)
# fig.colorbar(p)


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
It is easy to create other useful plots using DataFrame:
Can you write Python code for it?

fig, (ax0, ax1) = plt.subplots(ncols=2)
air_quality.boxplot(ax=ax0, column=['O3', 'PM10'])
air_quality.O3.plot(ax=ax1, kind="kde")


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Numpy as pandas's backend
It is always possible to fall back to a good old NumPy array to pass on to scientific libraries that need them: SciPy, scikit-learn, etc
Can you write Python code for it?

air_quality['Nitrogen oxides as nitrogen dioxide'].values

type(air_quality['Nitrogen oxides as nitrogen dioxide'].values)


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Q: What did you notice about "Status" columns? Compare them to the original text file.
Can you write Python code for it?

# View the last rows of data
air_quality.tail(n=2)  # Note the optional argument (available for head() too)


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Q: the plot is too small. How to make it bigger?
As well as just a simple line plot:
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Renaming columns
Can you write Python code for it?

air_quality.columns = ['Time', 'O3', 'O3_status', 'NOx', 'NOx_status',
                       'CO', 'CO_status', 'PM10', 'PM10_status', 'Co', 'Co_status']
air_quality.columns


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Saving data
For each read_** function to load data, there is a to_** method attached to Series and DataFrames.
Excel spreadsheets
Uncomment the following code cell and run to save the whole DataFrame to an Excel file.
Can you write Python code for it?

# with pd.ExcelWriter("test.xls") as our_writer:
#     air_quality.to_excel(writer, sheet_name='Blah-blah')


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Setting missing values
As you may notice, we have negative values of ozone concentration, which does not make sense. So, let us replace those negative values with NaN:
Q: how to list data entries with negative O3?
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Show me what you got! <img src=https://staticdelivery.nexusmods.com/mods/1151/images/543-1-1447533110.png style="width:40px;">
Can you write Python code for it?

# uncomment to see the values
# carbon_monoxide


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
Some statistics
Can you write Python code for it?


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
We also import the os module that is useful for building paths to files (among many other things). And numpy with matplotlib just in case too.
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np
import os
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

fname = '../data/air_quality_hourly_london_marylebone.csv'


I am working on the file "Day_2/17-Pandas-Intro.ipynb".
The first task is:
What happens if put subplots=True as an argument of the plot() method?
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
2. Plot the subset data

You can create figure and axis using matplotlib.pyplot
Or just use the plot() method of pandas DataFrame
Can you write Python code for it?

sub_soi_df.plot(lw=0.5, marker='d', ms=3, linestyle='-', color='k', figsize=(8, 3), grid=True)


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
3. Explore what rolling() method is

What does this method return?
Can you write Python code for it?

# df.rolling?

roll = sub_soi_df.rolling(window=10, center=False)


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
4. Plot the original series and the smoothed series
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(sub_soi_df, label='SOI')
ax.plot(roll.mean(), label='mean')
leg = ax.legend()


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
A lot of information at once including memory usage:
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Append the new data to the original DataFrame:
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
As well as just a simple line plot:
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Cleaning data

The truth about data science: cleaning your data is 90% of the work. Fitting the model is easy. Interpreting the results is the other 90%.
— Jake VanderPlas (@jakevdp) June 13, 2016

Dealing with dates and times
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Computing correlations
Both Series and DataFrames have a corr() method to compute the correlation coefficient.
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Correlations and regressions

Is there correlations between the timeseries we loaded?
First, let's take a glance at the whole DataFrame using a fancy scatter_matrix function.
Can you write Python code for it?

from pandas.tools.plotting import scatter_matrix

with plt.style.context('ggplot'):
    scatter_matrix(air_quality, figsize=(7, 7))


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Create a scatter plot
Can you write Python code for it?

ax = df.plot(kind='scatter', x='OLR', y='SOI', c='mpl_date',
             colormap='viridis', colorbar=False, edgecolors='none')
plt.colorbar(ax.collections[0], ticks=mdates.YearLocator(5),
             format=mdates.DateFormatter('%Y'))


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Creating DataFrames

DataFrame can also be created manually, by grouping several Series together.
Now just for fun we switch to another dataset:
create 2 Series objects from 2 CSV files
create a DataFrame by combining the two Series
Can you write Python code for it?

soi_df = pd.read_csv('../data/soi.csv', skiprows=1, parse_dates=[0], index_col=0, na_values=-999.9,
                     date_parser=lambda x: pd.datetime.strptime(x, '%Y%m'))

olr_df = pd.read_csv('../data/olr.csv', skiprows=1, parse_dates=[0], index_col=0, na_values=-999.9,
                     date_parser=lambda x: pd.datetime.strptime(x, '%Y%m'))

df = pd.DataFrame({'OLR': olr_df.Value,
                   'SOI': soi_df.Value})

df.describe()

df.plot()


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Exercise: rolling functions
1. Subset data

Start by subsetting the SOI DataFrame
Use either numerical indices, or, even better, datetime indices
Can you write Python code for it?

sub_soi_df = soi_df['1992':'2015']
sub_soi_df.head()


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Get descriptors for the horizontal axis (axis=1):
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Get descriptors for the vertical axis (axis=0):
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
It is easy to create other useful plots using DataFrame:
Can you write Python code for it?

fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(10, 4))
air_quality.boxplot(ax=ax0, column=['O3', 'PM10'])
air_quality.O3.plot(ax=ax1, kind="kde")


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Let's try to read the data using pandas.read_csv() function.
Can you write Python code for it?

# minimal setup to read the given file
air_quality = pd.read_csv(fname, header=4, skipfooter=4, na_values='No data', engine='python')


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Ordinary Least Square (OLS) regressions
<img src=http://statsmodels.sourceforge.net/devel/_static/statsmodels_hybi_banner.png>
The recommeded way to build ordinaty least square regressions is by using statsmodels.
Can you write Python code for it?

import statsmodels.formula.api as sm

sm_model = sm.ols(formula="SOI ~ OLR", data=df).fit()

# df['SOI'].plot()
# df['OLR'].plot()
# ax = sm_model.fittedvalues.plot(label="model prediction")
# ax.legend(loc="lower center", ncol=3)


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Q: What did you notice about "Status" columns? Compare them to the original text file.
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Q: What happens if you remove the header? skipfooter? engine?
Data structures: DataFrame and Series
Let's interrogate the DataFrame object!
Can you write Python code for it?

type(air_quality)

# Internal nature of the object
print(air_quality.shape)
print()
print(air_quality.dtypes)

# View just the tip of data
air_quality.head(5)


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Renaming columns
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Saving data
For each read_** function to load data, there is a to_** method attached to Series and DataFrames.
Excel spreadsheets
Uncomment the following code cell and run to save the whole DataFrame to an Excel file.
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Setting missing values
As you may notice, we have negative values of ozone concentration, which does not make sense. So, let us replace those negative values with NaN:
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Show me what you got! <img src=https://staticdelivery.nexusmods.com/mods/1151/images/543-1-1447533110.png style="width:40px;">
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Some of its attributes:
Can you write Python code for it?

print(type(carbon_monoxide))
print(carbon_monoxide.dtype)
print(carbon_monoxide.shape)
print(carbon_monoxide.nbytes)


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
Some statistics
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
We also import the os module that is useful for building paths to files (among many other things). And numpy with matplotlib just in case too.
Can you write Python code for it?


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
We can mask them out in the same way as you do with numpy arrays:
Can you write Python code for it?

# Replace negative ozone values with NaN
air_quality.O3[air_quality.O3.values < 0] = np.nan


I am working on the file "Day_2/solutions/17-Pandas-Intro-Solution.ipynb".
The first task is:
What happens if put subplots=True as an argument of the plot() method?
Can you write Python code for it?


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:

    NOTE:

    Default values are evaluated when the function is defined, not when it is called. This can be problematic when using mutable types (e.g. dictionary or list) and modifying them in the function body, since the modifications will be persistent across invocations of the function.

Using an immutable type in a keyword argument:

Can you write Python code for it?

bigx = 10

def double_it(x=bigx):
    return x * 2

bigx = 1e9
double_it()


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:
Alternative to overcome this problem:
Can you write Python code for it?

def add_to_dict(args=None):
    if not args:
        args = {'a': 1, 'b': 2}

    for i in args.keys():
        args[i] += 1

    print(args)

add_to_dict
add_to_dict()
add_to_dict()


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:
Functions are objects
Functions are first-class objects, which means they can be:

assigned to a variable
an item in a list (or any collection)
passed as an argument to another function.
Can you write Python code for it?

va = variable_args
va('three', x=1, y=2)


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:
Optional parameters (keyword or named arguments)
The order of the keyword arguments does not matter, but it is good practice to use the same ordering as the function's definition
Keyword arguments are a very convenient feature for defining functions with a variable number of arguments, especially when default values are to be used in most calls to the function.
Can you write Python code for it?

def double_it (x=1):
    return 2*x

print(double_it(3))

print(double_it())

def addition(int1=1, int2=1, int3=1):
    return int1 + 2*int2 + 3*int3

print(addition(int1=1, int2=1, int3=1))

print(addition(int1=1, int3=1, int2=1)) # sequence of these named arguments do not matter


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:
Parameters
Mandatory parameters (positional arguments)
Can you write Python code for it?

def double_it(x):
    return 2*x

double_it(3)

double_it()


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:
Using an mutable type in a keyword argument (and modifying it inside the function body)
Can you write Python code for it?

def add_to_dict(args={'a': 1, 'b': 2}):
    for i in args.keys():
        args[i] += 1
    print(args)

add_to_dict
add_to_dict()
add_to_dict()
add_to_dict()

#the {'a': 1, 'b': 2} was created in the memory on the moment that the definition was evaluated


I am working on the file "Day_2_Software_engineering_best_practices/03_functions.ipynb".
The first task is:
Variable number of parameters
Special forms of parameters:

*args: any number of positional arguments packed into a tuple
**kwargs: any number of keyword arguments packed into a dictionary
Can you write Python code for it?

def variable_args(*args, **kwargs):
    print('args is', args)
    print('kwargs is', kwargs)

variable_args('one', 'two', x=1, y=2, z=3)


I am working on the file "Day_2_Software_engineering_best_practices/04_reusing_code_modules.ipynb".
The first task is:
Packages
A directory that contains many modules is called a package. A package is a module with submodules (which can have submodules themselves, etc.). A special file called __init__.py (which may be empty) tells Python that the directory is a Python package, from which modules can be imported.
Can you write Python code for it?

import scipy

scipy.__file__

import scipy.ndimage.morphology

scipy.ndimage.morphology.__file__


I am working on the file "Day_2_Software_engineering_best_practices/04_reusing_code_modules.ipynb".
The first task is:
The script has been executed. Moreover the variables defined in the script (such as message) are now available inside the interpreter’s namespace.
Other interpreters also offer the possibility to execute scripts (e.g., execfile in the plain Python interpreter, etc.).
It is also possible to execute this script as a standalone program, by executing the script inside a shell terminal (Linux/Mac console or cmd Windows console). For example, if we are in the same directory as the test.py file, we can execute this in a console:
Can you write Python code for it?


I am working on the file "Day_2_Software_engineering_best_practices/04_reusing_code_modules.ipynb".
The first task is:
__main__ function and executing scripts
Can you write Python code for it?

%%file demo2.py

"A second demo module with a main function."

def print_a():
    "Prints a."
    print('a')

def print_b():
    "Prints b."
    print('b')

if __name__ == '__main__':
    print_a()

!python demo2.py

import demo2
demo2.print_b()


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:
4. Writing tests for the spectra_analysis module
This part explains how to organize your tests for testing a module. It uses the spectra_analysis module created and upgrade all along this course.
We will write tests for the two submodules:
- spectra_analysis/preprocessing.py
- spectra_analysis/regression.py
Directory setup
We'll start by copying a clean version of the spectra_analysis module in the current directory.
Can you write Python code for it?

# Cleaning current directory
!rm -rf spectra_analysis

# Copy clean version from the previous course
!cp -vr solutions/04_modules/spectra_analysis .


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:
5. Test coverage
A test coverage is a report on the percentage of lines of a module that have been run during a test.
The higher the coverage, the greater the number of code lines that have been executed at least once during a test.
Installation
To use coverage with pytest, one must first install pytest coverage plugin, e.g. the pytest-cov library:
conda install pytest-cov

or
pip install pytest-cov

Usage
The coverage can then be run alongside the testing, by setting the path to the module to run the coverage against with an extra argument --cov=<path to the module>.
Can you write Python code for it?

!py.test -v --cov=spectra_analysis spectra_analysis/tests/


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:

EXERCISE:

Same as above, implement tests for the two methods.

Advice:

Start by defining a common array for both tests

  X = np.array([[0, 0, 0],
                [0, 0, 0],
                [1, 1, 1],
                [1, 1, 1],
                [1, 1, 1],
                [2, 2, 2]])

Compute the expected output of the methods applied on this array

Again, verify that the tests pass.
Can you write Python code for it?


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:

Info:
Shell commands can be run from the notebook preceded with an exclamation mark "!". This functionality will be used thoughout the notebook.

Launching pytest is as easy as move to the right directory and using the command line
py.test

It will start a recursive search from the current directory for Python files, look for methods containing "test" and run them.
Can you write Python code for it?


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:
The numpy assert methods will be used to write the tests in the next section.
Clean current directory before next section
Can you write Python code for it?


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:
Then we'll create the structure for the tests
Can you write Python code for it?

# Test directory
!mkdir spectra_analysis/tests

# __init__.py file so the directory is recognized as a submodule
!touch spectra_analysis/tests/__init__.py

# create a local copy of test data
!mkdir spectra_analysis/tests/data
!cp solutions/data/data.csv spectra_analysis/tests/data/.


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:
We will create a file called test_preprocessing.py with two unit tests: one for testing the normal conditions, the other one for handling the exceptions.
Can you write Python code for it?

%%file spectra_analysis/tests/test_preprocessing.py

from spectra_analysis.preprocessing import read_spectra

def test_read_spectra():
    # Write here the tests for the normal conditions
    pass

def test_read_spectra_exceptions():
    # Write here the tests to handle errors
    pass


I am working on the file "Day_2_Software_engineering_best_practices/06_unit_testing.ipynb".
The first task is:
preprocessing.py
We start first with the tests for the read_spectra method of the preprocessing submodule.
Can you write Python code for it?

from spectra_analysis.preprocessing import read_spectra


I am working on the file "Day_2_Software_engineering_best_practices/solutions/01_function_factorization.ipynb".
The first task is:
IO: Reading and preprocess the data
We can define a function which will read the data and process them.
Can you write Python code for it?

def read_spectra(path_csv):
    s = pandas.read_csv(path_csv)
    c = s['concentration']
    m = s['molecule']
    s = s['spectra']
    x = []
    for spec in s:
        x.append(numpy.fromstring(spec[1:-1], sep=','))
    s = pandas.DataFrame(x)

    return s, c, m

f = pandas.read_csv('data/freq.csv')

filenames = ['data/spectra_{}.csv'.format(i)
             for i in range(4)]

stot = []
c = []
m = []
for filename in filenames:
    s_tmp, c_tmp, m_tmp = read_spectra(filename)
    stot.append(s_tmp)
    c.append(c_tmp)
    m.append(m_tmp)

stot = pandas.concat(stot)
c = pandas.concat(c)
m = pandas.concat(m)


I am working on the file "Day_2_Software_engineering_best_practices/solutions/01_function_factorization.ipynb".
The first task is:
Plot helper functions
We can create two functions: (i) to plot all spectra and (ii) plot the mean spectra with the std intervals.
We will make a "private" function which will be used by both plot types.
Can you write Python code for it?

def _apply_axis_layout(ax, title):
    ax.set_xlabel('Frequency')
    ax.set_ylabel('Concentration')
    ax.set_title(title)
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.get_xaxis().tick_bottom()
    ax.get_yaxis().tick_left()
    ax.spines['left'].set_position(('outward', 10))
    ax.spines['bottom'].set_position(('outward', 10))

def plot_spectra(f, s, title):
    fig, ax = pyplot.subplots()
    ax.plot(f, s.T)
    _apply_axis_layout(ax, title)

def plot_spectra_by_type(f, s, classes, title):
    fig, ax = pyplot.subplots()
    for c_type in numpy.unique(classes):
        i = numpy.nonzero(classes == c_type)[0]
        ax.plot(f, numpy.mean(s.iloc[i], axis=0), label=c_type)
        ax.fill_between(numpy.ravel(f), numpy.mean(s.iloc[i], axis=0) + numpy.std(s.iloc[i], axis=0), numpy.mean(s.iloc[i], axis=0) - numpy.std(s.iloc[i], axis=0), alpha=0.2)
    _apply_axis_layout(ax, title)
    ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))

plot_spectra(f, stot, 'All training spectra')

plot_spectra_by_type(f, stot, m, 'Mean spectra in function of the molecules')

plot_spectra_by_type(f, stot, c, 'Mean spectra in function of the concentrations')


I am working on the file "Day_2_Software_engineering_best_practices/solutions/01_function_factorization.ipynb".
The first task is:
Training and testing a machine learning model for classification
Can you write Python code for it?

def plot_cm(cm, classes, title):
    import itertools
    fig, ax = pyplot.subplots()
    pyplot.imshow(cm, interpolation='nearest', cmap='bwr')
    pyplot.title(title)
    pyplot.colorbar()
    tick_marks = numpy.arange(len(numpy.unique(classes)))
    pyplot.xticks(tick_marks, numpy.unique(classes), rotation=45)
    pyplot.yticks(tick_marks, numpy.unique(classes))

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        pyplot.text(j, i, format(cm[i, j], fmt),
                    horizontalalignment="center",
                    color="white" if cm[i, j] > thresh else "black")

    pyplot.tight_layout()
    pyplot.ylabel('True label')
    pyplot.xlabel('Predicted label')

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.metrics import confusion_matrix

for clf in [RandomForestClassifier(random_state=0),
            LinearSVC(random_state=0)]:
    p = make_pipeline(StandardScaler(), PCA(n_components=100, random_state=0), clf)
    p.fit(stot, m)
    pred = p.predict(s4)
    plot_cm(confusion_matrix(m4, pred), p.classes_, 'Confusion matrix using {}'.format(clf.__class__.__name__))
    print('Accuracy score: {0:.2f}'.format(p.score(s4, m4)))


I am working on the file "Day_2_Software_engineering_best_practices/solutions/02_docstring.ipynb".
The first task is:
IO: Reading and preprocess the data
We can define a function which will read the data and process them.
Can you write Python code for it?

def read_spectra(path_csv):
    """Read and parse data in pandas DataFrames.

    Parameters
    ----------
    path_csv : str
        Path to the CSV file to read.

    Returns
    -------
    s : pandas DataFrame, shape (n_spectra, n_freq_point)
        DataFrame containing all Raman spectra.

    c : pandas Series, shape (n_spectra,)
        Series containing the concentration of the molecule.

    m : pandas Series, shape (n_spectra,)
        Series containing the type of chemotherapeutic agent.

    """
    s = pandas.read_csv(path_csv)
    c = s['concentration']
    m = s['molecule']
    s = s['spectra']
    x = []
    for spec in s:
        x.append(numpy.fromstring(spec[1:-1], sep=','))
    s = pandas.DataFrame(x)

    return s, c, m

# read the frequency and get a pandas serie
f = pandas.read_csv('data/freq.csv')['freqs']

# read all data for training
filenames = ['data/spectra_{}.csv'.format(i)
             for i in range(4)]

stot = []
c = []
m = []
for filename in filenames:
    s_tmp, c_tmp, m_tmp = read_spectra(filename)
    stot.append(s_tmp)
    c.append(c_tmp)
    m.append(m_tmp)

stot = pandas.concat(stot)
c = pandas.concat(c)
m = pandas.concat(m)


I am working on the file "Day_2_Software_engineering_best_practices/solutions/02_docstring.ipynb".
The first task is:
Reusability for new data:
Can you write Python code for it?

s4, c4, m4 = read_spectra('data/spectra_4.csv')

plot_spectra(f, stot, 'All training spectra')
plot_spectra_by_type(f, s4, m4, 'Mean spectra in function of the molecules')
plot_spectra_by_type(f, s4, c4, 'Mean spectra in function of the concentrations')


I am working on the file "Day_2_Software_engineering_best_practices/solutions/02_docstring.ipynb".
The first task is:
Training and testing a machine learning model for classification
Can you write Python code for it?

def plot_cm(cm, classes, title):
    """Plot a confusion matrix.

    Parameters
    ----------
    cm : ndarray, shape (n_classes, n_classes)
        Confusion matrix.

    classes : array-like, shape (n_classes,)
        Array contining the different spectra classes used in the
        classification problem.

    title : str
        Title added to the plot.

    Returns
    -------
    None

    """
    import itertools
    fig, ax = pyplot.subplots()
    pyplot.imshow(cm, interpolation='nearest', cmap='bwr')
    pyplot.title(title)
    pyplot.colorbar()
    tick_marks = numpy.arange(len(numpy.unique(classes)))
    pyplot.xticks(tick_marks, numpy.unique(classes), rotation=45)
    pyplot.yticks(tick_marks, numpy.unique(classes))

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        pyplot.text(j, i, format(cm[i, j], fmt),
                    horizontalalignment="center",
                    color="white" if cm[i, j] > thresh else "black")

    pyplot.tight_layout()
    pyplot.ylabel('True label')
    pyplot.xlabel('Predicted label')

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.metrics import confusion_matrix

for clf in [RandomForestClassifier(random_state=0),
            LinearSVC(random_state=0)]:
    p = make_pipeline(StandardScaler(), PCA(n_components=100, random_state=0), clf)
    p.fit(stot, m)
    pred = p.predict(s4)
    plot_cm(confusion_matrix(m4, pred), p.classes_, 'Confusion matrix using {}'.format(clf.__class__.__name__))
    print('Accuracy score: {0:.2f}'.format(p.score(s4, m4)))


I am working on the file "Day_2_Software_engineering_best_practices/solutions/02_docstring.ipynb".
The first task is:
Training and testing a machine learning model for regression
Can you write Python code for it?

def plot_regression(y_true, y_pred, title):
    """Plot actual vs. predicted scatter plot.

    Parameters
    ----------
    y_true : array-like, shape (n_samples,)
        Ground truth (correct) target values.

    y_pred : array-like, shape (n_samples,)
        Estimated targets as returned by a regressor.

    title : str
        Title added to the plot.

    Returns
    -------
    None

    """
    from sklearn.metrics import r2_score, median_absolute_error
    fig, ax = pyplot.subplots()
    ax.scatter(y_true, y_pred)
    ax.plot([0, 25000], [0, 25000], '--k')
    ax.set_ylabel('Target predicted')
    ax.set_xlabel('True Target')
    ax.set_title(title)
    ax.text(1000, 20000, r'$R^2$=%.2f, MAE=%.2f' % (
       r2_score(y_true, y_pred), median_absolute_error(y_true, y_pred)))
    ax.set_xlim([0, 25000])
    ax.set_ylim([0, 25000])

from sklearn.decomposition import PCA
from sklearn.linear_model import RidgeCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline

for reg in [RidgeCV(), RandomForestRegressor(random_state=0)]:
    p = make_pipeline(PCA(n_components=100), reg)
    p.fit(stot, c)
    pred = p.predict(s4)
    plot_regression(c4, pred, 'Regression using {}'.format(reg.__class__.__name__))

def fit_params(data):
    """Compute statistics for robustly scale data.

    Compute the median and the variance, i.e. the difference
    between the 75th and 25th percentiles.
    These statistics are used later to scale data.

    Parameters
    ----------
    data : pandas DataFrame, shape (n_spectra, n_freq_point)
        DataFrame containing all Raman spectra.

    Returns
    -------
    median : ndarray, shape (n_freq_point,)
        Median for each wavelength.

    variance : ndarray, shape (n_freq_point,)
        Variance (difference between the 75th and 25th
        percentiles) for each wavelength.

    """
    median = numpy.median(data, axis=0)
    p_25 = numpy.percentile(data, 25, axis=0)
    p_75 =...


I am working on the file "Day_2_Software_engineering_best_practices/solutions/03_code_style.ipynb".
The first task is:
IO: Reading and preprocess the data
We can define a function which will read the data and process them.
Can you write Python code for it?

def read_spectra(path_csv):
    """Read and parse data in pandas DataFrames.

    Parameters
    ----------
    path_csv : str
        Path to the CSV file to read.

    Returns
    -------
    spectra : pandas DataFrame, shape (n_spectra, n_freq_point)
        DataFrame containing all Raman spectra.

    concentration : pandas Series, shape (n_spectra,)
        Series containing the concentration of the molecule.

    molecule : pandas Series, shape (n_spectra,)
        Series containing the type of chemotherapeutic agent.

    """
    if not isinstance(path_csv, six.string_types):
        raise TypeError("'path_csv' needs to be string. Got {}"
                        " instead.".format(type(path_csv)))
    else:
        if not path_csv.endswith('.csv'):
            raise ValueError('Wrong file format. Expecting csv file')

    data = pd.read_csv(path_csv)
    concentration = data['concentration']
    molecule = data['molecule']
    spectra_string = data['spectra']
    spectra = []
    for spec in spectra_string:
        # remove the first and last bracket and convert to a numpy array
        spectra.append(np.fromstring(spec[1:-1], sep=','))
    spectra = pd.DataFrame(spectra)

    return spectra, concentration, molecule

# read the frequency and get a pandas serie
frequency = pd.read_csv('data/freq.csv')['freqs']

# read all data for training
filenames = ['data/spectra_{}.csv'.format(i)
             for i in range(4)]

spectra, concentration, molecule = [], [], []
for filename in filenames:
    spectra_file, concentration_file, molecule_file = read_spectra(filename)
    spectra.append(spectra_file)
    concentration.append(concentration_file)
    molecule.append(molecule_file)

# Concatenate in single DataFrame and Serie
spectra = pd.concat(spectra)
concentration = pd.concat(concentration)
molecule = pd.concat(molecule)


I am working on the file "Day_2_Software_engineering_best_practices/solutions/03_code_style.ipynb".
The first task is:
Training and testing a machine learning model for classification
Can you write Python code for it?

def plot_cm(cm, classes, title):
    """Plot a confusion matrix.

    Parameters
    ----------
    cm : ndarray, shape (n_classes, n_classes)
        Confusion matrix.

    classes : array-like, shape (n_classes,)
        Array contining the different spectra classes used in the
        classification problem.

    title : str
        Title added to the plot.

    Returns
    -------
    None

    """
    fig, ax = plt.subplots()
    plt.imshow(cm, interpolation='nearest', cmap='bwr')
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    return fig, ax

for clf in [RandomForestClassifier(random_state=0),
            LinearSVC(random_state=0)]:

    pipeline = make_pipeline(StandardScaler(),
                             PCA(n_components=100, random_state=0),
                             clf)
    y_pred = pipeline.fit(spectra, molecule).predict(spectra_test)
    plot_cm(confusion_matrix(molecule_test, y_pred),
            pipeline.classes_,
            'Confusion matrix using {}'.format(clf.__class__.__name__))
    print('Accuracy score: {0:.2f}'.format(pipeline.score(spectra_test,
                                                          molecule_test)))


I am working on the file "Day_2_Software_engineering_best_practices/solutions/03_code_style.ipynb".
The first task is:
Training and testing a machine learning model for regression
Can you write Python code for it?

def plot_regression(y_true, y_pred, title):
    """Plot actual vs. predicted scatter plot.

    Parameters
    ----------
    y_true : array-like, shape (n_samples,)
        Ground truth (correct) target values.

    y_pred : array-like, shape (n_samples,)
        Estimated targets as returned by a regressor.

    title : str
        Title added to the plot.

    Returns
    -------
    None

    """
    fig, ax = plt.subplots()
    ax.scatter(y_true, y_pred)
    ax.plot([0, 25000], [0, 25000], '--k')
    ax.set_ylabel('Target predicted')
    ax.set_xlabel('True Target')
    ax.set_title(title)
    ax.text(1000, 20000, r'$R^2$=%.2f, MAE=%.2f' % (
       r2_score(y_true, y_pred), median_absolute_error(y_true, y_pred)))
    ax.set_xlim([0, 25000])
    ax.set_ylim([0, 25000])
    return fig, ax

def regression_experiment(X_train, X_test, y_train, y_test):
    """Perform regression experiment.

    Build a pipeline using PCA and either a Ridge
    or a RandomForestRegressor model.

    Parameters
    ----------
    X_train : pandas DataFrame, shape (n_spectra, n_freq_points)
        DataFrame containing training Raman spectra.

    X_test : pandas DataFrame, shape (n_spectra, n_freq_points)
        DataFrame containing testing Raman spectra.

    y_training : pandas Serie, shape (n_spectra,)
        Serie containing the training concentrations acting as targets.

    y_testing : pandas Serie, shape (n_spectra,)
        Serie containing the testing concentrations acting as targets.

    Returns
    -------
    None

    """
    for reg in [RidgeCV(), RandomForestRegressor(random_state=0)]:
        pipeline = make_pipeline(PCA(n_components=100), reg)
        y_pred = pipeline.fit(X_train, y_train).predict(X_test)
        plot_regression(y_test, y_pred,
                        'Regression using {}'.format(reg.__class__.__name__))

regression_experiment(spectra, spectra_test,
                      concentration, concentration_test)

def fit_params(data):
   ...


I am working on the file "Day_2_Software_engineering_best_practices/solutions/04_modules/04_modules.ipynb".
The first task is:
IO: Reading and preprocess the data
We can define a function which will read the data and process them.
Can you write Python code for it?

# read the frequency and get a pandas serie
frequency = pd.read_csv('data/freq.csv')['freqs']

# read all data for training
filenames = ['data/spectra_{}.csv'.format(i)
             for i in range(4)]

spectra, concentration, molecule = [], [], []
for filename in filenames:
    spectra_file, concentration_file, molecule_file = read_spectra(filename)
    spectra.append(spectra_file)
    concentration.append(concentration_file)
    molecule.append(molecule_file)

# Concatenate in single DataFrame and Serie
spectra = pd.concat(spectra)
concentration = pd.concat(concentration)
molecule = pd.concat(molecule)