I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
Fitting the GridSearchCV object not only searches for the best parameters, but also automatically fits a new model on the whole training dataset with the parameters that yielded the best cross-validation performance. What happens in fit is therefore equivalent to the result of the code we saw at the beginning of this section. The GridSearchCV class provides a very convenient interface to access the retrained model using the predict and score methods. To evaluate how well the best found parameters generalize, we can call score on the test set:
Can you write Python code for it?

print("Test set score: {:.2f}".format(grid_search.score(X_test, y_test)))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
From the plots, we can see RM has a strong positive linear relationship with MEDV and LSTAT has a strong negative one.  This makes sense - the housing price should go up as the number of rooms increases and the housing prices should go down as the percentage of lower class/income families in the neighborhood increases.
Can you write Python code for it?

# Get a high-level overview of the data
boston_df.describe()

# Find which features are most highly correlated with the housing prices
df = boston_df
df['MEDV'] = y_train
df.corr()['MEDV']


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
Grid Search
Now that we know how to evaluate how well a model generalizes, we can take the next step and improve the model’s generalization performance by tuning its parameters. We discussed the parameter settings of the Ridge model for ridge regression earlier. Finding the values of the important parameters of a model (the ones that provide the best generalization performance) is a tricky task, but necessary for almost all models and datasets. Because it is such a common task, there are standard methods in scikit-learn to help you with it. The most commonly used method is grid search, which basically means trying all possible combinations of the parameters of interest.
Consider the case of ridge regression, as implemented in the Ridge class. As we discussed earlier, there is one important parameters: the regularization parameter, alpha. Say we want to try the values 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, and 100 for alpha. Because we have eleven different settings for alpha and alpha is the only parameter, we have 11 combinations of parameters in total. Looking at all possible combinations creates a table (or grid) of parameter settings for the Ridge regression model.
Simple Grid Search
We can implement a simple grid search just as a for loop over the parameter, training and evaluating a classifier for each value:
Can you write Python code for it?

X, y = load_extended_boston(scaler='standard')
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
print("Size of training set: {}   size of test set: {}".format(X_train.shape[0], X_test.shape[0]))

best_score = 0

for alpha in [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100]:
    # for each combination of parameters, train an SVC
    ridge = Ridge(alpha=alpha)
    ridge.fit(X_train, y_train)
    # evaluate the SVC on the test set
    score = ridge.score(X_test, y_test)
    # if we got a better score, store the score and parameters
    if score > best_score:
        best_score = score
        best_parameters = {'alpha': alpha}

print("Best score: {:.2f}".format(best_score))
print("Best parameters: {}".format(best_parameters))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
GridSearchCV will use cross-validation in place of the split into a training and validation set that we used before. However, we still need to split the data into a training and a test set, to avoid overfitting the parameters:
Can you write Python code for it?

X, y = load_extended_boston(scaler='standard')
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
Now the basic OLS model is doing a dramatically better job fitting the training set (R^2 of 0.95 vs 0.77).
This discrepancy between performance on the training set and the test set is a clear sign of overfitting, and therefore we should try to find a model that allows us to control complexity. One of the most commonly used alternatives to standard linear regression is ridge regression, which we will look into next.
Ridge Regression
Ridge regression is also a linear model for regression, so the formula it uses to make predictions is the same one used for ordinary least squares. In ridge regression, though, the coefficients (w) are chosen not only so that they predict well on the training data, but also to fit an additional constraint. We also want the magnitude of coefficients to be as small as possible; in other words, all entries of w should be close to zero. Intuitively, this means each feature should have as little effect on the outcome as possible (which translates to having a small slope), while still predicting well. This constraint is an example of what is called regularization. Regularization means explicitly restricting a model to avoid overfitting. The particular kind used by ridge regression is known as L2 regularization.
Ridge regression is implemented in linear_model.Ridge. Let’s see how well it does on the extended Boston Housing dataset:
Can you write Python code for it?

from sklearn.linear_model import Ridge

ridge = Ridge().fit(X_train, y_train)
print("Training set score: {:.2f}".format(ridge.score(X_train, y_train)))
print("Test set score: {:.2f}".format(ridge.score(X_test, y_test)))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
Sometimes it is helpful to have access to the actual model that was found—for example, to look at coefficients or feature importances. You can access the model with the best parameters trained on the whole training set using the best_estimator_ attribute:
Can you write Python code for it?


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
The Danger of Overfitting the Parameters and the Validation Set
Given this result, we might be tempted to report that we found a model that performs with 78% accuracy on our dataset. However, this claim could be overly optimistic (or just wrong), for the following reason: we tried many different parameters and selected the one with best accuracy on the test set, but this accuracy won’t necessarily carry over to new data. Because we used the test data to adjust the parameters, we can no longer use it to assess how good the model is. This is the same reason we needed to split the data into training and test sets in the first place; we need an independent dataset to evaluate, one that was not used to create the model.
One way to resolve this problem is to split the data again, so we have three sets: the training set to build the model, the validation (or development) set to select the parameters of the model, and the test set to evaluate the performance of the selected parameters.
After selecting the best parameters using the validation set, we can rebuild a model using the parameter settings we found, but now training on both the training data and the validation data. This way, we can use as much data as possible to build our model. This leads to the following implementation:
Can you write Python code for it?

X, y = load_extended_boston(scaler='standard')
# split data into train+validation set and test set
X_trainval, X_test, y_trainval, y_test = train_test_split(X, y, random_state=0)
# split train+validation set into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(X_trainval, y_trainval, random_state=1)
print("Size of training set: {}   size of validation set: {}   size of test set:"
      " {}\n".format(X_train.shape[0], X_valid.shape[0], X_test.shape[0]))

best_score = 0

for alpha in [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100]:
    # for each combination of parameters, train an SVC
    ridge = Ridge(alpha=alpha)
    ridge.fit(X_train, y_train)
    # evaluate the Ridge on the test set
    score = ridge.score(X_valid, y_valid)
    # if we got a better score, store the score and parameters
    if score > best_score:
        best_score = score
        best_parameters = {'alpha': alpha}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
ridge = Ridge(**best_parameters)
ridge.fit(X_trainval, y_trainval)
test_score = ridge.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
The best score on the validation set is 92%. However, the score on the test set—the score that actually tells us how well we generalize—is lower, at 78%. So we can claim to classify new data 78% correctly.  This happens to be the same as before, now we can make a stronger claim since the final test set wasn't used in any way shape or form during hyper-parameter tuning.
The distinction between the training set, validation set, and test set is fundamentally important to applying machine learning methods in practice. Any choices made based on the test set accuracy “leak” information from the test set into the model. Therefore, it is important to keep a separate test set, which is only used for the final evaluation. It is good practice to do all exploratory analysis and model selection using the combination of a training and a validation set, and reserve the test set for a final evaluation—this is even true for exploratory visualization. Strictly speaking, evaluating more than one model on the test set and choosing the better of the two will result in an overly optimistic estimate of how accurate the model is.
Grid Search with Cross-Validation
While the method of splitting the data into a training, a validation, and a test set that we just saw is workable, and relatively commonly used, it is quite sensitive to how exactly the data is split. From the output of the previous code snippet we can see that GridSearchCV selects 'alhpa': 50 as the best parameter.  But if we were to take a different part of the training data as the validation set, it may optimize for a different value.  For a better estimate of the generalization performance, instead of using a single split into a training and a validation set, we can use cross-validation to evaluate the performance of each parameter combination. This method can be coded up as follows:
Can you write Python code for it?

best_score = 0
for alpha in [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50, 100]:
    # for each combination of parameters, train an SVC
    ridge = Ridge(alpha=alpha)

    # perform cross-validation
    scores = cross_val_score(ridge, X_trainval, y_trainval, cv=5)

    # compute mean cross-validation accuracy
    score = np.mean(scores)

    # if we got a better score, store the score and parameters
    if score > best_score:
        best_score = score
        best_parameters = {'alpha': alpha}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
ridge = Ridge(**best_parameters)
ridge.fit(X_trainval, y_trainval)
test_score = ridge.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
The grid_search object that we created behaves just like a classifier; we can call the standard methods fit, predict, and score on it. However, when we call fit, it will run cross-validation for each combination of parameters we specified in param_grid:
Can you write Python code for it?


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
The lr object encapsulates the algorithm that will be used to build the model from the training data, as well the algorithm to make predictions on new data points.  It will also hold the information that the algorithm has extracted from the training data.
To build the model on the training set, we call the fit method of the lr object, which takes as arguments the NumPy array X_train containing the training data and the NumPy array y_train of the corresponding training labels.
Can you write Python code for it?


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
The “slope” parameters (w), also called weights or coefficients, are stored in the coef_ attribute, while the offset or intercept (b) is stored in the intercept_ attribute:
Can you write Python code for it?

print("lr.coef_: {}".format(lr.coef_))
print("lr.intercept_: {}".format(lr.intercept_))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
Using the mean cross-validation we can conclude that we expect the model to be around 96% accurate on average. Looking at all five scores produced by the five-fold cross-validation, we can also conclude that there is a relatively high variance in the accuracy between folds, ranging from 100% accuracy to 90% accuracy. This could imply that the model is very dependent on the particular folds used for training, but it could also just be a consequence of the small size of the dataset.
Benefits of Cross-Validation
There are several benefits to using cross-validation instead of a single split into a training and a test set. First, remember that train_test_split performs a random split of the data. Imagine that we are “lucky” when randomly splitting the data, and all examples that are hard to classify end up in the training set. In that case, the test set will only contain “easy” examples, and our test set accuracy will be unrealistically high. Conversely, if we are “unlucky,” we might have randomly put all the hard-to-classify examples in the test set and consequently obtain an unrealistically low score. However, when using cross-validation, each example will be in the training set exactly once: each example is in one of the folds, and each fold is the test set once. Therefore, the model needs to generalize well to all of the samples in the dataset for all of the cross-validation scores (and their mean) to be high.
Having multiple splits of the data also provides some information about how sensitive our model is to the selection of the training dataset. For the iris dataset, we saw accuracies between 90% and 100%. This is quite a range, and it provides us with an idea about how the model might perform in the worst case and best case scenarios when applied to new data.
Another benefit of cross-validation as compared to using a single split of the data is that we use our data more a single split of the data is that we use our data more effectively. When using train_test_split, we usually use 75% of the data for...
Can you write Python code for it?

lr = LinearRegression()
scores = cross_val_score(lr, boston.data, boston.target)
print("Cross-validation scores: {}".format(scores))


I am working on the file "SL2_Regression_and_Classification.ipynb".
The first task is:
We can now instantiate the GridSearchCV class with the model (Ridge), the parameter grid to search (param_grid), and the cross-validation strategy we want to use (say, five-fold stratified cross-validation):
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Ridge
grid_search = GridSearchCV(Ridge(), param_grid, cv=5)


I am working on the file "SL3_Neural_Networks.ipynb".
The first task is:
As you probably have realized by now, there are many ways to control the complexity of a neural network: the number of hidden layers, the number of units in each hidden layer, and the regularization (alpha). There are actually even more, which we won’t go into here.
An important property of neural networks is that their weights are set randomly before learning is started, and this random initialization affects the model that is learned. That means that even when using exactly the same parameters, we can obtain very different models when using different random seeds. If the networks are large, and their complexity is chosen properly, this should not affect accuracy too much, but it is worth keeping in mind (particularly for smaller networks). The figure below shows plots of several models, all learned with the same settings of the parameters:
Can you write Python code for it?

fig, axes = plt.subplots(2, 4, figsize=(20, 8))
for i, ax in enumerate(axes.ravel()):
    mlp = MLPClassifier(hidden_layer_sizes=[100, 100], solver='lbfgs', random_state=i)
    mlp.fit(X_train, y_train)
    plot_2d_separator(mlp, X_train, fill=True, alpha=.3, ax=ax)
    discrete_scatter(X_train[:, 0], X_train[:, 1], y_train, ax=ax)


I am working on the file "SL3_Neural_Networks.ipynb".
The first task is:
Finally, we can also control the complexity of a neural network by using an l2 penalty to shrink the weights toward zero, as we did in ridge regression and the linear classifiers. The parameter for this in the MLPClassifier is alpha (as in the linear regression models), and it’s set to a very low value (little regularization) by default. The figure below shows the effect of different values of alpha on the two_moons dataset, using two hidden layers of 10 or 100 units each:
Can you write Python code for it?

fig, axes = plt.subplots(2, 4, figsize=(20, 8))
for axx, n_hidden_nodes in zip(axes, [10, 100]):
    for ax, alpha in zip(axx, [0.0001, 0.01, 0.1, 1]):
        mlp = MLPClassifier(hidden_layer_sizes=[n_hidden_nodes, n_hidden_nodes], activation='relu', solver='lbfgs',
                            alpha=alpha, random_state=0)
        mlp.fit(X_train, y_train)
        plot_2d_separator(mlp, X_train, fill=True, alpha=.3, ax=ax)
        discrete_scatter(X_train[:, 0], X_train[:, 1], y_train, ax=ax)
        ax.set_title("n_hidden=[{}, {}]\nalpha={:.4f}".format(n_hidden_nodes, n_hidden_nodes, alpha))


I am working on the file "SL3_Neural_Networks.ipynb".
The first task is:
For a small neural network with a single hidden layer with three nodes, the full formula for computing ŷ in the case of regression would be (when using a tanh nonlinearity):
h[0] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])
h[1] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])
h[2] = tanh(w[0, 0] * x[0] + w[1, 0] * x[1] + w[2, 0] * x[2] + w[3, 0] * x[3])
ŷ = v[0] * h[0] + v[1] * h[1] + v[2] * h[2]

Here, w are the weights between the input x and the hidden layer h, and v are the weights between the hidden layer h and the output ŷ. The weights v and w are learned from data, x are the input features, ŷ is the computed output, and h are intermediate computations. An important parameter that needs to be set by the user is the number of nodes in the hidden layer. This can be as small as 10 for very small or simple datasets and as big as 10,000 for very complex data. It is also possible to add additional hidden layers.
Having large neural networks made up of many of these layers of computation is what inspired the term “deep learning.”
Advantages of Neural Networks

Able to capture information contained in large amounts of data and build incredibly complex models
Given enough computation time, data, and careful tuning of the parameters, neural networks often beat other machine learning algorithms (for classification and regression tasks)

Disadvantages of Neural Networks

Neural networks—particularly the large and powerful ones—often take a long time to train
Require careful preprocessing of the data
They work best with “homogeneous” data, where all the features have similar meanings
For data that has very different kinds of features, tree-based models might work better

Tuning neural network parameters is an art and generally more complex than tuning parameters for other algorithms

Neural Networks in scikit-learn
Recent versions of scikit-learn have added rudimentary support for neural networks.  The implementation in scikit-learn is not intended for...
Can you write Python code for it?

# Tweak some colormap stuff for Matplotlib
from matplotlib.colors import ListedColormap
cm2 = ListedColormap(['#0000aa', '#ff2020'])

# Helper function for classification plots
def plot_2d_separator(classifier, X, fill=False, ax=None, eps=None, alpha=1, cm=cm2, linewidth=None, threshold=None,
                      linestyle="solid"):
    # binary?
    if eps is None:
        eps = X.std() / 2.

    if ax is None:
        ax = plt.gca()

    x_min, x_max = X[:, 0].min() - eps, X[:, 0].max() + eps
    y_min, y_max = X[:, 1].min() - eps, X[:, 1].max() + eps
    xx = np.linspace(x_min, x_max, 100)
    yy = np.linspace(y_min, y_max, 100)

    X1, X2 = np.meshgrid(xx, yy)
    X_grid = np.c_[X1.ravel(), X2.ravel()]
    try:
        decision_values = classifier.decision_function(X_grid)
        levels = [0] if threshold is None else [threshold]
        fill_levels = [decision_values.min()] + levels + [decision_values.max()]
    except AttributeError:
        # no decision_function
        decision_values = classifier.predict_proba(X_grid)[:, 1]
        levels = [.5] if threshold is None else [threshold]
        fill_levels = [0] + levels + [1]
    if fill:
        ax.contourf(X1, X2, decision_values.reshape(X1.shape), levels=fill_levels, alpha=alpha, cmap=cm)
    else:
        ax.contour(X1, X2, decision_values.reshape(X1.shape), levels=levels, colors="black", alpha=alpha, linewidths=linewidth,
                   linestyles=linestyle, zorder=5)

    ax.set_xlim(x_min, x_max)
    ax.set_ylim(y_min, y_max)
    ax.set_xticks(())
    ax.set_yticks(())

# Helper function for classification plots
import matplotlib as mpl
from matplotlib.colors import colorConverter
def discrete_scatter(x1, x2, y=None, markers=None, s=10, ax=None,
                     labels=None, padding=.2, alpha=1, c=None, markeredgewidth=None):
    """Adaption of matplotlib.pyplot.scatter to plot classes or clusters.

    Parameters
    ----------

    x1 : nd-array
        input data, first axis

    x2 : nd-array
        input data, second axis

   ...


I am working on the file "SL3_Neural_Networks.ipynb".
The first task is:
If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).
Can you write Python code for it?

# You can now iterate on your training data in batches
model.fit(X_train, y_train, nb_epoch=20, batch_size=32)

# Evaluate your performance in one line
loss_and_metrics = model.evaluate(X_test, y_test, batch_size=32)
print(loss_and_metrics)


I am working on the file "SL3_Neural_Networks.ipynb".
The first task is:
Increasing the number of iterations only slightly increased the training and generalization performance. Still, the model is performing quite well. As there is some gap between the training and the test performance, we might try to decrease the model’s complexity to get better generalization performance. Here, we choose to increase the alpha parameter (quite aggressively, from 0.0001 to 1) to add stronger regularization of the weights:
Can you write Python code for it?

mlp = MLPClassifier(max_iter=1000, alpha=1, random_state=0)
mlp.fit(X_train_scaled, y_train)

print("Accuracy on training set: {:.3f}".format(mlp.score(X_train_scaled, y_train)))
print("Accuracy on test set: {:.3f}".format(mlp.score(X_test_scaled, y_test)))


I am working on the file "SL3_Neural_Networks.ipynb".
The first task is:
To get a better understanding of neural networks on real-world data, let’s apply the MLPClassifier to the Breast Cancer dataset which is built into scikit-learn. We start with the default parameters:
Can you write Python code for it?

from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
print("Cancer data per-feature maxima:\n{}".format(cancer.data.max(axis=0)))

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, random_state=0)

mlp = MLPClassifier(random_state=42)
mlp.fit(X_train, y_train)

print("Accuracy on training set: {:.2f}".format(mlp.score(X_train, y_train)))
print("Accuracy on test set: {:.2f}".format(mlp.score(X_test, y_test)))


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
A First Application: Classifying iris species
One of the most famous datasets for classification in a supervised learning setting is the Iris flower data set.  It is a multivariate dataset introduced in a 1936 paper which records sepal length, sepal width, petal length, and petal width for three species of iris.
scikit-learn has a number of small toy datasets included with it which makes it quick and easy to experiment with different machine learning algorithms on these datasets.
The sklearn.datasets.load_iris() method can be used to load the iris dataset.
Meet the data
The iris object that is returned by load_iris is a Bunch object, which is very similar to a dictionary.  It contains keys and values.
Can you write Python code for it?

from sklearn.datasets import load_iris
iris_dataset = load_iris()

print("Keys of iris_dataset: {}".format(iris_dataset.keys()))

# The value of the key DESCR is a short description of the dataset.  Here we show the beinning of the description.
print(iris_dataset['DESCR'][:193] + "\n...")

# The value of the key target_names is an array of strings, containing the species of flower that we want to predict
print("Target names: {}".format(iris_dataset['target_names']))

# The value of feature_names is a list of strings, giving the description of each feature
print("Feature names: {}".format(iris_dataset['feature_names']))

# The data itself is contained in the target and data fields.
# data contains the numeric measurements of sepal length, sepal width, petal length, and petal width in a NumPy array
print("Type of data: {}".format(type(iris_dataset['data'])))

# The rows in the data array correspond to flowers, while the columns represent the four measurements for each flower.
print("Shape of data: {}".format(iris_dataset['data'].shape))

# We see that the array contains measurements for 150 different flowers (samples).  Here are values for the first 5.
print("First five columns of data:\n{}".format(iris_dataset['data'][:5]))

# The target array contains the species of each of the flowers that were measured, also as a NumPy array
print("Type of target: {}".format(type(iris_dataset['target'])))

# target is a one-dimensional array, with one entry per flower
print("Shape of target: {}".format(iris_dataset['target'].shape))

# The species are encoded as integers from 0 to 2.  The meanings of the numbers are given by the target_names key.
print("Target:\n{}".format(iris_dataset['target']))


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
Effect of k
Let's Investigate the effect of varying k.  But the iris dataset isn't the ideal dataset for this purpose.  So let's create a synthetic dataset which this would be good for.
Can you write Python code for it?

import numbers
import numpy as np

from sklearn.utils import check_array, check_random_state
from sklearn.utils import shuffle as shuffle_

def make_blobs(n_samples=100, n_features=2, centers=2, cluster_std=1.0,
               center_box=(-10.0, 10.0), shuffle=True, random_state=None):
    """Generate isotropic Gaussian blobs for clustering.

    Read more in the :ref:`User Guide <sample_generators>`.

    Parameters
    ----------
    n_samples : int, or tuple, optional (default=100)
        The total number of points equally divided among clusters.

    n_features : int, optional (default=2)
        The number of features for each sample.

    centers : int or array of shape [n_centers, n_features], optional
        (default=3)
        The number of centers to generate, or the fixed center locations.

    cluster_std: float or sequence of floats, optional (default=1.0)
        The standard deviation of the clusters.

    center_box: pair of floats (min, max), optional (default=(-10.0, 10.0))
        The bounding box for each cluster center when centers are
        generated at random.

    shuffle : boolean, optional (default=True)
        Shuffle the samples.

    random_state : int, RandomState instance or None, optional (default=None)
        If int, random_state is the seed used by the random number generator;
        If RandomState instance, random_state is the random number generator;
        If None, the random number generator is the RandomState instance used
        by `np.random`.

    Returns
    -------
    X : array of shape [n_samples, n_features]
        The generated samples.

    y : array of shape [n_samples]
        The integer labels for cluster membership of each sample.

    Examples
    --------
    >>> from sklearn.datasets.samples_generator import make_blobs
    >>> X, y = make_blobs(n_samples=10, centers=3, n_features=2,
    ...                   random_state=0)
    >>> print(X.shape)
    (10, 2)
    >>> y
    array([0, 0, 1, 0, 2, 2, 2, 1, 1, 0])

    See also
    --------
   ...


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
Evaluating the model
How do we know whether we can trust our model?  This is where the test set that we created earlier comes in.  This data was not used to build the model, but we do know what the correct speciies is for each iris in the test set.
Therefore, we can make a prediction for each iris in the test data and compare it against its lable (the known species).  We can measure how well the model works by computing the accuracy, which is the fraction of flowers for which the right species was predicted.
We can also use the score method of the tree object, which will compute the test set accuracy for us.
Can you write Python code for it?

y_pred = knn.predict(X_test)
print("Test set predictions:\n {}".format(y_pred))

print("Test set score: {:.2f}".format(np.mean(y_pred == y_test)))

print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
First things first: Look at your data
Before building a machine learning model, it is often a good idea to inspect the data, to see if the task is easily solvable without machine learning, or if the desired information might not be contained in the data.
Additionally, inspecting the data is a good way to find abnormalities and peculiarities.  Maybe some of your irises were measured using inches and not centimeters, for example.  In the real world, inconsistencies in the data and unexpected measurements are very common, as are missing data and not-a-number (NaN) or infinite values.
One of the best ways to inspect data is to visualize it. One way to do this is by using a scatter plot.  A scatter plot of the data puts one feature along the x-axis and another along the y-axis, and draws a dot for each data point.  Unfortunately, computer screens have only two dimensions, which allows us to plot only two (or maybe three) features at a time.  It is difficult to plot datasets with more than three features this way.  One way around this problem is to do a pair plot, which looks at all possible pairs of features.  If you have a small number of features, such as the four we have here, this is quite reasonable.  You should keep in mind, however, that a pair plot does not show the interaction of all of the features at once, so some interesting aspects of the data may not be revealed when visualizing it this way.
In Python, the pandas library has a convenient function called scatter_matrix for creating pair plots for a DataFrame.
Can you write Python code for it?

# create dataframe from data in X_train
# label the columns using the strings in iris_dataset.feature_names
iris_dataframe = pd.DataFrame(X_train, columns=iris_dataset.feature_names)
# create a scatter matrix from the dataframe, color by y_train
grr = pd.scatter_matrix(iris_dataframe, c=y_train, figsize=(15, 15), marker='o',
                        hist_kwds={'bins': 20}, s=60, alpha=.8)


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
From the plots, we can see that the three classes seem to be relatively well separated using the sepal and petal measurements.  This means that a machine learning model will likely be able to learn to separate them quite well.
Building your model: k-Nearest Neighbors
Now we can start building the actual machine learning model.  There are many classification algorithms in scikit-learn that we could use.  Here we will use a k-nearest neighbors classifier, which is easy to understand.
The k in k-nearest neighbors signifies that instead of using only the closest neighbor to the new data point, we can consider any fixed number k of neighbors in the training (for example, the closest three or five neighbors). Then, we can make a prediction using the majority class among these neighbors. For starters, we’ll use only a single neighbor.
All machine learning models in scikit-learn are implemented in their own classes, which are called Estimator classes.  The k-nearest neighbors classification algorithm is implemented in the KNeighborsClassifier class in the neighbors module.  Before we can use the model, we need to instantiate the class into an object.  This is when we will set any parameters of the model.  The most important parameter of KNeighborsClassifier is n_neighbors , the number of neighbors, which we will set to 1. The default value for n_neighbors is 5.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=1)


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
Here, we added three new data points, shown as stars. For each of them, we marked the closest point in the training set. The prediction of the one-nearest-neighbor algorithm is the label of that point (shown by the color of the cross).
Instead of considering only the closest neighbor, we can also consider an arbitrary number, k, of neighbors. This is where the name of the k-nearest neighbors algorithm comes from. When considering more than one neighbor, we use voting to assign a label. This means that for each test point, we count how many neighbors belong to class 0 and how many neighbors belong to class 1. We then assign the class that is more frequent: in other words, the majority class among the k-nearest neighbors. The following example uses the three closest neighbors.
Can you write Python code for it?

# k = 3
plot_knn_classification(n_neighbors=3)


I am working on the file "SL4_Instance_Based_Learning.ipynb".
The first task is:
Measuring Success: Training and testing data
We want to build a machine learning model from this data that can predict the species of iris for a new set of measurements.  But before we can apply our model to new measurements, we need to know whether it actually works -- that is, whether we should trust its predictions.
Unfortunately, we cannot use the data we used to build the model to evaluate it.  This is because our model can always simply remember the whole training set, and will therefore always predict the correct label for any point in the training set.  This "remembering" does not indicate to us whether the model will generalize well (in other words, whether it will also perform well on new data).
To assess the model's performance, we show it new data (data that it hasn't seen before) for which we have labels.  This is usually done by splitting the labeled data we have collected (here, our 150 flower measurements) into two parts.  One part of the data is used to build our machine learning model, and is called the training data or training set.  The rest of the data will be used to assess how well the model works; this is called the test data, test set, or hold-out set.
scikit-learn contains a function that shuffles the dataset and splits it for you:  the train_test_split function.  This function extracts 75% of the rows in the data as the training set, together with the corresponding labels for this data.  The remaining 25% of the data, together with the remaining labels, is declared as the test set.  Deciding how much data you want to put into the training and the test set respectively is somewhat arbitrary, but scikit-learn's default 75/25 split is a reasonable starting point.
In scikit-learn, data is usually denoted with a capital X, while labels are denoted by a lowercase y.  This is inspired by the standard formulation f(x)=y in mathematics, where x is the input to a function and y is the output.  Following more conventions from mathematics, we use a capital X because the data is a...
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(iris_dataset['data'], iris_dataset['target'], random_state=0)

print("X_train shape: {}".format(X_train.shape))
print("y_train shape: {}".format(y_train.shape))

print("X_test shape: {}".format(X_test.shape))
print("y_test shape: {}".format(y_test.shape))


I am working on the file "SL5_Ensemble_Methods.ipynb".
The first task is:
As the training set accuracy is 100%, we are likely to be overfitting. To reduce overfitting, we could either apply stronger pre-pruning by limiting the maximum depth or lower the learning rate:
Can you write Python code for it?

gbrt = GradientBoostingClassifier(random_state=0, max_depth=1)
gbrt.fit(X_train, y_train)

print("Accuracy on training set: {:.3f}".format(gbrt.score(X_train, y_train)))
print("Accuracy on test set: {:.3f}".format(gbrt.score(X_test, y_test)))

gbrt = GradientBoostingClassifier(random_state=0, learning_rate=0.01)
gbrt.fit(X_train, y_train)

print("Accuracy on training set: {:.3f}".format(gbrt.score(X_train, y_train)))
print("Accuracy on test set: {:.3f}".format(gbrt.score(X_test, y_test)))


I am working on the file "SL5_Ensemble_Methods.ipynb".
The first task is:
As you can see, the random forest gives nonzero importance to many more features than the single tree. Similarly to the single decision tree, the random forest also gives a lot of importance to the “worst radius” feature, but it actually chooses “worst perimeter” to be the most informative feature overall. The randomness in building the random forest forces the algorithm to consider many possible explanations, the result being that the random forest captures a much broader picture of the data than a single tree.
Strengths, weaknesses, and parameters
Random forests for regression and classification are currently among the most widely used machine learning methods. They are very powerful, often work well without heavy tuning of the parameters, and don’t require scaling of the data.
Essentially, random forests share all of the benefits of decision trees, while making up for some of their deficiencies. One reason to still use decision trees is if you need a compact representation of the decision-making process. It is basically impossible to interpret tens or hundreds of trees in detail, and trees in random forests tend to be deeper than decision trees (because of the use of feature subsets). Therefore, if you need to summarize the prediction making in a visual way to nonexperts, a single decision tree might be a better choice. While building random forests on large datasets might be somewhat time consuming, it can be parallelized across multiple CPU cores within a computer easily. If you are using a multi-core processor (as nearly all modern computers do), you can use the n_jobs parameter to adjust the number of cores to use. Using more CPU cores will result in linear speed-ups (using two cores, the training of the random forest will be twice as fast), but specifying n_jobs larger than the number of cores will not help. You can set n_jobs=-1 to use all the cores in your computer.
You should keep in mind that random forests, by their nature, are random, and setting different random states (or not setting the...
Can you write Python code for it?

from sklearn.ensemble import GradientBoostingClassifier

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, random_state=0)

gbrt = GradientBoostingClassifier(random_state=0)
gbrt.fit(X_train, y_train)

print("Accuracy on training set: {:.3f}".format(gbrt.score(X_train, y_train)))
print("Accuracy on test set: {:.3f}".format(gbrt.score(X_test, y_test)))


I am working on the file "SL5_Ensemble_Methods.ipynb".
The first task is:
Both methods of decreasing the model complexity reduced the training set accuracy, as expected. In this case, lowering the maximum depth of the trees provided a significant improvement of the model, while lowering the learning rate only increased the generalization performance slightly.
As for the other decision tree–based models, we can again visualize the feature importances to get more insight into our model. As we used 100 trees, it is impractical to inspect them all, even if they are all of depth 1:
Can you write Python code for it?

gbrt = GradientBoostingClassifier(random_state=0, max_depth=1)
gbrt.fit(X_train, y_train)

plot_feature_importances_cancer(gbrt)


I am working on the file "SL5_Ensemble_Methods.ipynb".
The first task is:
The random forest gives us an accuracy of 97%, better than the linear models or a single decision tree, without tuning any parameters. We could adjust the max_features setting, or apply pre-pruning as we did for the single decision tree. However, often the default parameters of the random forest already work quite well.
Similarly to the decision tree, the random forest provides feature importances, which are computed by aggregating the feature importances over the trees in the forest. Typically, the feature importances provided by the random forest are more reliable than the ones provided by a single tree. Take a look at the figure below:
Can you write Python code for it?

def plot_feature_importances_cancer(model):
    n_features = cancer.data.shape[1]
    plt.figure(figsize=(10,6))
    plt.barh(range(n_features), model.feature_importances_, align='center')
    plt.yticks(np.arange(n_features), cancer.feature_names)
    plt.xlabel("Feature importance")
    plt.ylabel("Feature")

plot_feature_importances_cancer(forest)


I am working on the file "SL5_Ensemble_Methods.ipynb".
The first task is:
The trees that are built as part of the random forest are stored in the estimator_ attribute. Let’s visualize the decision boundaries learned by each tree, together with their aggregate prediction as made by the forest:
Can you write Python code for it?

from scipy import ndimage

def plot_tree_partition(X, y, tree, ax=None):
    if ax is None:
        ax = plt.gca()
    eps = X.std() / 2.

    x_min, x_max = X[:, 0].min() - eps, X[:, 0].max() + eps
    y_min, y_max = X[:, 1].min() - eps, X[:, 1].max() + eps
    xx = np.linspace(x_min, x_max, 1000)
    yy = np.linspace(y_min, y_max, 1000)

    X1, X2 = np.meshgrid(xx, yy)
    X_grid = np.c_[X1.ravel(), X2.ravel()]

    Z = tree.predict(X_grid)
    Z = Z.reshape(X1.shape)
    faces = tree.apply(X_grid)
    faces = faces.reshape(X1.shape)
    border = ndimage.laplace(faces) != 0
    ax.contourf(X1, X2, Z, alpha=.4, cmap=cm2, levels=[0, .5, 1])
    ax.scatter(X1[border], X2[border], marker='.', s=1)

    discrete_scatter(X[:, 0], X[:, 1], y, ax=ax)
    ax.set_xlim(x_min, x_max)
    ax.set_ylim(y_min, y_max)
    ax.set_xticks(())
    ax.set_yticks(())
    return ax

from matplotlib.colors import ListedColormap
cm2 = ListedColormap(['#0000aa', '#ff2020'])

def plot_2d_separator(classifier, X, fill=False, ax=None, eps=None, alpha=1,
                      cm=cm2, linewidth=None, threshold=None, linestyle="solid"):
    # binary?
    if eps is None:
        eps = X.std() / 2.

    if ax is None:
        ax = plt.gca()

    x_min, x_max = X[:, 0].min() - eps, X[:, 0].max() + eps
    y_min, y_max = X[:, 1].min() - eps, X[:, 1].max() + eps
    xx = np.linspace(x_min, x_max, 100)
    yy = np.linspace(y_min, y_max, 100)

    X1, X2 = np.meshgrid(xx, yy)
    X_grid = np.c_[X1.ravel(), X2.ravel()]
    try:
        decision_values = classifier.decision_function(X_grid)
        levels = [0] if threshold is None else [threshold]
        fill_levels = [decision_values.min()] + levels + [decision_values.max()]
    except AttributeError:
        # no decision_function
        decision_values = classifier.predict_proba(X_grid)[:, 1]
        levels = [.5] if threshold is None else [threshold]
        fill_levels = [0] + levels + [1]
    if fill:
        ax.contourf(X1, X2, decision_values.reshape(X1.shape),
                   ...


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
As a function of the original features, the linear SVM model is not actually linear anymore. It is not a line, but more of an ellipse, as you can see from the plot created here:
Can you write Python code for it?

ZZ = YY ** 2
dec = linear_svm_3d.decision_function(np.c_[XX.ravel(), YY.ravel(), ZZ.ravel()])
plt.contourf(XX, YY, dec.reshape(XX.shape), levels=[dec.min(), 0, dec.max()], cmap=cm2, alpha=0.5)
discrete_scatter(X[:, 0], X[:, 1], y, c=c)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
From this plot we can determine that features in the Breast Cancer dataset are of completely different orders of magnitude. This can be somewhat of a problem for other models (like linear models), but it has devastating effects for the kernel SVM. Let’s examine some ways to deal with this issue.
Preprocessing data for SVMs
One way to resolve this problem is by rescaling each feature so that they are all approximately on the same scale. A common rescaling method for kernel SVMs is to scale the data such that all features are between 0 and 1. We will see how to do this using the MinMaxScaler preprocessing method:
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

svc = SVC()
svc.fit(X_train_scaled, y_train)

print("Accuracy on training set: {:.3f}".format(svc.score(X_train_scaled, y_train)))
print("Accuracy on test set: {:.3f}".format(svc.score(X_test_scaled, y_test)))


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
Going from left to right, we increase the value of the parameter gamma from 0.1 to 10. A small gamma means a large radius for the Gaussian kernel, which means that many points are considered close by. This is reflected in very smooth decision boundaries on the left, and boundaries that focus more on single points further to the right. A low value of gamma means that the decision boundary will vary slowly, which yields a model of low complexity, while a high value of gamma yields a more complex model.
Going from top to bottom, we increase the C parameter from 0.1 to 1000. As with the linear models, a small C means a very restricted model, where each data point can only have very limited influence. You can see that at the top left the decision boundary looks nearly linear, with the misclassified points barely having any influence on the line. Increasing C, as shown on the bottom right, allows these points to have a stronger influence on the model and makes the decision boundary bend to correctly classify them.
Let’s apply the RBF kernel SVM to the Breast Cancer dataset. By default, C=1 and gamma=1/n_features:
Can you write Python code for it?

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

cancer = load_breast_cancer()

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, random_state=0)

svc = SVC()
svc.fit(X_train, y_train)

print("Accuracy on training set: {:.2f}".format(svc.score(X_train, y_train)))
print("Accuracy on test set: {:.2f}".format(svc.score(X_test, y_test)))


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
In this case, the SVM yields a very smooth and nonlinear (not a straight line) boundary. We adjusted two parameters here: the C parameter and the gamma parameter, which we will now discuss in detail.
Tuning SVM parameters
The gamma parameter is the one shown in the formula given in the previous section, which controls the width of the Gaussian kernel. It determines the scale of what it means for points to be close together. The C parameter is a regularization parameter, similar to that used in other models.  It limits the importance of each point (or more precisely, their dual_coef_).
Let’s have a look at what happens when we vary these parameters:
Can you write Python code for it?

def plot_svm(log_C, log_gamma, ax=None):
    X, y = make_handcrafted_dataset()
    C = 10. ** log_C
    gamma = 10. ** log_gamma
    svm = SVC(kernel='rbf', C=C, gamma=gamma).fit(X, y)
    if ax is None:
        ax = plt.gca()
    plot_2d_separator(svm, X, ax=ax, eps=.5)
    # plot data
    discrete_scatter(X[:, 0], X[:, 1], y, ax=ax, c=c)
    # plot support vectors
    sv = svm.support_vectors_
    # class labels of support vectors are given by the sign of the dual coefficients
    sv_labels = svm.dual_coef_.ravel() > 0
    discrete_scatter(sv[:, 0], sv[:, 1], sv_labels, s=15, markeredgewidth=3, ax=ax, c=c)
    ax.set_title("C = %.4f gamma = %.4f" % (C, gamma))

fig, axes = plt.subplots(3, 3, figsize=(15, 10))

for ax, C in zip(axes, [-1, 0, 3]):
    for a, gamma in zip(ax, range(-1, 2)):
        plot_svm(log_C=C, log_gamma=gamma, ax=a)

axes[0, 0].legend(["class 0", "class 1", "sv class 0", "sv class 1"], ncol=4, loc=(.9, 1.2))


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
In this figure, we have the first feature of the dataset on the x-axis and the second feature on the y-axis. We display the decision boundary found by LinearSVC as a straight line, separating the area classified as class 1 on the top from the area classified as class 0 on the bottom. In other words, any new data point that lies above the black line will be classified as class 1 by the respective classifier, while any point that lies below the black line will be classified as class 0.
Note that the model misclassifies two of the points. By default, LinearSVC applies an L2 regularization, in the same way that Ridge does for regression.
For LinearSVC the trade-off parameter that determines the strength of the regularization is called C, and higher values of C correspond to less regularization. In other words, when you use a high value for the parameter C, LinearSVC tries to fit the training set as best as possible, while with low values of the parameter C, the model puts more emphasis on finding a coefficient vector (w) that is close to zero.
There is another interesting aspect of how the parameter C acts. Using low values of C will cause the algorithm to try to adjust to the “majority” of data points, while using a higher value of C stresses the importance that each individual data point be classified correctly. Here is an illustration:
Can you write Python code for it?

from sklearn.svm import SVC

def plot_linear_svc_regularization():
    X, y = make_blobs(centers=2, random_state=4, n_samples=30)
    fig, axes = plt.subplots(1, 3, figsize=(12, 4))

    # a carefully hand-designed dataset lol
    y[7] = 0
    y[27] = 0
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5

    for ax, C in zip(axes, [1e-2, 1, 1e2]):
        discrete_scatter(X[:, 0], X[:, 1], y, ax=ax)

        svm = SVC(kernel='linear', C=C, tol=0.00001).fit(X, y)
        w = svm.coef_[0]
        a = -w[0] / w[1]
        xx = np.linspace(6, 13)
        yy = a * xx - (svm.intercept_[0]) / w[1]
        ax.plot(xx, yy, c='k')
        ax.set_xlim(x_min, x_max)
        ax.set_ylim(y_min, y_max)
        ax.set_xticks(())
        ax.set_yticks(())
        ax.set_title("C = %f" % C)
    axes[0].legend(loc="best")

plot_linear_svc_regularization()


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
Kernelized Support Vector Machines
Kernelized support vector machines (often just referred to as SVMs) are an extension that allows for more complex models that are not defined simply by hyperplanes in the input space. While there are support vector machines for classification and regression, we will restrict ourselves to the classification case, as implemented in SVC. Similar concepts apply to support vector regression, as implemented in SVR.
The math behind kernelized support vector machines is a bit involved. You can find the details in Chapter 1 of Hastie, Tibshirani, and Friedman’s The Elements of Statistical Learning.
Linear models and nonlinear features
As you saw above, linear models can be quite limiting in low-dimensional spaces, as lines and hyperplanes have limited flexibility. One way to make a linear model more flexible is by adding more features—for example, by adding interactions or polynomials of the input features.
Let’s look at another synthetic dataset to help illustrate this:
Can you write Python code for it?

c=['b', 'r', 'g']
X, y = make_blobs(centers=4, random_state=8)
y = y % 2

discrete_scatter(X[:, 0], X[:, 1], y, c=c)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
Now let’s expand the set of input features, say by also adding feature1^2, the square of the second feature, as a new feature. Instead of representing each data point as a two-dimensional point, (feature0, feature1), we now represent it as a three-dimensional point, (feature0, feature1, feature1^22). This new representation is illustrated below in a three-dimensional scatter plot:
Can you write Python code for it?

# add the squared first feature
X_new = np.hstack([X, X[:, 1:] ** 2])

from mpl_toolkits.mplot3d import Axes3D, axes3d
figure = plt.figure()
# visualize in 3D
ax = Axes3D(figure, elev=-152, azim=-26)
# plot first all the points with y == 0, then all with y == 1
mask = y == 0
ax.scatter(X_new[mask, 0], X_new[mask, 1], X_new[mask, 2], c='b', cmap=cm2, s=60)
ax.scatter(X_new[~mask, 0], X_new[~mask, 1], X_new[~mask, 2], c='r', marker='^', cmap=cm2, s=60)
ax.set_xlabel("feature0")
ax.set_ylabel("feature1")
ax.set_zlabel("feature1 ** 2")


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
Now, we train a LinearSVC classifier on the dataset:
Can you write Python code for it?

linear_svm = LinearSVC().fit(X, y)
print("Coefficient shape: ", linear_svm.coef_.shape)
print("Intercept shape: ", linear_svm.intercept_.shape)


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
Scaling the data made a huge difference! Now we are actually in an underfitting regime, where training and test set performance are quite similar but less close to 100% accuracy. From here, we can try increasing either C or gamma to fit a more complex model. For example:
Can you write Python code for it?

svc = SVC(C=1000)
svc.fit(X_train_scaled, y_train)

print("Accuracy on training set: {:.3f}".format(svc.score(X_train_scaled, y_train)))
print("Accuracy on test set: {:.3f}".format(svc.score(X_test_scaled, y_test)))


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
The kernel trick
The lesson here is that adding nonlinear features to the representation of our data can make linear models much more powerful. However, often we don’t know which features to add, and adding many features (like all possible interactions in a 100-dimensional feature space) might make computation very expensive. Luckily, there is a clever mathematical trick that allows us to learn a classifier in a higher-dimensional space without actually computing the new, possibly very large representation. This is known as the kernel trick, and it works by directly computing the distance (more precisely, the scalar products) of the data points for the expanded feature representation, without ever actually computing the expansion.
There are two ways to map your data into a higher-dimensional space that are commonly used with support vector machines: the polynomial kernel, which computes all possible polynomials up to a certain degree of the original features (like feature1^2 x feature2^5); and the radial basis function (RBF) kernel, also known as the Gaussian kernel. The Gaussian kernel is a bit harder to explain, as it corresponds to an infinite-dimensional feature space. One way to explain the Gaussian kernel is that it considers all possible polynomials of all degrees, but the importance of the features decreases for higher degrees.
In practice, the mathematical details behind the kernel SVM are not that important, though, and how an SVM with an RBF kernel makes a decision can be summarized quite easily—we’ll do so in the next section.
Understanding SVMs
During training, the SVM learns how important each of the training data points is to represent the decision boundary between the two classes. Typically only a subset of the training points matter for defining the decision boundary: the ones that lie on the border between the classes. These are called support vectors and give the support vector machine its name.
To make a prediction for a new point, the distance to each of the support vectors is measured. A...
Can you write Python code for it?

def make_handcrafted_dataset():
    # a carefully hand-designed dataset lol
    X, y = make_blobs(centers=2, random_state=4, n_samples=30)
    y[np.array([7, 27])] = 0
    mask = np.ones(len(X), dtype=np.bool)
    mask[np.array([0, 1, 5, 26])] = 0
    X, y = X[mask], y[mask]
    return X, y

from sklearn.svm import SVC
X, y = make_handcrafted_dataset()
svm = SVC(kernel='rbf', C=10, gamma=0.1).fit(X, y)
plot_2d_separator(svm, X, eps=.5)
discrete_scatter(X[:, 0], X[:, 1], y, c=c)
# plot support vectors
sv = svm.support_vectors_
# class labels of support vectors are given by the sign of the dual coefficients
sv_labels = svm.dual_coef_.ravel() > 0
discrete_scatter(sv[:, 0], sv[:, 1], sv_labels, s=15, markeredgewidth=3, c=c)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
We can apply the LinearSVC model to a synthetic dataset, and visualize the decision boundary:
Can you write Python code for it?

from sklearn.svm import LinearSVC

X, y = make_forge()
model = LinearSVC()
clf = model.fit(X,y)
plot_2d_separator(clf, X, fill=False, eps=0.5, alpha=.7)
discrete_scatter(X[:, 0], X[:, 1], y)
plt.title("{}".format(clf.__class__.__name__))
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")
plt.legend()


I am working on the file "SL6_Kernel_Methods_and_SVMs.ipynb".
The first task is:
We see that the shape of the coef_ is (3, 2), meaning that each row of coef_ contains the coefficient vector for one of the three classes and each column holds the coefficient value for a specific feature (there are two in this dataset). The intercept_ is now a one-dimensional array, storing the intercepts for each class.
Let’s visualize the lines given by the three binary classifiers:
Can you write Python code for it?

discrete_scatter(X[:, 0], X[:, 1], y, c=['b', 'r', 'g'])
line = np.linspace(-15, 15)
for coef, intercept, color in zip(linear_svm.coef_, linear_svm.intercept_, ['b', 'r', 'g']):
    plt.plot(line, -(line * coef[0] + intercept) / coef[1], c=color)
plt.ylim(-10, 15)
plt.xlim(-10, 8)
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")
plt.legend(['Class 0', 'Class 1', 'Class 2', 'Line class 0', 'Line class 1', 'Line class 2'], loc=(1.01, 0.3))


I am working on the file "SL9_Bayseian_Learning.ipynb".
The first task is:
Here, we have four data points, with four binary features each. There are two classes, 0 and 1. For class 0 (the first and third data points), the first feature is zero two times and nonzero zero times, the second feature is zero one time and nonzero one time, and so on. These same counts are then calculated for the data points in the second class. Counting the nonzero entries per class in essence looks like this:
Can you write Python code for it?

counts = {}
for label in np.unique(y):
    # iterate over each class
    # count (sum) entries of 1 per feature
    counts[label] = X[y == label].sum(axis=0)
print("Feature counts:\n{}".format(counts))


I am working on the file "SL9_Bayseian_Learning.ipynb".
The first task is:
The other two naive Bayes models, MultinomialNB and GaussianNB, are slightly different in what kinds of statistics they compute. MultinomialNB takes into account the average value of each feature for each class, while GaussianNB stores the average value as well as the standard deviation of each feature for each class.
To make a prediction, a data point is compared to the statistics for each of the classes, and the best matching class is predicted. Interestingly, for both MultinomialNB and BernoulliNB, this leads to a prediction formula that is of the same form as in the linear models. Unfortunately, coef_ for the naive Bayes models has a somewhat different meaning than in the linear models, in that coef_ is not the same as w.
GaussianNB
Let's apply a GaussianNB model to the Iris dataset:
Can you write Python code for it?

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

iris = datasets.load_iris()

X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=5)

gnb = GaussianNB()
gnb.fit(X_train, y_train)

print("Accuracy on training set: {:.2f}".format(gnb.score(X_train, y_train)))
print("Accuracy on test set: {:.2f}".format(gnb.score(X_test, y_test)))


I am working on the file "SLN.ipynb".
The first task is:
Assortativity
The average degree connectivity is the average nearest neighbor degree of nodes with degree k.
Can you write Python code for it?

print("Nodes with highest average degree connectivity: \n")
adc_high = sort_dictionary_by_value_desc(assort.average_degree_connectivity(graph))
adc_high_count = Counter(adc_high)
for k, v in adc_high_count.most_common(5):
    print('k=%i: %f\n' % (k, v))


I am working on the file "SLN.ipynb".
The first task is:
Betweenness
Computing the shortest-path betweenness centrality for nodes. Betweenness centrality of a node v is the sum of the fraction of all-pairs shortest paths that pass through v. <br/>
Betweenness centrality quantifies the number of times a node actts as a bridge along the shortest path between two other nodes.
Can you write Python code for it?

print("Nodes with lowest betweenness centrality: \n")
bc_low = sort_dictionary_by_value_asc(central.betweenness_centrality(graph))
bc_low_count = islice(bc_low.items(), 0, 5)
for k, v in bc_low_count:
    print('%s: %f\n' % (num_doc_mapper[k], v))


I am working on the file "SLN.ipynb".
The first task is:
Closeness
Closenes centrality of a node u is the reciprocal of the sum of the shortest path distances form u to all n-1 other nodes. Thus the more central a node is, the closer it is to all other nodes.
Can you write Python code for it?

print("Nodes with highest closeness centrality: \n")
cc_high = sort_dictionary_by_value_desc(central.closeness_centrality(graph))
cc_high_count = Counter(cc_high)
for k, v in cc_high_count.most_common(5):
    print('%s: %f\n' % (num_doc_mapper[k], v))


I am working on the file "SLN.ipynb".
The first task is:
Clustering
In graph theory, a clustering coefficient is a measure of the degree to which nodes in a graph tend to cluster together. The global version of this measure was designed to give an overall indication of the clustering in the network.

The global clustering coefficient is based on triplets of nodes. A triplet consists of three connected nodes. A triangle therefore includes three closed triplets, one centered on each of  the nodes. The global clustering coefficient is the number of closed triplets (or 3 x triangles) over the total number of triplets (both open and closed):

C = 3 x number of triangles / number of connected triplets of vertices = number of closed triplets / number of connected triplets of vertices.
Can you write Python code for it?

print("Global Clustering Coefficient: {0}".format(nx.transitivity(graph)))


I am working on the file "SLN.ipynb".
The first task is:
Community Detection
Latapy and Pons algorithm for community detection base on random walks. This algorithm (Walktrap) can be adatped to directed edges, weights, and can alter resolution by walk length.
Basic idea: Simulate many short random walks on the network and compute pairwise similarity measures bassed on these walks. Use these similarity values to aggreggate vertices into communities.
Time Complexity: depends on walk length, O(|V|^2 log |V|) typically.
Can you write Python code for it?

dendogram = this_igraph.community_walktrap(steps=10)
clusters = dendogram.as_clustering()
membership = clusters.membership
memberships_dict = {}
for name, membership in zip(this_igraph.vs["name"], membership):
    if membership not in memberships_dict:
        memberships_dict[membership] = []
    memberships_dict[membership].append(num_doc_mapper[int(name)])
print("Total number of communities: {0}".format(len(memberships_dict)))
# number of members
memership_nums = {}
for k in memberships_dict:
    memership_nums[k] = len(memberships_dict[k])

# top communities
print("Communities with most members: \n")
mem_num_high = sort_dictionary_by_value_desc(memership_nums)
mem_num_high_count = Counter(mem_num_high)
for k, v in mem_num_high_count.most_common(5):
    print('\nCommunity %i with %i members:' % (k, v))
    #print(memberships_dict[k])

# bottom communities
print("Communities with least members: \n")
mem_num_low = sort_dictionary_by_value_asc(memership_nums)
mem_num_low_count = islice(mem_num_low.items(), 0, 5)
for k, v in mem_num_low_count:
    print('\nCommunity %i with %i members:' % (k, v))
    print(memberships_dict[k])


I am working on the file "SLN.ipynb".
The first task is:
Cycles
Cycle found via a directed, depth-first traversal.
Can you write Python code for it?

print("Cycles: \n")
cycles = nx.algorithms.find_cycle(graph)
for cycle in cycles:
    print('%s --> %s\n' % (num_doc_mapper[cycle[0]], num_doc_mapper[cycle[1]]))


I am working on the file "SLN.ipynb".
The first task is:
Katz
Katz centrality computes the centrality for a node based on the centrality of its neigbors. It is a generalization of the eigenvector centrality.
Can you write Python code for it?

# highest K
print("Nodes with highest Katz centrality: \n")
k_high = sort_dictionary_by_value_desc(nx.katz_centrality_numpy(graph))
k_high_count = Counter(k_high)
for k, v in k_high_count.most_common(10):
    print ('%s: %f\n' % (num_doc_mapper[k], v))

# lowest Katz
print("Nodes with lowest Katz centrality: \n")
k_low = sort_dictionary_by_value_asc(nx.katz_centrality_numpy(graph))
k_low_count = islice(k_low.items(), 0, 10)
for k, v in k_low_count:
    print('%s: %f\n' % (num_doc_mapper[k], v))


I am working on the file "SLN.ipynb".
The first task is:
Network Science Measures
Can you write Python code for it?

# reading num - name dictionary
num_doc_mapper = load_num_doc('dataset/new_data/new_num_doc_sorted.txt')

# reading and creating network using networkx
graph = nx.read_edgelist('dataset/new_data/all_num_lines.txt', create_using=nx.DiGraph(), nodetype=int)
print(nx.info(graph))

# highest degrees
print("Nodes with highest degrees: (in + out)\n")
degrees_high = sort_dictionary_by_value_desc(graph.degree())
degrees_high_count = Counter(degrees_high)
for k, v in degrees_high_count.most_common(5):
    print('%s: %i (%i + %i)\n' % (num_doc_mapper[k], v, graph.in_degree(k), graph.out_degree(k)))

# lowest degrees
print("Nodes with lowest degrees: (in + out)\n")
deegrees_low = sort_dictionary_by_value_asc(graph.degree())
deegrees_low_count = islice(deegrees_low.items(), 0, 5)
for k, v in deegrees_low_count:
    print('%s: %i (%i + %i)\n' % (num_doc_mapper[k], v, graph.in_degree(k), graph.out_degree(k)))

# highest in_degrees
print("Nodes with highest in degrees:\n")
in_degrees_high = sort_dictionary_by_value_desc(graph.in_degree())
in_degrees_high_count = Counter(in_degrees_high)
for k, v in in_degrees_high_count.most_common(5):
    print('%s: %i\n' % (num_doc_mapper[k], v))

# highest out_degrees
print("Nodes with highest out degrees:\n")
out_degrees_high = sort_dictionary_by_value_desc(graph.out_degree())
out_degrees_high_count = Counter(out_degrees_high)
for k, v in out_degrees_high_count.most_common(5):
    print('%s: %i\n' % (num_doc_mapper[k], v))


I am working on the file "SLN.ipynb".
The first task is:
Page Rank
PageRank computes a ranking of the nodes in the graph based on the structure of the incoming links.
Can you write Python code for it?

# highest PR
print("Nodes with highest PageRank values: \n")
pr_high = sort_dictionary_by_value_desc(nx.algorithms.pagerank(graph))
pr_high_count = Counter(pr_high)
for k, v in pr_high_count.most_common(10):
    print('%s: %f\n' % (num_doc_mapper[k], v))

# lowest PR
print("Nodes with lowest PageRank values: \n")
pr_low = sort_dictionary_by_value_asc(nx.algorithms.pagerank(graph))
pr_low_count = islice(pr_low.items(), 0, 10)
for k,  v in pr_low_count:
    print('%s: %f\n' % (num_doc_mapper[k], v))


I am working on the file "SLN.ipynb".
The first task is:
The diameter of a graph is the maximum eccentricity of any vertex in the graph. That is, d is the greatest distance between any pair of vertices.

For directed graphs, the graph density is defined as D = |E| / (|V| (|V| - 1)) where E is the number of edges and V is the number of vertices in the graph.
Can you write Python code for it?

print("Graph diameter: {0}".format(this_igraph.diameter()))
print("Average path length: {0}".format(this_igraph.average_path_length()))
print("Graph density: {0}".format(this_igraph.density()))


I am working on the file "SNAP_Python_Tutorial_1/SNAP-Python_Tutorial_1.ipynb".
The first task is:
Now the data product can be easily readed  in Python if its folder have been stored in the current working  directory of his Jupyter session. In this case we can operate as follows:
Can you write Python code for it?

file_path = 'C:\Program Files\snap\S2A_MSIL1C_20170202T090201_N0204_R007_T35SNA_20170202T090155.SAFE\MTD_MSIL1C.xml'
product = ProductIO.readProduct(file_path)


I am working on the file "SNAP_Python_Tutorial_1/SNAP-Python_Tutorial_1.ipynb".
The first task is:
The data of Band 5 are now completely available and it is then possible to go on and try to visualize its related image directly in Python. Python can easily handle images as bidimensional arrays with the Numpy library, and the snappy module allows us to store the band data in a numpy array thanks to the specific method readPixels.
Can you write Python code for it?

import numpy as np
B5_data = np.zeros(Width*Height, dtype = np.float32)
B5.readPixels(0,0,Width,Height,B5_data)
B5_data.shape = (Height,Width)


I am working on the file "SNAP_Python_Tutorial_1/SNAP-Python_Tutorial_1.ipynb".
The first task is:
The image obtained is the same of the SNAP toolbox view showed at the beginning, but there is noticeable difference in the contrast between the two results. It is possible to apply some contrast enhancement technique to process our last image so that we obtain the same result of the desktop SNAP. A very useful Python package that allows to play with contrast enhancement is surely the scikit-image package.
Can you write Python code for it?

import skimage.exposure as exposure
val1,val2 = np.percentile(B5_data, (2.5,97.5))
B5_data_new = exposure.rescale_intensity(B5_data, in_range=(val1,val2))

plt.figure(figsize=(8, 8))
fig = plt.imshow(B5_data_new, cmap = cm.gray)
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.show()


I am working on the file "SNAP_Python_Tutorial_1/SNAP-Python_Tutorial_1.ipynb".
The first task is:
This operation should have last some seconds and at the end the user could try to verify the shape of the B5_data array to confirm that everything has been done correctly so far. B5_data is now a bidimensional array that can be handled with every type of operation or image processing technique. In this case we are just going to visualize the band as an image in its grey-scaled version. The image will be displayed using the Matplotlib package:
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib.cm as cm
%matplotlib inline
plt.figure(figsize=(8, 8))                 # adjusting the figure window size
fig = plt.imshow(B5_data, cmap = cm.gray)  #matplotlib settings for the current image
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.show()


I am working on the file "SNAP_Python_Tutorial_2/SNAP-Python_Tutorial_2.ipynb".
The first task is:
So the resolution in this specific case will be 20 meters per pixel. After the parameter definition it is possible to invoke the resampling operator using a syntax that is the same for all the GPF operators:
createProduct(String operatorName,Map(String,Object) parameters,Product sourceProduct)
and the Python implementation is given in the following line:
Can you write Python code for it?


I am working on the file "SNAP_Python_Tutorial_2/SNAP-Python_Tutorial_2.ipynb".
The first task is:
The obtained results confirm that the two bands have been detected with different resolutions and if the user wanted to implement some processing that involves those bands, he should first operate a RESAMPLING of the S-2 product, according to a selected pixel resolution. The resampling operation can be directly executed in SNAP because it is included in the SNAP Graph Processing Framework(GPF), a wide collection of data processors that can be applied to a Sentinel data product. Each data processor is called a GPF operator and it can be invoked in the desktop version of SNAP, in Python with the snappy module, or directly in the Windows/Linux command-line.
The resampling operation is a typical example of a GPF operator because it is provided with a dedicated user interface that is available in the desktop version of SNAP. It is really important to look at the input parameters that must be set when the user wants to invoke a specific GPF operator. As for most of the GPF operators, also for the RESAMPLE operator the list of input parameters can be found in its user interface, as follows:
<img src="Resampling_list.jpg">
The displayed list can change depending on the type of resampling the user wants to implement; the available options can be found in the desktop version of SNAP, looking into the user interface of the operator.
In Python a GPF operator can be invoked only after the definition of the list of input parameters , using a Java HashMap object (java.util.HashMap). For this reason, whenever the user wants to work with a GPF operator he must always import that Java class in Python using the Jpy module:
Can you write Python code for it?

from snappy import jpy
HashMap = snappy.jpy.get_type('java.util.HashMap')


I am working on the file "SNAP_Python_Tutorial_2/SNAP-Python_Tutorial_2.ipynb".
The first task is:
The output variable is a new data product and all its bands are now represented with the same resolution. As a test to confirm the successful operation it is possible to look again at band 4 and 5 to see what happened with them:
Can you write Python code for it?

B4 = result.getBand('B4')
B5 = result.getBand('B5')

Width_4 = B4.getRasterWidth()
Height_4 = B4.getRasterHeight()
print("Band 4 Size: " + str(Width_4) +','+ str(Height_4))

Width_5 = B5.getRasterWidth()
Height_5 = B5.getRasterHeight()
print("Band 5 Size: " + str(Width_5) +','+ str(Height_5))


I am working on the file "SNAP_Python_Tutorial_3/SNAP-Python_Tutorial_3.ipynb".
The first task is:
Now sub_product is the new data product obtained from the source S-2 product. We are going to verify the result by showing the list of available bands, and visualizing one of them.
Can you write Python code for it?

list(result.getBandNames())

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
%matplotlib inline

sub_b6 = sub_product.getBand('B6')
print("band read")
width = sub_b6.getRasterWidth()
height = sub_b6.getRasterHeight()
print("subset size : ", width, height)
sub_b6_data = np.zeros(width*height, dtype = np.float32)
sub_b6.readPixels(0,0,width,height, sub_b6_data)
sub_b6_data.shape = height, width
print("reading data...")
plt.figure(1)
fig = plt.imshow(sub_b6_data, cmap = cm.gray)
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.show()


I am working on the file "SNAP_Python_Tutorial_3/SNAP-Python_Tutorial_3.ipynb".
The first task is:
Once the geometry has been  properly defined, the subset operator can be invoked with its object constructor, and then it will be possible to call its methods that will allow to set the geometry object and the source product:
Can you write Python code for it?

op = SubsetOp()
op.setSourceProduct(result)
op.setGeoRegion(geometry)
sub_product = op.getTargetProduct()
print("subset product ready")


I am working on the file "SNAP_Python_Tutorial_3/SNAP-Python_Tutorial_3.ipynb".
The first task is:
The user could now verify some property of the result, for example extracting the raster size of one band or getting the list of all the bands.
The obtained product is now ready for the extraction of the subset. The next step is the definition of the subset operator to extract a region of interest from the whole S-2 product:
<img src="SNAP_subset.jpg">
The subset operator has to be set in a different way, if compared with the resampling operator. First of all it is necessary to invoke the operator and to define the geometry of the subset, that should be described according to the Geometry Java class (com.vividsolutions.jts.geom.Geometry). The coordinates of the polygon can be extracted directly from the desktop version of SNAP full view using other GIS applications.
SubsetOp = jpy.get_type('org.esa.snap.core.gpf.common.SubsetOp')
WKTReader =jpy.get_type('com.vividsolutions.jts.io.WKTReader')
Now for the implementation of the operator the geometry has to be defined and associated to the object of class Geometry:
Can you write Python code for it?


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
14% encounters that have both records got a different diagnosis
Can you write Python code for it?

temp = d4[d4.sno_diagnosis.notnull()]
print(temp[temp.vision_class!=temp.sno_diagnosis].shape[0]/float(temp.shape[0]))
print(temp[temp.vision_class!=temp.sno_diagnosis].shape[0])
print(temp.shape[0])


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
Extract the encounters that are diagnosed as no_DR, no_vision_threatening and vision_threatening with SNOMED codes
Can you write Python code for it?

d3 = d2[d2.sno_diagnosis.notnull()][['Person_Nbr', 'Enc_Nbr', 'sno_diagnosis']].copy()
d3.head()


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
Less than 100 patients diagnosed as some kind of vision class got a different diagnosis with snomed
Can you write Python code for it?

temp = d5[d5.worst_vis_icd!=0]
temp = temp[temp.worst_vis_sno!=0]
temp[temp.worst_vis_icd!=temp.worst_vis_sno].shape[0]

temp = d5[d5.recent_vis_icd!=0]
temp = temp[temp.recent_vis_sno!=0]
temp[temp.recent_vis_icd!=temp.recent_vis_sno].shape[0]


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
Merge snomed-diagnosed encounters to the original encoutners by Person_Nbr and Enc_Nbr
Can you write Python code for it?

d4 = d4.merge(d3, left_on = ['Person_Nbr', 'Enc_Nbr'], right_on = ['Person_Nbr', 'Enc_Nbr'], how='left')
d4.head()


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
Transfer the original DR-diagnosis to vision-related class
Can you write Python code for it?

d4 = data['all_encounter_data'][['Person_Nbr', 'DR_diagnosis', 'Enc_Date']].copy()
d4['Enc_Nbr'] = d4.index
mapping = {
    'no_DR': 0,
    'mNPDR': 1,
    'MNPDR': 1,
    'SNPDR': 2,
    'PDR':2
}
d4['vision_class'] = d4.DR_diagnosis.map(lambda x: mapping.get(x))
d4.head()


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
Tried to merge the snomed_problem_list with all_encounter_data by date.
Can you write Python code for it?

import datetime

d = data['all_encounter_data'][['Enc_Date', 'Person_Nbr']].copy()
d['Date'] = d.Enc_Date.map(lambda x: datetime.datetime(x.year, x.month, x.day).strftime('%Y-%m-%d'))
d['Enc_Nbr'] = d.index
d.head()

d.shape[0]

import pandas as pd
sno = pd.read_csv('snocodeTally.csv')
snocode = {k:list(v.Class)[0] for k,v in sno.groupby('SNOMED code')}

d1 = data['SNOMED_problem_list']
d1['Date'] = d1.Date_Created.map(lambda x: datetime.datetime(x.year, x.month, x.day).strftime('%Y-%m-%d'))
d1.head()

d1['sno_diagnosis'] = d1.Concept_ID.map(lambda x: snocode.get(x, float('nan')))
d1 = d1[d1.sno_diagnosis.notnull()]
d1.shape[0]


I am working on the file "SNOMED_exploration/SNOMED_exploration.ipynb".
The first task is:
With ICD we had in total 4030 patients with DR, and with SNOMED we had in total 1053 patients with DR
Can you write Python code for it?

d5[d5.worst_vis_icd!=0].shape[0]

d5[d5.recent_vis_icd!=0].shape[0]

d5[d5.worst_vis_sno!=0].shape[0]

d5[d5.recent_vis_sno!=0].shape[0]


I am working on the file "SNP-to-Gene.ipynb".
The first task is:
Compute SNPs per gene using bedtools
Can you write Python code for it?

platforms = 'hh550', 'ho1', 'affy500', 'exac', 'kg3'

# Base pairs added upstream and downstream of each entry
window = 10000

# Count the overlaps for genotyping platforms
for platform in platforms:
    ! bedtools window -w {window} -c -a download/entrez-genes.bed \
        -b download/platforms/bed/{platform}.bed.gz > data/platforms/{platform}-entrez.bed


I am working on the file "SNP-to-Gene.ipynb".
The first task is:
Create BAM Files for SNPs for genotyping chips
Can you write Python code for it?

# Download genotyping platform SNPs
adf_files = [
    'A-GEOD-8882/A-GEOD-8882.adf.txt',
    'A-GEOD-6434/A-GEOD-6434.adf.txt',
    'A-AFFY-107/A-AFFY-107.adf.txt',
    'A-AFFY-72/A-AFFY-72.adf.txt',
]
base_url = 'http://www.ebi.ac.uk/arrayexpress/files'
for adf_file in adf_files:
    ! wget --no-verbose --timestamping --directory-prefix download/chips/adf {base_url}/{adf_file}
    ! gzip --force download/chips/{adf_file.rsplit('/', 1)[-1]}

def parse_adf(path):
    """Parse array description files from ArrayExpress."""
    with gzip.open(path, 'rt') as read_file:
        for i, line in enumerate(read_file):
            if line.strip() == '[main]':
                skip = i + 1
                break
    return pandas.read_table(path, skiprows=skip)

#Illumina HumanHap550 Platform: https://www.ebi.ac.uk/arrayexpress/arrays/A-GEOD-6434/
df = parse_adf('download/chips/A-GEOD-6434.adf.txt.gz')
rsids = df['Reporter Database Entry [dbsnp]']
rsids = rsids[rsids.str.startswith('rs').fillna(False)]
with open("download/chips/hh550-snp-for-bed.txt", 'w') as write_file:
    write_file.write('\n'.join(rsids))
'{} SNPs with rs#s on HumanHap550'.format(len(rsids))

#Illumina HumanOmni1-Quad: https://www.ebi.ac.uk/arrayexpress/arrays/A-GEOD-8882/
df = parse_adf('download/chips/A-GEOD-8882.adf.txt.gz')
rsids = df['Reporter Database Entry [dbsnp]']
rsids = rsids[rsids.str.startswith('rs').fillna(False)]
with open('download/chips/ho1-snp-for-bed.txt', 'w') as write_file:
    write_file.write('\n'.join(rsids))
'{} SNPs with rs#s on HumanOmni1'.format(len(rsids))

#Affy 500k Array set is made up of 2 arrays (Sty and Nsp)

#Sty Array: https://www.ebi.ac.uk/arrayexpress/arrays/A-AFFY-72/
df = parse_adf('download/chips/A-AFFY-72.adf.txt.gz')
rsids = df['Composite Element Database Entry[ncbi_dbsnp:126:126]']
rsids_0 = rsids[rsids.str.startswith('rs').fillna(False)]

#Nsp Array: https://www.ebi.ac.uk/arrayexpress/arrays/A-AFFY-107/
df = parse_adf('download/chips/A-AFFY-107.adf.txt.gz')
rsids = df['Composite Element Database...


I am working on the file "SNP-to-Gene.ipynb".
The first task is:
Manual step: Follow instructions at http://www.gettinggeneticsdone.com/2011/06/mapping-snps-to-genes-for-gwas.html to get a BED file. Use GRCh37/hg19 and dbSNP Common SNPs(142). Name the downloaded files hh550.bed, ho1.bed, and affy500k.bed. Move these files to download/bed and gzip.
Download 1000 Genomes Phase 3 common SNPs
Can you write Python code for it?

url = 'https://raw.githubusercontent.com/dhimmel/kg/de9c303442f01acde89aa956acb2f53888295169/data/common-SNPs/common-snps.bed.gz'
! wget --no-verbose --output-document download/platforms/bed/kg3.bed.gz {url}


I am working on the file "SNR_SINAD_THD.ipynb".
The first task is:
Signal generation and specturm estimation
We are generating a sampled complex sine wave with added white noise, DC offset and distortion. Alternatively, you can load a recorded RF signal (with the same FS and FREQ parameters).
We intentionally avoided the built-in spectrum estimation methods (e.g. scipy.signal.periodogram) to show the details of the process. NOTE: the result vector is power spectrum, thus each element represents the total / integrated power within the bin.
Can you write Python code for it?

# Option 1: generate signal
t = np.linspace(0, N / FS, N)
signal = A_SIGNAL * np.exp(2j * np.pi * FREQ * t)
noise = A_NOISE * (np.random.randn((N)) + 1j * np.random.randn((N)))
dc = A_DC * np.ones((N))
samples = signal + noise + dc

# add third order distortion - harmonics
samples += samples * (samples * samples.conj()) * BETA

# Option 2: load real RF signal (recorded on the SC2 Colosseum)
#samples = np.fromfile('colosseum_tx32_rx16.dat', dtype=np.complex64)

# power spectrum estimation
f = np.fft.fftfreq(N_FFT, 1/FS)
win = get_window('blackmanharris', N_FFT)
norm = N_FFT * np.sqrt(np.sum(win ** 2)  / N_FFT)
offsets = list(range(0, len(samples) - N_FFT, int(N_FFT/ 2))) # half overlap
specgram = np.empty((len(offsets), N_FFT), dtype=np.float)
for i, offset in enumerate(offsets):
    x = np.fft.fft(win * samples[offset:offset+N_FFT], n=N_FFT) / norm
    specgram[i, :] = np.abs(x) ** 2
spectrum = np.mean(specgram, axis=0)
spectrum_db = 10*np.log10(spectrum + EPSILON)

plt.plot(f, spectrum_db)
plt.grid()
plt.xlabel('Frequency (Hz)')
plt.ylabel('Power (dBFS)')
plt.title('Power Spectrum')
plt.show()


I am working on the file "SOAP.ipynb".
The first task is:
2. Construct a global SOAP
Use the atomic environments to construct an average SOAP descriptor for molecules
Can you write Python code for it?

# atomic positions as matrix
molxyz = numpy.load("./data/molecule.coords.npy")
# atom types
moltyp = numpy.load("./data/molecule.types.npy")

atoms_sys = Atoms(positions=molxyz, numbers=moltyp)
view(atoms_sys)

# build SOAP at each atom location
# ...
# compute average soap for each specie
# ...
# concatenate the soaps to the the overall global one
# ...


I am working on the file "SOAP.ipynb".
The first task is:
Recompute SOAP for the same atom, after rotation and show the difference in descriptors:
Can you write Python code for it?

soap2 = soaper.create(NaCl_conv, positions=[calcpos])
print(numpy.linalg.norm(soap1 - soap2))


I am working on the file "SOAP.ipynb".
The first task is:
Remark
The power spectrum at a desired position x is the fingerprint of the local chemical environment at this specific position. Thus, it can be used to:
1. Compare the similarity of two local chemical environments by comparing their SOAP descriptors.
2. Machine learn local properties, like charges, adsorption energies, etc.
Exercises
1. Smoothness
Verify that the SOAP is smooth under translations of point of interest.
Can you write Python code for it?


I am working on the file "SOAP.ipynb".
The first task is:
Rotational invariance
Can you write Python code for it?

#Rotation of positions

print("Original positions:\n {}".format(NaCl_conv.positions))

NaCl_conv.rotate(90, [0,1,1], center=calcpos)

print("Rotated positions:\n {}".format(NaCl_conv.positions))

view(NaCl_conv)


I am working on the file "SOF_Contratos.ipynb".
The first task is:
'Top 10' Contratos de 2017
Agora com a lista de todos os contratos do ano, vamos montar uma tabela e ordenar os dados pelo Valor Principal do Contrato. No Manual da API, aprende-se que esse campo 'valPrincipal'significa o "Valor do contrato sem ocorrência de reajustamentos, ou aditamentos".
Can you write Python code for it?

top10 = df_contratos_empenhados[['txtDescricaoModalidade',
                                 'txtObjetoContrato',
                                 'txtRazaoSocial',
                                 'numCpfCnpj',
                                 'valPrincipal']].sort_values(['valPrincipal'], ascending=False)[:10]

top10


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Agora já temos o DataFrame que junta todos Empenhos do Verde de 2017 com as informações de Contrato, deixando a base mais rica -- e corrigindo essa falha da falta de Razão Social e CNPJ na consulta de contratos!
Can you write Python code for it?


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Agora temos duas bases de dados para trabalhar:
* df_contratos = Primeira base que extraímos, contém todos os contratos (sem Razão Social ou CNPJ, por limitação da API)
* df_empenhos_c_contratos = Base que contém todos os empenhos com código de contrato, depois do cruzamento que fizemos
Ambas têm várias colunas em comum. Então, antes de juntá-las, vamos apenas retirar as colunas únicas nas duas, com o método "difference":
Can you write Python code for it?

cols_to_use = df_empenhos_c_contratos.columns.difference(df_contratos.columns)

df_contratos_empenhados = df_contratos.merge(df_empenhos_c_contratos[cols_to_use], left_index=True, right_index=True, how='outer')


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Atenção para uma distinção conceitual importante!

ÓRGÃO = Administração Direta (Prefeituras Regionais, Secretarias)
EMPRESA = Administração Indireta (Fundações, Empresas Públicas)

Os campos na consulta da API são diferentes. Então se você inserir o Código da Prodam (uma empresa pública) no Campo Órgão, por exemplo, a consulta vai retornar 0 Registros.
Pronto, aí temos a lista completa. Fecha parêntesis!
Seguindo, vamos fazer a consulta dos contratos da Secretaria do Verde e Meio Ambiente -- Órgão 27, conforme a lista acima.
Can you write Python code for it?

url_contratos = '{base_url}/consultaContrato?anoContrato=2017&codOrgao=27'.format(base_url=base_url)

request_contratos = requests.get(url_contratos,
                                headers=headers,
                                verify=True).json()

number_of_pages = request_contratos['metadados']['qtdPaginas']

todos_contratos = []
todos_contratos = todos_contratos + request_contratos['lstContratos']
if number_of_pages>1:
    for p in range(2, number_of_pages+1):
        request_contratos = requests.get(url_contratos + pagination.format(PAGE=p), headers=headers, verify=True).json()
        todos_contratos = todos_contratos + request_contratos['lstContratos']


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Cruzando as bases
Infelizmente, a consulta não vem "pronta". Vamos precisar cruzar com a base de empenhos do primeiro tutorial para conseguir, por exemplo, saber quais são os credores POR CONTRATO. A imagem abaixo foi exibida durante o lançamento da API, no Café Hacker da Controladoria Geral do Município e da Secretaria Municipal da Fazenda, e explica como as consultas se relacionam (o 'Simplificada' é por conta deles, hehe):

O Código de Contrato é um parâmetro facultativo da consulta de empenhos, mas não vem no retorno de dados (outro possível furo da API, que vamos alertar aos desenvolvedores para tentar corrigir). O que isso significa? Que precisaríamos consultar um a um para obter todos os contratos na lista de empenhos! Felizmente, a programação tá aí pra isso. Vamos criar uma estrutura de repetição semelhante à que fizemos para a paginação:
Can you write Python code for it?

url_empenho = '{base_url}/consultaEmpenhos?anoEmpenho=2017&mesEmpenho=08&codOrgao=27'.format(base_url=base_url)
num_contrato = '&codContrato={CONTRATO}'

lista_contratos = list(df_contratos['codContrato'])
len(lista_contratos)

request_empenhos = requests.get(url_empenho,
                                headers=headers,
                                verify=True).json()

def add_codigo_contrato(empenhos, cod_contrato):
    """""
    Adiciona Código de Contrato no dict de cada empenho consultado.
    """""
    for item in empenhos:
        item.update({'codContrato': cod_contrato})
    return empenhos

todos_empenhos = []

todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']

for n in lista_contratos:
    response = requests.get(url_empenho + num_contrato.format(CONTRATO=n), headers=headers, verify=True).json()
    empenhos_c_cod = add_codigo_contrato(response['lstEmpenhos'], n)
    todos_empenhos = todos_empenhos + empenhos_c_cod

df_empenhos_c_contratos = pd.DataFrame(todos_empenhos)


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Passo "Bônus" - Comparando com o Portal da Transparência
Como mencionei lá em cima, o Portal da Transparência de São Paulo tem algo muito importante que é publicar os contratos na íntegra nesta página. Não vou entrar nos detalhes dos poréns que surgem aqui -- mas saiba que a ação de publicar o contrato depende de as pessoas subirem o "anexo" certo na hora de enviar o extrato para o Diário Oficial; o que acaba acontecendo é que são cerca de 700 usuários que fazem isso em toda a prefeitura e os erros são frequentes (ex.: subir arquivo de extrato no local de íntegra; indicar modalidade errada; digitar metadado do valor errado etc etc).
Eu já baixei (e arrumei alguns campos) um arquivo de lá com o mesmo exemplo da SVMA. Vamos comparar com o que vem na API do SOF:
Can you write Python code for it?

df_contratos_portal = pd.read_excel('exemplos/contratos_portal.xls')

df_contratos_portal.sort_values('Valor (R$)', ascending=False).head()

df_contratos_portal.groupby('Modalidade')['Valor (R$)'].sum()


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Passo 3 - Só quer salvar em Excel ou CSV?
Can you write Python code for it?


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Por modalidade de contratação
Can you write Python code for it?


I am working on the file "SOF_Contratos.ipynb".
The first task is:
São muitas as colunas dessa consulta (a visualização acima omite algumas). Vamos checar quais são:
Can you write Python code for it?


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Vamos checar como ficou nossa base de dados de contratos, mostrando os últimos 5 registros:
Can you write Python code for it?


I am working on the file "SOF_Contratos.ipynb".
The first task is:
Vamos remover o txtRazaoSocial, para não nos atrapalhar mais tarde (vamos obter o dado de outro local). Esta consulta não retorna o nome das empresas contratadas, mas "PREFEITURA DE SÃO PAULO" para todos os valores -- provavelmente um erro da API.
Can you write Python code for it?


I am working on the file "SOF_Execucao_Orcamentaria_PMSP.ipynb".
The first task is:
A API fornece apenas uma página na consulta. O script abaixo checa a quantidade de páginas nos metadados da consulta e itera o número de vezes necessário para obter todas as páginas:
Can you write Python code for it?

number_of_pages = request_empenhos['metadados']['qtdPaginas']

todos_empenhos = []
todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']
if number_of_pages>1:
    for p in range(2, number_of_pages+1):
        request_empenhos = requests.get(url_empenho + pagination.format(PAGE=p), headers=headers, verify=True).json()
        todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']

df_empenhos = pd.DataFrame(todos_empenhos)


I am working on the file "SOF_Execucao_Orcamentaria_PMSP.ipynb".
The first task is:
Com os passos acima, fizemos a requisição de todas as páginas e convertemos o arquivo formato json em um DataFrame. Agora podemos trabalhar com a análise desses dado no Pandas. Para checar quantos registros existentes, vamos ver o final da lista:
Can you write Python code for it?


I am working on the file "SOF_Execucao_Orcamentaria_PMSP.ipynb".
The first task is:
Fontes de recursos
Agrupamento dos empenhos por fonte de recursos:
Can you write Python code for it?

fonte = pd.pivot_table(df_empenhos,
                       values=['valLiquidado', 'valPagoExercicio'],
                       index=['txtDescricaoFonteRecurso'],
                       aggfunc=np.sum).sort_values('valPagoExercicio', axis=0, ascending=False, inplace=False, kind='quicksort', na_position='last')

fonte


I am working on the file "SOF_Execucao_Orcamentaria_PMSP.ipynb".
The first task is:
Maiores despesas de 2017
Aqui vamos produzir a lista das 15 maiores despesas da Saúde neste ano:
Can you write Python code for it?

despesas = pd.pivot_table(df_empenhos,
                              values=['valLiquidado', 'valPagoExercicio'],
                              index=['numCpfCnpj', 'txtRazaoSocial', 'txtDescricaoPrograma'],
                              aggfunc=np.sum).sort_values('valPagoExercicio', axis=0, ascending=False, inplace=False, kind='quicksort', na_position='last')

despesas.head(15)


I am working on the file "SOF_Execucao_Orcamentaria_PMSP.ipynb".
The first task is:
Modalidades de Aplicação
Aqui vemos a quantidade de recursos aplicados na Saúde, a título de exemplo, por Modalidade -- se é aplicação na rede direta ou repasse a organizações sociais. Note que o mesmo poderia ser feito para qualquer órgão, ou mesmo para a Prefeitura como um todo:
Can you write Python code for it?

modalidades = df_empenhos.groupby('txtModalidadeAplicacao')['valTotalEmpenhado', 'valLiquidado'].sum()

modalidades

# Outra maneira de fazer a mesma operação:
#pd.pivot_table(df_empenhos, values='valTotalEmpenhado', index=['txtModalidadeAplicacao'], aggfunc=np.sum)


I am working on the file "SOF_Execucao_Orcamentaria_PMSP.ipynb".
The first task is:
Passo 4. Quer salvar um csv?
O objetivo deste tutorial não era fazer uma análise exaustiva da base, mas apenas mostrar o que é possível a partir do consumo da API. Você também pode salvar toda a base de empenhos num arquivo .csv e trabalhar no seu Excel (super te entendo). O Pandas também ajuda nisso! Assim:
Can you write Python code for it?