I am working on the file "02.1 Scaling and Normalization.ipynb".
The first task is:
The effect of preprocessing on supervised learning
Can you write Python code for it?

from sklearn.svm import SVC

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target,
                                                    random_state=0)

svm = SVC(C=100)
svm.fit(X_train, y_train)
print(svm.score(X_test, y_test))

# preprocessing using 0-1 scaling
scaler = MinMaxScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# learning an SVM on the scaled training data
svm.fit(X_train_scaled, y_train)
# scoring on the scaled test set

svm.score(X_test_scaled, y_test)

# preprocessing using zero mean and unit variance scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# learning an SVM on the scaled training data
svm.fit(X_train_scaled, y_train)
# scoring on the scaled test set
svm.score(X_test_scaled, y_test)


I am working on the file "02.2 Feature preprocessing, feature selection, interactions.ipynb".
The first task is:
Binning, Discretization, Linear Models and Trees
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor

X, y = mglearn.datasets.make_wave(n_samples=100)
plt.plot(X[:, 0], y, 'o')
line = np.linspace(-3, 3, 1000)[:-1].reshape(-1, 1)

reg = LinearRegression().fit(X, y)
plt.plot(line, reg.predict(line), label="linear regression")

reg = DecisionTreeRegressor(min_samples_split=3).fit(X, y)
plt.plot(line, reg.predict(line), label="decision tree")
plt.ylabel("regression output")
plt.xlabel("input feature")
plt.legend(loc="best")

np.set_printoptions(precision=2)
bins = np.linspace(-3, 3, 11)
bins

which_bin = np.digitize(X, bins=bins)
print("\nData points:\n", X[:5])
print("\nBin membership for data points:\n", which_bin[:5])

from sklearn.preprocessing import OneHotEncoder
# transform using the OneHotEncoder.
encoder = OneHotEncoder(sparse=False)
# encoder.fit finds the unique values that appear in which_bin
encoder.fit(which_bin)
# transform creates the one-hot encoding
X_binned = encoder.transform(which_bin)
print(X_binned[:5])

X_binned.shape

line_binned = encoder.transform(np.digitize(line, bins=bins))

plt.plot(X[:, 0], y, 'o')
reg = LinearRegression().fit(X_binned, y)
plt.plot(line, reg.predict(line_binned), label='linear regression binned')

reg = DecisionTreeRegressor(min_samples_split=3).fit(X_binned, y)
plt.plot(line, reg.predict(line_binned), label='decision tree binned')
for bin in bins:
    plt.plot([bin, bin], [-3, 3], ':', c='k')
plt.legend(loc="best")
plt.suptitle("linear_binning")


I am working on the file "02.2 Feature preprocessing, feature selection, interactions.ipynb".
The first task is:
Checking string-encoded categorical data
Can you write Python code for it?

data.gender.value_counts()

print("Original features:\n", list(data.columns), "\n")
data_dummies = pd.get_dummies(data)
print("Features after get_dummies:\n", list(data_dummies.columns))

data_dummies.head()

# Get only the columns containing features, that is all columns from 'age' to 'occupation_ Transport-moving'
# This range contains all the features but not the target

features = data_dummies.ix[:, 'age':'occupation_ Transport-moving']
# extract numpy arrays
X = features.values
y = data_dummies['income_ >50K'].values
print(X.shape, y.shape)

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
print(logreg.score(X_test, y_test))


I am working on the file "02.2 Feature preprocessing, feature selection, interactions.ipynb".
The first task is:
Exercises
Choose either the Boston housing dataset or the adult dataset from above. Compare a linear model with interaction features against one without interaction features.
Use feature selection to determine which interaction features were most important.
Can you write Python code for it?

data = pd.read_csv("data/adult.data", header=None, index_col=False,
                   names=['age', 'workclass', 'fnlwgt', 'education',  'education-num',
                          'marital-status', 'occupation', 'relationship', 'race', 'gender',
                          'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'income'])
y = data.income.values
X = pd.get_dummies(data.drop("income", axis=1))

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

scaler = MinMaxScaler().fit(X_train)
X_train_ = scaler.transform(X_train)
X_test_ = scaler.transform(X_test)

LogisticRegression().fit(X_train_, y_train).score(X_test_, y_test)

X_train.shape

select = SelectFromModel(RandomForestClassifier(n_estimators=100), threshold="5 * median")
X_train_selected = select.fit_transform(X_train_, y_train)
X_test_selected = select.transform(X_test_)

LogisticRegression().fit(X_train_selected, y_train).score(X_test_selected, y_test)

X_train_selected.shape

poly = PolynomialFeatures(degree=2).fit(X_train_selected)
X_train_selected_poly = poly.transform(X_train_selected)
X_test_selected_poly = poly.transform(X_test_selected)

lr = LogisticRegression(C=0.01, penalty="l1").fit(X_train_selected_poly, y_train)
lr.score(X_test_selected_poly, y_test)

np.array(poly.get_feature_names(X.columns[select.get_support()]))[lr.coef_.ravel() != 0]


I am working on the file "02.2 Feature preprocessing, feature selection, interactions.ipynb".
The first task is:
Interactions and Polynomials
Can you write Python code for it?

X_combined = np.hstack([X, X_binned])
print(X_combined.shape)

plt.plot(X[:, 0], y, 'o')

reg = LinearRegression().fit(X_combined, y)

line_combined = np.hstack([line, line_binned])
plt.plot(line, reg.predict(line_combined), label='linear regression combined')

for bin in bins:
    plt.plot([bin, bin], [-3, 3], ':', c='k')
plt.legend(loc="best")

X_product = np.hstack([X_binned, X * X_binned])
print(X_product.shape)

plt.plot(X[:, 0], y, 'o')

reg = LinearRegression().fit(X_product, y)

line_product = np.hstack([line_binned, line * line_binned])
plt.plot(line, reg.predict(line_product), label='linear regression combined')

for bin in bins:
    plt.plot([bin, bin], [-3, 3], ':', c='k')
plt.legend(loc="best")

from sklearn.preprocessing import PolynomialFeatures

# include polynomials up to x ** 10:
poly = PolynomialFeatures(degree=10)
poly.fit(X)
X_poly = poly.transform(X)

X_poly.shape

poly.get_feature_names()

plt.plot(X[:, 0], y, 'o')

reg = LinearRegression().fit(X_poly, y)

line_poly = poly.transform(line)
plt.plot(line, reg.predict(line_poly), label='polynomial linear regression')
plt.legend(loc="best")

from sklearn.svm import SVR
plt.plot(X[:, 0], y, 'o')

for gamma in [1, 10]:
    svr = SVR(gamma=gamma).fit(X, y)
    plt.plot(line, svr.predict(line), label='SVR gamma=%d' % gamma)

plt.legend(loc="best")

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

boston = load_boston()
X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target, random_state=0)

# rescale data:
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

poly = PolynomialFeatures(degree=2).fit(X_train_scaled)
X_train_poly = poly.transform(X_train_scaled)
X_test_poly = poly.transform(X_test_scaled)
print(X_train.shape)
print(X_train_poly.shape)

print(poly.get_feature_names())

from sklearn.linear_model import Ridge
ridge = Ridge().fit(X_train_scaled,...


I am working on the file "02.2 Feature preprocessing, feature selection, interactions.ipynb".
The first task is:
Numbers can encode categoricals
Can you write Python code for it?

# create a dataframe with an integer feature and a categorical string feature
demo_df = pd.DataFrame({'Integer Feature': [0, 1, 2, 1], 'Categorical Feature': ['socks', 'fox', 'socks', 'box']})
demo_df

pd.get_dummies(demo_df)

demo_df['Integer Feature'] = demo_df['Integer Feature'].astype(str)
pd.get_dummies(demo_df)


I am working on the file "02.2 Feature preprocessing, feature selection, interactions.ipynb".
The first task is:
Sequential Feature Selection
Can you write Python code for it?

from mlxtend.feature_selection import SequentialFeatureSelector
sfs = SequentialFeatureSelector(LogisticRegression(), k_features=40,
                                 forward=True, scoring='accuracy',cv=5)
sfs = sfs.fit(X_train, y_train)

mask = np.zeros(80, dtype='bool')
mask[np.array(sfs.k_feature_idx_)] = True

plt.matshow(mask.reshape(1, -1), cmap='gray_r')

LogisticRegression().fit(sfs1.transform(X_train), y_train).score(sfs.transform(X_test), y_test)


I am working on the file "02.Our Dataset.ipynb".
The first task is:
<b>Describe the dataset with some summary statitsics<b>
Can you write Python code for it?


I am working on the file "02.Our Dataset.ipynb".
The first task is:
<b>Sneak a peek at data (a reminder of your pandas dataframe methods)<b>
Can you write Python code for it?

# convert to pandas df (adding real column names)
iris.df = pd.DataFrame(iris.data,
                       columns = iris.feature_names)

# first few rows
iris.df.head()


I am working on the file "02.Our Dataset.ipynb".
The first task is:
Let's Dive In!
Can you write Python code for it?

import pandas as pd
import numpy as np

%matplotlib inline


I am working on the file "02.Our Dataset.ipynb".
The first task is:
PRO TIP: To save our standardization and reapply later (say to the test set or some new data), create a transformer object like so:
```python
scaler = preprocessing.StandardScaler().fit(X_train)

apply to a new dataset (e.g. test set):
scaler.transform(X_test)
```
Normalization - scaling samples <i>individually</i> to have unit norm

This type of scaling is really important if doing some downstream transformations and learning (see sklearn docs here for more) where similarity of pairs of samples is examined
A basic intro to normalization and the unit vector can be found here
Can you write Python code for it?

# Standardization aka scaling
from sklearn import preprocessing, datasets

# make sure we have iris loaded
iris = datasets.load_iris()

X, y = iris.data, iris.target

# scale it to a gaussian distribution
X_norm = preprocessing.normalize(X, norm='l1')

# how does it look now
pd.DataFrame(X_norm).tail()

# let's just confirm our standardization worked (mean is 0 w/ unit variance)
pd.DataFrame(X_norm).describe()

# cumulative sum of normalized and original data:
#print(pd.DataFrame(X_norm.cumsum().reshape(X.shape)).tail())
#print(pd.DataFrame(X).cumsum().tail())

# unit norm (convert to unit vectors) - all row sums should be 1 now
X_norm.sum(axis = 1)


I am working on the file "02.Our Dataset.ipynb".
The first task is:
Targets (aka labels)
Can you write Python code for it?

import pandas as pd
from sklearn import datasets
iris = datasets.load_iris()

pd.DataFrame({'target name': iris.target_names})


I am working on the file "02.Our Dataset.ipynb".
The first task is:
We don't have to do much with the iris dataset.  It has no missing values.  It's already in numpy arrays and has the correct shape for sklearn.  However we could try standardization and/or normalization. (later, in the transforms section, we will show one hot encoding, a preprocessing step)

Preprocessing (Bonus Material)

What you might have to do before using a learner in `sklearn`:

Non-numerics transformed to numeric (tip: use applymap() method from pandas)
Fill in missing values
Standardization
Normalization
Encoding categorical features (e.g. one-hot encoding or dummy variables)

Features should end up in a numpy.ndarray (hence numeric) and labels in a list.
Data options:
* Use pre-processed datasets from scikit-learn
* Create your own
* Read from a file
If you use your own data or "real-world" data you will likely have to do some data wrangling and need to leverage pandas for some data manipulation.
Standardization - make our data look like a standard Gaussian distribution (commonly needed for sklearn learners)

FYI: you'll commonly see the data or feature set (ML word for data without it's labels) represented as a capital X and the targets or labels (if we have them) represented as a lowercase y.  This is because the data is a 2D array or list of lists and the targets are a 1D array or simple list.
Can you write Python code for it?

# Standardization aka scaling
from sklearn import preprocessing, datasets

# make sure we have iris loaded
iris = datasets.load_iris()

X, y = iris.data, iris.target

# scale it to a gaussian distribution
X_scaled = preprocessing.scale(X)

# how does it look now
pd.DataFrame(X_scaled).head()

# let's just confirm our standardization worked (mean is 0 w/ unit variance)
pd.DataFrame(X_scaled).describe()

# also could:
#print(X_scaled.mean(axis = 0))
#print(X_scaled.std(axis = 0))


I am working on the file "02.Our Dataset.ipynb".
The first task is:
sklearn TIP: all included datasets for have at least feature_names and sometimes target_names

Get to know the data - visualize and explore

Features (columns/measurements) come from this diagram (links out to source on kaggle):

Shape
Peek at data
Summaries

Shape and representation
Can you write Python code for it?

import pandas as pd
from sklearn import datasets

iris = datasets.load_iris()

# How many data points (rows) x how many features (columns)
print(iris.data.shape)
print(iris.target.shape)


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Assignment
Assigment is one of the most important steps. And the simplest!
Think back to algebra in school:
x is equal to 10. So what is x?
x + y is equal to 15. x is 10. What is y?
x and y have been assigned a value. Which makes x and y a variable.
Can you write Python code for it?

x = 5

y = 10

z = x + y

z

x = "Hello!"

y = "World!"

z = x + y

z


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Combining strings (letters) and numbers
Can you write Python code for it?

print('Guess how many cupcakes I ate:', 5)

print("Tomorrow I will be", 5, "years old.")


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Comments
Triple quotation marks can also be used to leave yourself a comment. It is more common to use the # symbol though.
Can you write Python code for it?

# For example, this is a comment.
# This is a comment, too.
"""But
this
is
a
comment
too.
A
multi-line
comment."""

"""
This is another comment.
"""

print ("The meaning of life - ")
"""may
or
may
not
be"""
print ("42")

"""This is another comment."""


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Exercise
Type your own Hello World program! If you haven't written code in Python before, this could be a momentous occassion for you, so pat yourself on the back while you're at it.
Can you write Python code for it?


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Help
Can you write Python code for it?


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Other Conventions in Naming
Can you write Python code for it?

# Cannot begin with a number
01alpha = 1
print(01alpha)

# Only a few non-alphanumeric characters allowed
alpha_01 = 1
print(alpha_01)

alpha-02 = 10
print(alpha-02) #Will give an error

# Use descriptive name so it's easy to know what the value refers to
a1 = 100
boiling_point = 100
print(a1)
print(boiling_point)


I am working on the file "02.Python_Basics.ipynb".
The first task is:
Return
We will cover this in detail later, but here's a quick introduction. In Python, as in other programming languages, you can define functions that take a specified output, and return an output. The return function in Python does exactly what it says on the box - it causes the function to exit and hand back a value to its caller.
Again, don't worry about what that means, we will see this in action.
Reserved names
False      class      finally    is         return
None       continue   for        lambda     try
True       def        from       nonlocal   while
and        del        global     not        with
as         elif       if         or         yield
assert     else       import     pass
break      except     in         raise
Can you write Python code for it?

# example
False = 20
print(False)

False01 = 3.142
print(False01)


I am working on the file "02.Python_Basics.ipynb".
The first task is:
The format to print in Python 3.x is:
print ("message in single, double or triple quotation marks")
Can you write Python code for it?

print ('Hello World!')

print ("""Hello World!""")


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Build the network
For the neural network, we'll build each layer into a function.  Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function.  This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.
Note: None for shapes in TensorFlow allow for a dynamic size.
Can you write Python code for it?

import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    #Implement Function
    print(image_shape)
    return tf.placeholder(tf.float32, [None, image_shape[0], image_shape[1], image_shape[2]], name="x")

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    #Implement Function
    #print(n_classes)
    return tf.placeholder(tf.float32, [None, n_classes], name="y")

def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # Implement Function
    return tf.placeholder(tf.float32, name="keep_prob")

tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Check Point
This is our first checkpoint.  If we ever decide to come back to this notebook or have to restart the notebook, we can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Checkpoint
The model has been saved to disk.
Test Model
Test the model against the test dataset.  This will be our final accuracy.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')

        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0

        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
       ...


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Convolution and Max Pooling Layer
Convolution layers have a lot of success with images. For this code cell, we should implement the function conv2d_maxpool to apply convolution then max pooling.
Can you write Python code for it?

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    #Implement Function
    print(conv_ksize)
    print(x_tensor.shape)
    print(conv_num_outputs)
    weights = tf.Variable(tf.truncated_normal((conv_ksize[0], conv_ksize[1], int(x_tensor.shape[3]), conv_num_outputs ),\
                                              mean=0, stddev=0.1))
    bias = tf.Variable(tf.zeros(conv_num_outputs))

    print(conv_strides)
    conv_layer = tf.nn.conv2d(x_tensor, weights, strides=[1, conv_strides[0], conv_strides[1], 1], padding = 'SAME')
    conv_layer = tf.nn.bias_add(conv_layer, bias)

    conv_layer = tf.nn.relu(conv_layer)

    print(pool_ksize)
    print(pool_strides)
    conv_layer = tf.nn.max_pool(conv_layer, ksize=[1, pool_ksize[0], pool_ksize[1], 1], \
                                strides=[1, pool_strides[0], pool_strides[1], 1], padding='SAME' )

    return conv_layer

tests.test_con_pool(conv2d_maxpool)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Explore the Data
The dataset is broken into batches to prevent your machine from running out of memory.  The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:
* airplane
* automobile
* bird
* cat
* deer
* dog
* frog
* horse
* ship
* truck
Understanding a dataset is part of making predictions on the data.  Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 3
sample_id = 2
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Fully Train the Model
Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.
Can you write Python code for it?

save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)

    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs).
Can you write Python code for it?

def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # Implement Function
    print(x_tensor.shape)
    print(num_outputs)
    weights = tf.Variable(tf.truncated_normal( (int(x_tensor.shape[1]), num_outputs), mean=0, stddev=0.1 ) )
    bias = tf.Variable(tf.zeros(num_outputs))

    layer = tf.add(tf.matmul(x_tensor, weights), bias)
    print(layer.shape)
    return layer

tests.test_fully_conn(fully_conn)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Hyperparameters
Tune the following parameters:
* Set epochs to the number of iterations until the network stops learning or start overfitting
* Set batch_size to the highest number that your machine has memory for.
* Set keep_probability to the probability of keeping a node using dropout
Can you write Python code for it?

# Tune Parameters
epochs = 40
batch_size = 128
keep_probability = 0.5


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, we'll be implementing a function for preprocessing.  This time, we'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Can you write Python code for it?

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    #Implement Function
    #print(x)
    result_one_hot_encode = np.eye(10)[x]
    #print(result_one_hot_encode)
    return result_one_hot_encode

tests.test_one_hot_encode(one_hot_encode)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but we don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Show Stats
Implement the function print_stats to print loss and validation accuracy.  Use the global variables valid_features and valid_labels to calculate validation accuracy.  Use a keep probability of 1.0 to calculate the loss and validation accuracy.
Can you write Python code for it?

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # Implement Function
    loss = session.run(cost, feed_dict={x:feature_batch, y:label_batch, keep_prob:1.})
    valid_acc = session.run(accuracy, feed_dict={x:valid_features, y:valid_labels, keep_prob:1.})
    print ('Loss: {:>10.4f} Validation Accuracy: {:.6f} '.format(loss,valid_acc))


I am working on the file "02.convolutinal_network/image_classification.ipynb".
The first task is:
Train the Neural Network
Single Optimization
Implement the function train_neural_network to do a single optimization.  The optimization should use optimizer to optimize in session with a feed_dict of the following:
* x for image input
* y for labels
* keep_prob for keep probability for dropout
This function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
Can you write Python code for it?

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # Implement Function
    #print(optimizer)
    #print(feature_batch.size)
    #cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=feature_batch, labels=label_batch))
    #optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)
    session.run(optimizer, feed_dict={x: feature_batch, y:label_batch, keep_prob:keep_probability})

tests.test_train_nn(train_neural_network)


I am working on the file "02/chapter2.ipynb".
The first task is:
11. タブをスペースに置換
タブ1文字につきスペース1文字に置換せよ．確認にはsedコマンド，trコマンド，もしくはexpandコマンドを用いよ．
Can you write Python code for it?

hightemp = "".join(map(str, [i.replace('\t', ' ') for i in open('hightemp.txt', 'r')]))
print(hightemp)


I am working on the file "02/chapter2.ipynb".
The first task is:
12. 1列目をcol1.txtに，2列目をcol2.txtに保存
各行の1列目だけを抜き出したものをcol1.txtに,
2列目だけを抜き出したものをcol2.txtとしてファイルに保存せよ.
確認にはcutコマンドを用いよ.
Can you write Python code for it?

col1 = open('col1.txt', 'w')
col2 = open('col2.txt', 'w')
hightemp = [i.replace('\t', ' ').split() for i in open('hightemp.txt', 'r')]

col1.write("\n".join(map(str, [i[0] for i in hightemp])))
col1.close()
col2.write("\n".join(map(str, [i[1] for i in hightemp])))
col2.close()


I am working on the file "02/chapter2.ipynb".
The first task is:
15. 末尾のN行を出力
自然数Nをコマンドライン引数などの手段で受け取り, 入力のうちの末尾のN行だけを表示せよ.
確認にはtailコマンドを用いよ.
Can you write Python code for it?

def display_back_nline(n, filename):
    return "".join(map(str, [i for i in open(filename, 'r')][-n-1:-1]))

display = display_back_nline(5, "col3.txt")
print(display)


I am working on the file "02/chapter2.ipynb".
The first task is:
16. ファイルをN分割する
自然数Nをコマンドライン引数などの手段で受け取り, 入力のファイルを行単位でN分割せよ.
同様の処理をsplitコマンドで実現せよ.
Can you write Python code for it?

%%timeit
def split_file(n, filename):
    line = [i.strip('\n') for i in open(filename, 'r')]
    length = len(line)
    n = length//n
    return [line[i:i+n] for i in range(0, length, n)]

split_file(2, "col1.txt")


I am working on the file "02/chapter2.ipynb".
The first task is:
17. 1列目の文字列の異なり
1列目の文字列の種類(異なる文字列の集合)を求めよ.
確認にはsort, uniqコマンドを用いよ.
Can you write Python code for it?

def first_char(filename):
    return set(i[0] for i in open(filename, 'r'))

print(first_char('hightemp.txt'))


I am working on the file "02/chapter2.ipynb".
The first task is:
18. 各行を3コラム目の数値の降順にソート
各行を3コラム目の数値の逆順で整列せよ (注意:各行の内容は変更せずに並び替えよ).
確認にはsortコマンドを用いよ(この問題はコマンドで実行した結果とあわなくても良い).
Can you write Python code for it?

%%timeit
from operator import itemgetter

def column_sort(sort_key, filename):
    return sorted([i.split() for i in open(filename, 'r')], key=itemgetter(sort_key-1))
column_sort(3, 'hightemp.txt')


I am working on the file "02/homework_day2.ipynb".
The first task is:
2. Count capital letters
In this exercises you're going to deal with YOUR DATA. Indeed, in the list below there are stored your Favorite Tv Series. But, as you can see, there is something weird. There are too much CaPITal LeTTErs. Your task is to count the capital letters in all the strings and then print the total number of capital letters in all the list.
Can you write Python code for it?

tv_series = ['Game of THRroneS',
             'big bang tHeOrY',
             'MR robot',
             'WesTWorlD',
             'fIRefLy',
             "i haven't",
             'HOW I MET your mothER',
             'friENds',
             'bRon broen',
             'gossip girl',
             'prISon break',
             'breaking BAD']

# write here your code
import re
pattern = re.compile("[A-Z]")
capitals = 0
for c in ''.join(tv_series):
    if pattern.match(c):
        capitals+=1
capitals


I am working on the file "02/homework_day2.ipynb".
The first task is:
2. More math...
Write a program which will find all such numbers which are divisible by 7 but are not a multiple of 5, between 0 and 1000 (both included). The numbers obtained should be printed in a comma-separated sequence on a single line. (range and CFS)
Can you write Python code for it?


I am working on the file "02/homework_day2.ipynb".
The first task is:
3. A remark
Using the list above, create a dictionary where the keys are Unique IDs and values the TV Series.
You have to do the exercise keeping in mind these 2 constraints:

The order of the IDs has to be dependent on the alphabetical order of the titles, i.e. 0: first_title_in_alphabetical_order and so on...

Solve the mess of the capital letter: we want them only at the start of the words ("prISon break" should be "Prison Break")
Can you write Python code for it?

# write here your code
series_dict = {}
titled = [t.title() for t in tv_series]
for idx,movie_title in enumerate(sorted(titled)):
    series_dict[idx]=movie_title.title()
series_dict


I am working on the file "02/homework_day2.ipynb".
The first task is:
4. Dictionary to its maximum
Invert the keys with the values in the dictionary built before.
Can you write Python code for it?

# write here your code
series_dict = dict(zip(series_dict.values(),series_dict.keys()))
series_dict


I am working on the file "02/homework_day2.ipynb".
The first task is:
4. Other boring math
Let's talk about our beloved exams. Starting from the exams and CFU below, are you able to compute the weighted mean of them?
Let's do it and print the result.
Description of the data:
exams[1] = $(title_1, grade_1)$
cfu[1] = $CFU_1$
Can you write Python code for it?

exams = [('BIOINFORMATICS', 29),
         ('DATA MANAGEMENT FOR DATA SCIENCE', 30),
         ('DIGITAL EPIDEMIOLOGY', 26),
         ('NETWORKING FOR BIG DATA AND LABORATORY',28),
         ('QUANTITATIVE MODELS FOR ECONOMIC ANALYSIS AND MANAGEMENT','30 e lode'),
         ('DATA MINING TECHNOLOGY FOR BUSINESS AND SOCIETY', 30),
         ('STATISTICAL LEARNING',30),
         ('ALGORITHMIC METHODS OF DATA MINING AND LABORATORY',30),
         ('FUNDAMENTALS OF DATA SCIENCE AND LABORATORY', 29)]

cfu = sum([6,6,6,9,6,6,6,9,9])

cfu_ex = [6,6,6,9,6,6,6,9,9]

# write here your code
grades_sum = 0
mean = 0
for idx,w in enumerate(cfu_ex):
    try:
        mean += int(exams[idx][1])/cfu*w
        grades_sum += int(exams[idx][1])
    except ValueError:
        if exams[idx][1] == "30 e lode":
            mean += 30/cfu*w
            grades_sum += 30
        else:
            pass

print(mean)


I am working on the file "02/homework_day2.ipynb".
The first task is:
6. C
And now, write down a code which computes the first 10 Fibonacci numbers
(details: https://en.wikipedia.org/wiki/Fibonacci_number)
Can you write Python code for it?

# write here your code
fibonacci = []
for x in range(10):
    if x<2:
        fibonacci.append(1)
    else:
        fibonacci.append(fibonacci[x-1] + fibonacci[x-2])

fibonacci


I am working on the file "02/homework_day2.ipynb".
The first task is:
6. StackOverflow
Let's start using your new best friend. Now I'm going to give other task, slightly more difficult BUT this time, just googling, you will find easily the answer on the www.stackoverflow.com. You can use the code there for solving the exercise BUT you have to understand the solution there COMMENTING the code, showing me you understood the thinking process behind the code.
6. A
Show me an example of how to use PROPERLY the Try - Except statements
Can you write Python code for it?

# write here your code
from random import randint
while True:
    try:
        randint(0, 9) / randint(0, 9) # execute some random division
    except ZeroDivisionError:         # until a division by zero occurs
        break


I am working on the file "02/homework_day2.ipynb".
The first task is:
Have you done in one line of code? If not, try now!
Can you write Python code for it?

# write here your code
print(dict(zip(series_dict.values(),series_dict.keys())))


I am working on the file "02/keras_CNN_classification.ipynb".
The first task is:
Compute test accuracy by hand
Can you write Python code for it?

classProb = model.predict(x=tstData)
print(classProb.shape)

correctProb = (classProb * tstLabels).sum(axis=1)
wrongProb = (classProb * (1-tstLabels)).max(axis=1)
print(correctProb.shape, wrongProb.shape)

accuracy = (correctProb > wrongProb).mean()
print('Accuracy: ',  accuracy)


I am working on the file "02/keras_CNN_classification.ipynb".
The first task is:
Define TensorBoard callback
TensorBoard is able to store network statistics (loss, accuracy, weight histograms, activation histograms, ...) and view them through web interface. To view the statistics, run 'tensorboard --logdir=path/to/log-directory' and go to localhost:6006.
Can you write Python code for it?

import keras
tbCallBack = keras.callbacks.TensorBoard(
    log_dir='./Graph',
    histogram_freq=1,
    write_graph=True, write_images=True)

model.fit(
    x=trnData, y=trnLabels,
    batch_size=64, epochs=100, verbose=1,
    validation_data=[tstData, tstLabels], shuffle=True)#, callbacks=[tbCallBack])


I am working on the file "02/keras_CNN_classification.ipynb".
The first task is:
Define net
Can you write Python code for it?

from keras.layers import Input, Reshape, Dense, Dropout, Flatten
from keras.layers import Activation, Conv2D, MaxPooling2D
from keras.models import Model
from keras import regularizers

w_decay = 0.0001
w_reg = regularizers.l2(w_decay)

def get_simple_FC_network(input_data, layer_cout, layer_dim):

    net = Conv2D(32,3, activation='relu')(input_data)
    net = MaxPooling2D(2, 2)(net)
    net = Conv2D(32,3, activation='relu')(net)
    net = Flatten()(net)
    for i in range(layer_cout):
        FC = Dense(layer_dim, activation='relu')
        net = Dropout(rate=0.33)(FC(net))

    net = Dense(10, name='out', activation='softmax')(net)

    return net


I am working on the file "02/keras_CNN_classification.ipynb".
The first task is:
Normalize data
This maps all values in trn. and tst. data to range <-0.5,0.5>.
Some kind of value normalization is preferable to
provide consistent behavior accross different problems and datasets.
Can you write Python code for it?

trnData = trnData.astype(np.float32) / 255.0 - 0.5
tstData = tstData.astype(np.float32) / 255.0 - 0.5


I am working on the file "02/keras_CNN_classification.ipynb".
The first task is:
Predict and evaluate
Can you write Python code for it?

classProb = model.predict(x=tstData[0:2])
print('Class probabilities:', classProb, '\n')
loss, acc = model.evaluate(x=tstData, y=tstLabels, batch_size=1024)
print()
print('loss', loss)
print('acc', acc)


I am working on the file "02/keras_CNN_classification.ipynb".
The first task is:
Read CIFAR10 dataset
Can you write Python code for it?

from tools import readCIFAR, mapLabelsOneHot

# First run ../data/downloadCIFAR.sh
# This reads the dataset
trnData, tstData, trnLabels, tstLabels = readCIFAR('../data/cifar-10-batches-py')

plt.subplot(1, 2, 1)
img = collage(trnData[:16])
print(img.shape)
plt.imshow(img)
plt.subplot(1, 2, 2)
img = collage(tstData[:16])
plt.imshow(img)
plt.show()

# Convert categorical labels to one-hot encoding which
# is needed by categorical_crossentropy in Keras.
# This is not universal. The loss can be easily implemented
# with category IDs as labels.
trnLabels = mapLabelsOneHot(trnLabels)
tstLabels = mapLabelsOneHot(tstLabels)
print('One-hot trn. labels shape:', trnLabels.shape)


I am working on the file "020 Poisson.ipynb".
The first task is:
Definice problému: Konstantní nábojová hustota s Dirichletovou OP. Síť se 100 body, krok $h=1$, $\epsilon_0=1$. Maximální chyba iteračních metod $10^{-3}$
Can you write Python code for it?

eps0 = 1
dx = 1
imax = 100
M = imax
rho = ones(imax)
b = -rho/eps0
epsilon = 1e-3


I am working on the file "020 Poisson.ipynb".
The first task is:
Fourier method
Can you write Python code for it?

from scipy.fftpack import dst, idst

%%timeit
rho_k = dst(b, type=1) # type 1 assumes odd func around n=-1 and n=imax
k = arange(1, M+1)
lambda_k = 2*cos(k*pi/(M+1))-2
U_k = rho_k/lambda_k
U = idst(U_k, type=1)/(2*(M+1))

k = arange(1, M+1)
lambda_k = 2*cos(k*pi/(M+1))-2

%%timeit
rho_k = dst(b, type=1) # type 1 assumes odd func around n=-1 and n=imax
U_k = rho_k/lambda_k
U = idst(U_k, type=1)/(2*(M+1))


I am working on the file "020 Poisson.ipynb".
The first task is:
Iterativní řešení Gauss Seidelovou methodou. Nelze v pythonu jednoduše vektorizovat - používáme cyklus, což ji zpomaluje.
Can you write Python code for it?

%%timeit -n 2 -r 3
U = zeros(imax+2)
iter = 0
while True:
    err=0
    for i in range(1,imax+1):
        res = 0.5*(U[i-1] + U[i+1] - dx**2*b[i-1]) - U[i]
        U[i] += res
        err += res**2
    iter += 1
    if err < epsilon**2: break
print(iter)


I am working on the file "020 Poisson.ipynb".
The first task is:
Iterativní řešení Jacobiho methodou
Can you write Python code for it?

%%timeit -n 2 -r 3
U = zeros(imax+2)
iter = 0
while True:
    res = 0.5*(U[:-2] + U[2:] - dx**2*b) - U[1:-1]
    U[1:-1] += res
    iter += 1
    if sum(res**2) < epsilon**2: break
print(iter)


I am working on the file "020 Poisson.ipynb".
The first task is:
Superrelaxace (Successive overrelaxation (SOR)) s faktorem $\omega$, bez vektorizace
Can you write Python code for it?

%%timeit -n 4 -r 3
U = zeros(imax+2)
omega = 2/(1+sqrt(1-cos(pi/imax)**2))

iter = 0
while True:
    err=0
    for i in range(1,imax+1):
        res = 0.5*(U[i-1] + U[i+1] - dx**2*b[i-1]) - U[i]
        U[i] += omega*res
        err += res**2
    iter += 1
    if err < epsilon**2: break
print(iter)


I am working on the file "020 Poisson.ipynb".
The first task is:
The d2matrix is a matrix of second differences in sparse format:
Can you write Python code for it?

print(d2matrix(10).todense())

#%%timeit
%prun U = spsolve(d2x, b)


I am working on the file "020 Poisson.ipynb".
The first task is:
Try direct solver with separate factorization
Can you write Python code for it?

from scipy.sparse.linalg import factorized
LU = factorized(d2x)

# %%timeit
%prun U = LU(b)


I am working on the file "020 Poisson.ipynb".
The first task is:
Výše uvedené metody lze vektorizovat iterace zvlášť přes liché a sudé elementy v šachovnicovitém vzoru (red-black ordering)
Can you write Python code for it?


I am working on the file "020 Poisson.ipynb".
The first task is:
We have obtained a solver, which is more than 1000 faster than our initial attempts
Can you write Python code for it?


I am working on the file "0204_BiasVarianceTradeOff.ipynb".
The first task is:
And here is how our random sample looks like. Without knowing the true relation between feature and response one could easily argue the dependent variable follow some linear function (or maybe a polynomail function with low order).
Can you write Python code for it?

plt.figure(figsize=(12, 8))
plt.scatter(x, y_noise);


I am working on the file "0204_BiasVarianceTradeOff.ipynb".
The first task is:
Bias
We defined 'bias' as follows:
* The inability for a machine learning method (like linear regression) to capture the true relationship is called bias.
* Refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model.
* The lower the bias, the closer the model approaches the underlying true pattern. The higher the bias, the further the model is from the true complexity of the underlying true pattern.
What if we approximate our sample with a linear regression? Below code does exactly this.
Can you write Python code for it?

# Create OLS object
reg = sm.OLS(y_noise, exog=sm.add_constant(x)).fit()

print('R2 = ', reg.rsquared)

reg_ln = sm.OLS(y_noise, exog=sm.add_constant(np.log(x))).fit()
print('R2 = ', reg_ln.rsquared)

# Plot data
plt.figure(figsize=(12, 8))
plt.scatter(x, y_noise, label="Sample")
plt.plot(x, y, color='g', label="True function")
plt.plot(x, reg.fittedvalues, c='k', label='Linear Fit')  # Linear fit
#plt.plot(x, reg_ln.fittedvalues, c='r', label='Ln Fit')
plt.legend();


I am working on the file "0204_BiasVarianceTradeOff.ipynb".
The first task is:
Here we see that a Kernel regression is much better at approximating the true function than a simple linear model. Therefore, the bias for the more complex function is lower and should be preferred.
Variance
This is how we defined variance:
* Variance refers to the amount by which $\hat{f}$ would change if we estimated it using a different (training) sample.
* Ideally, our estimate of $\hat{f}$ does not vary too much between training sets.
* Tells us how sensitive the learning algorithm is to small changes in the training set
Now let us see how the two approximation functions (linear and kernel regression) fare when we just use a subset of overall sample. This we do to see the variance in the function $\hat{f}$ between different training sets.
Can you write Python code for it?

plt.figure(figsize=(16, 8))
for i in range(1, 7):
    indices = np.sort(np.random.randint(0, x.shape[0], 20))
    reg = sm.OLS(y_noise[indices], exog=sm.add_constant(x[indices])).fit()
    kreg = sm.nonparametric.KernelReg(y_noise[indices], exog=x[indices], var_type='o', bw=[0.05]).fit()
    plt.subplot(2, 3, i)
    plt.plot(x[indices], reg.fittedvalues, c='k', label='Fit')
    plt.plot(x[indices], kreg[0], c='cyan', label='Kernel Fit')
    plt.plot(x, y, color='g', label="True")
    plt.ylim(-10, 4)
    plt.scatter(x[indices], y_noise[indices], label="Sample")
    plt.text(0.1, -8, 'Lin.fit: f(x) = {0: .2f} + {1: .2f} $x$'.format(reg.params[0], reg.params[1]))
    plt.legend();


I am working on the file "0204_BiasVarianceTradeOff.ipynb".
The first task is:
Knowing the true relationship between feature and response we can see that we have a certain bias present in that we approximate a logarithmic function with a much more static linear function. Especially towards the edges of the spectrum we tend to misclasify the sample.
Let us instead look at a kernel regression, which (given low bandwidth, follows the observation very closely.
Can you write Python code for it?

# Kreate kernel regression object
kreg = sm.nonparametric.KernelReg(y_noise, exog=x, var_type='o', bw=[0.20]).fit()

# Plot results
plt.figure(figsize=(12, 8))
plt.plot(x, y, color='g', label="True")
plt.scatter(x, y_noise, label="Sample")
plt.plot(x, reg.fittedvalues, c='k', label='Linear Fit') # Linear fit
plt.plot(x, kreg[0], c='cyan', label='Kernel Fit')  # Kernel fit
plt.legend();


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
As far as I know, Statsmodels does not provide a function to calculate 'greater than' or 'smaller than' alternative hypothesis. Reason being: because with symmetric distributions, the one-sided test can be derived from the two-sided test. A one-sided p-value is just half of the two-sided p-value. This means that given p and $t$ values from a two-tailed test, you would reject the null hypothesis of a greater-than test when p/2 < alpha and $t$ > 0, and of a less-than test when p/2 < alpha and $t$ < 0.
Coefficient of Determination
The $R^2$ measure, or "coefficient of determination", displays the proportion of the variability in $y$ that is well explained by the regression fit. It is defined as
$$\begin{equation}
R^2 = \frac{TSS - SSR}{TSS} = 1 - \frac{SSR}{TSS}
\end{equation}$$
where TSS is the total sum of squares, defined as $TSS = \sum (y_i - \bar{y})^2$, and SSR is the sum of squared residuals, given by $SSR = \sum (y_i - \hat{y}_i)^2$.
It is easy to call the $R^2$ value from the regression object reg as the following line shows.
Can you write Python code for it?


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
As toy data we will use the 'Advertising' data set introduced in the chapter 4 of the script. The data is taken from James et al. (2013). A copy is provided on the book's website where we will download it from.
Can you write Python code for it?

# From Advertising data set read cols 2:4
url = 'https://www.statlearning.com/s/Advertising.csv'
ad = pd.read_csv(url, sep=',', usecols=(np.arange(1, 5, 1)))
print(ad.head())


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Based on the regression output we have no reason to reject the null of Geberit's stock beta being equal to zero. The $R^2$ measure, though, shows that only a small portion of the variation in Geberits monthly returns is explained by SMI's monthly returns.
Can you write Python code for it?

# Get relevant information
beta = regBeta.params['SMIrets']
alpha = regBeta.params["const"]
rsqr = regBeta.rsquared

# Plot scatter & lm; add text with alpha, beta , R2
plt.figure(figsize=(12, 8))
plt.scatter(dfRets.iloc[:months, 1],
            dfRets.iloc[:months, 0],
            marker='.', label='Monthly Returns')
plt.plot(dfRets.iloc[:months, 1], regBeta.fittedvalues, c='k', label='Fit')
plt.gca().set_aspect('equal')
plt.gca().set_xlim(-0.1, 0.15)
plt.ylabel('Geberit Monthly Returns')
plt.xlabel('SMI Monthly Returns')
plt.legend(loc='lower right')
plt.text(-0.08, 0.13, 'Beta: {0: .2f}'.format(beta))
plt.text(-0.08, 0.12, 'Alpha: {0: .2f}'.format(alpha))
plt.text(-0.08, 0.11, 'R^2: {0: .2f}'.format(rsqr));


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Calculate Fama-French Coefficients
In order to run the regression we need to have matching indices. Fama-French's index is 'yyyy-mm' while our dataframe with returns has format 'yyyy-mm-dd'. Since we know that the length of both dataframes is equal, we can simply overwrite the index of one of the dataframes.
Can you write Python code for it?

# Create matching indices
ff.index = shsRets.index


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Having done that, we are already set to run the regression and print the results.
Can you write Python code for it?

# Set observation period (last 60 monthly returns)
months = 60

# Create OLS object, run regression and calculate fit
regBeta = sm.OLS(endog=dfRets.iloc[:months, 0],
                 exog=sm.add_constant(dfRets.iloc[:months, 1])).fit()

# Show table on coefficients
print(regBeta.summary())


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Hedge a Portolio
Now that we have all the factors, let's assume we want to build a portfolio with all 20 SMI stocks that maximizes the Sharpe ratio (SR). As a further condition we want to limit our exposure to the SMB factor to, let's say, $\beta_p^{SMB} = 0$. How would we allocate our investment under these conditions? In mathematical terms we have the following optimization problem:
$$\begin{equation}
\max_{w_i} SR = \frac{\mathbb{E}[r_p] - r_f}{\sigma_p} \qquad s.t. \qquad
\begin{cases}
\sum w_i &= 1 \
\beta_p^{SMB} &= 0
\end{cases}
\end{equation}$$
Usually, to calculate the expected return $\mathbb{E}[r_p]$, historical returns are taken. For our case here, we will take the expected returns given by our Fama-French 3 Factor model (denoted $\mathbf{R_{ff}}$). The portfolio variance $\sigma_p$ however, we estimate using historical data. Alternatively one could think of taking the SMI volatility index value as proxy. But this is only approximately true because we will not have the same weights per stock as the SMI and this thus might be questionable. With that we have
$$\begin{equation}
\max_{w_i} SR = \frac{\mathbf{w}^T \left(r_f + \mathbf{\alpha} + \mathbf{\beta}^{M} (r_{M} - r_{f}) + \mathbf{\beta}^{SMB} SMB + \mathbf{\beta}^{HML} HML \right) - r_f}{\mathbf{w}^T \mathbf{\Sigma}\mathbf{w}} \qquad s.t. \qquad
\begin{cases}
\sum w_i &= 1 \
\beta_p^{SMB} &= 0
\end{cases}
\end{equation}$$
Python can solve this problem numerically. We first set the stage by defining a auxiliary function pfStats that returns the expected portfolio return, volatility and Sharpe ratio given a vector of weights. Note that the function also makes use of other data like monthly returns and the riskfree rate as previously defined (which is set to 0) but only weights are a function input value. This is necessary for the optimization function.
Can you write Python code for it?

# Define rf and (equally spread) start weights
rf = 0
wghts = np.repeat(1. / len(tickers), len(tickers))

# Expected stock returns based on ff3f model
expShsRets = rf + ff3f.T.Alpha + \
    ff3f.T.BetaMkt * ff['Mkt-RF'].mean() + \
    ff3f.T.BetaSMB * ff.SMB.mean() + \
    ff3f.T.BetaHML * ff.HML.mean()

def pfStats(weights):
    '''Returns basic measures for a portfolio

    Parameters
    ==========
    weights : array-like
        weights for different securities in portfolio

    Returns
    =======
    expPfRet : float
        weighted, annualized expected portfolio return based on ff3f model
    pfVol : float
        historical annualized portfolio volatility
    SR : float
        portfolio Sharpe ratio for given riskfree rate
    '''

    expPfRet = np.sum(weights * expShsRets) * 12
    pfVol    = np.sqrt(np.dot(weights.T, np.dot(shsRets.cov() * 12, weights)))
    SR       = (expPfRet - rf) / pfVol

    return np.array([expPfRet, pfVol, SR])


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
If you wish to run tests with heteroskedastistic robust standard errors you can either access the reg object's robust standard errors (reg.HC0_se, reg.HC1_se, reg.HC2_se, reg.HC3_se) or, more conveniently, directly define the covariance estimator (e.g. 'HC3' as below) when you generate the object in the first place. Below example shows how you can do this. See also here or here for some further information.
Can you write Python code for it?

regRobust = sm.OLS(ad.sales, exog=sm.add_constant(ad.TV)).fit(cov_type='HC3')
print(regRobust.HC3_se, '\n')
print(reg.HC3_se)


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
If you wish to test a different null hypothesis, e.g. $H_0: \hat{\beta}{TV} = 0.054$ vs. $H_1: \hat{\beta}{TV} \neq 0.054$ use the following code:
Can you write Python code for it?


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Let us plot residuals versus fitted values to do some visual regression diagnostics:
Can you write Python code for it?

plt.figure(figsize=(12, 8))
plt.scatter(ad.TV, reg.resid)
plt.axhline(y=0, c='k')  #Black horizontal line at 0
plt.xlabel('TV (fitted)')
plt.ylabel('Residuals');


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Multiple Linear Regression
Estimating the Regression Coefficients
Simple linear regression serves well to introduce the concept and to build a good understanding. However, in reality we often have to work with more than one predictor. In the advertising data set for example we had not only data on TV advertising spendings but also on radio newspaper. It thus makes sense to extend the simple to a multiple linear regression model.
We again use the Advertising data set to see how this is done in Python. The same functions from the statsmodels package apply to multiple linear regression. We run the following regression
$$\text{sales} = \beta_0 + \beta_1 \text{TV} + \beta_2 \text{radio} + \beta_3 \text{newspaper} + \epsilon$$
Can you write Python code for it?

# Assign features and response to X and y
y = ad.sales
X = ad[['TV', 'radio', 'newspaper']]
X = sm.add_constant(X)

# Run regression and print summary
mlReg = sm.OLS(endog=y, exog=X).fit()
print(mlReg.summary())


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Next we run a linear regression of TV on sales to calculate the coefficients and print a summary output.
Can you write Python code for it?

# Run regression and calculate fit
reg = sm.OLS(ad.sales, exog=sm.add_constant(ad.TV)).fit()
# Alternatively: reg = sm.OLS(ad.sales, sm.add_constant(ad.TV)).fit()
print(reg.summary())


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Now we are ready to call the function. This time we use the last two years of monthly returns. Thus we set window=24 to overwrite the default value of 60.
Can you write Python code for it?

# Call function and save output to 'rollBeta'
rollBeta = rollingBeta(df=dfRets, window=24)


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Our constraints are as stated above: $\sum w_i = 1$, $\beta_P^{SMB} = 0$. Additionally we set bounds for the weights such that short/long position are allowed but only up to 100% per share ($w_i \in [-1, 1]\; \forall i \in [1, 2, \ldots, n]$).
Can you write Python code for it?

# Constraints and bounds
constr = [{'type': 'eq', 'fun': lambda x: np.sum(x) - 1},
          {'type': 'eq', 'fun': lambda x: np.sum(x * ff3f.T.BetaSMB) - 0}]
bnds   = tuple((-1,1) for x in range(len(tickers)))

# Minimization function
optPf = sco.minimize(minSR, x0=wghts, method='SLSQP', bounds=bnds, constraints=constr)

# Check if conditions are actually met
print('Sum of weights:  ', np.sum(optPf['x']))
print('Beta SMB factor: ', np.sum(optPf['x'] * ff3f.T.BetaSMB))

# Calculate portfolio stats given optimal weights
rsltsOptPf = pfStats(optPf['x'])

# Format weights into dataframe with Tickers as heading
optWghts = pd.DataFrame(data=optPf['x'], index=tickers)
optWghts.columns = ['optimalWghts']

# Print results
print('Portfolio return:     ', str(rsltsOptPf[0]))
print('Portfolio volatility: ', str(rsltsOptPf[1]))
print('Portfolio SR:         ', str(rsltsOptPf[2]), '\n')
print(str(optWghts))


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Regression Diagnostics
Test of Homoskedasticity
In general we assume a constant variance of the error term (homoskedasticity; $Var(\epsilon_i) = \sigma^2$ for $i = 1, \ldots, N$). From the residuals vs. fitted plot we have to question this assumption. To test it mathematically, you can run a heteroskedasticity test. The stats package offers several test options; the more common ones are White's or the one from Breusch-Pagan. See here for more details on tests on heteroskedasticity.
Below the White test is applied as an example. The parameter 'reg.model.exog' simply contains the matrix X (here it is a [200 x 2] matrix with constant 1 in first column and values for TV in second). The output should become more clear when you check the function's help page (use ?sm.stats.diagnostic.het_white).
The null hypothesis is that the error variance does not depend on X, thus is homoskedastic. Based on the large f-statistic value we can gently reject the null-hypothesis that the error variance is homoskedastic.
Can you write Python code for it?

# Test for heteroskedasticity with White test
wht = sm.stats.diagnostic.het_white(resid=reg.resid, exog=reg.model.exog)
print('f-statistic: {0:>19.4f} \n'
      'p-value for f-statistic: {1:>7.4f}'.format(wht[2], wht[3]))


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Side note: If you are an R guy and prefer their syntax, you could import the statsmodels.formula.api subpackage and run something like:

from statsmodels.formula.api import ols
reg = ols("Sales ~ TV", data=ad).fit()
reg.summary()

Intercept will automatically be calculated in the above setting.
Both p-values for intercept ($\hat{\beta}_0$) and slope ($\hat{\beta}_1$) are smaller than any reasonable significance level and thus we can reject the null hypothesis that either of the coefficients is zero (or irrelevant).
Instead of printing the whole summary, we could also access each of the three summary as follows:
Can you write Python code for it?


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
The above two plots are just standard matplotlib plots serving the purpose of visual diagnostics. Beyond that, the statsmodel package has separate built-in plotting functions suited for visual regression diagnostics. We will not discuss them here but if interested you'll find plenty of sources on the web (e.g. here, here or here)
Accessing the Output
The reg object contains a ton of information which you can all access. To see what there is, type reg. and press tab. Two examples are shown below. Notice that some are attributes (like x.shape, x.size) and do not need paranthesis to call them. Others are methods (similar to .sum(), .min()) and require parenthesis.
Can you write Python code for it?

# Regression coefficients
print(reg.params, '\n')
print(reg.resid.head())


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
The dataframe df contains monthly closing prices on all SMI-stocks (incl. SMI index) with a date index in descending order. Let us create a separate Pandas dataframe with the returns of the past 60 months (dfRets) for both GEBN as well as the index.
Can you write Python code for it?

# Calculate returns and assign to variable dfRets
dfRets = pd.DataFrame()
dfRets['GEBNrets'] = np.log(df['GEBN'] / df['GEBN'].shift(-1))
dfRets['SMIrets'] = np.log(df['SMI'] / df['SMI'].shift(-1))
print(dfRets.head())


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
The output shows that we fail to reject the null hypothesis that $\beta_{TV} = 0.0475$.
Beyond the element-wise hypothesis tests the regression summary also provides F-statistic (and the corresponding p-value) on the combined hypothesis that
$$\begin{align}
H_0&: \quad \beta_j = \beta_1, \beta_2, \ldots \beta_p = 0 \
H_a&: \quad \beta_j \neq 0 \text{ for at least one $j$}
\end{align}$$
On the basis of the corresponding p-value (i.e. 1.58e-96) we can reject the null at any reasonable significance level. Should we be interested in assessing a particular hypothesis, say
$$\begin{align}
H_0&: \quad \beta_{TV} = \beta_{\text{Radio}} = 0.1 \
H_a&: \quad \beta_{TV} \neq \beta_{\text{Radio}} \neq 0.1
\end{align}$$
we use the .f_test() method.
Can you write Python code for it?

# Test H0: beta(radio) = beta(newspaper) = 0.1
mlReg.f_test('radio = newspaper = 0.1')


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
The regression summary provides $t$-statistic and p-value for the null hypothesis $H_0: \hat{\beta}_j = 0$, $H_a: \hat{\beta}_j \neq 0$. You can call the resulting $t$-statistic and p-value with its attributes.
Can you write Python code for it?


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
Variable data is a dictionary with three entries: monhtly data, annual data and a description. We select the monthly rates in dictionary 0 and format the data.
Can you write Python code for it?

# Select monthly data
ff = data[0]

# Sort data in descending order
ff = ff.sort_index(ascending=False)

# Convert returns to decimal percentages
ff = ff/100
print(ff.head(3))


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
We are now in a position to run the multiple linear regression. We will again use the past 60 months. From the Fama French set we just need the first three columns. Column 4 is the risk free rate which we do not use.
Can you write Python code for it?

# Add constant to matrix for alphas (=intercept)
X = sm.add_constant(ff.iloc[:months, :3])

# Assign ticker to variable
tickers = shsRets.columns

# Create results matrix to paste beta factors
res = np.empty(shape=(5, len(tickers)))

# Run regression for each ticker
for i in range(0, len(tickers)):

    # Select returns of share i
    sub = shsRets.iloc[:months, i]
    # Run regression
    model = sm.OLS(endog=sub, exog=X).fit()
    # Paste beta factors to 'res' matrix
    res[0:4, i] = model.params
    res[4, i] = model.rsquared_adj

# Format output to dataframe
ff3f = pd.DataFrame(data=res, index=['Alpha', 'BetaMkt', 'BetaSMB', 'BetaHML', 'R2_adj'])
ff3f.columns = tickers


I am working on the file "0205_LinearRegression.ipynb".
The first task is:
What we calculated above is often referred to as the raw beta. The beta value of a stock has been found to be on average closer to the mean value of 1.0, the beta of an average-systematic-risk portfolio, than to the value of the raw beta (Pinto et al. (2016)). This is why data providers such as Bloomberg publish the adjusted beta as first introduced by Blume (1971), which is calculated as
$$ \text{Adjusted beta} = 2/3 \cdot \text{raw beta} + 1/3 \cdot 1$$
Now, let us assume we are given the task to investigate whether a beta indeed regresses to 1 over time. For that we could, as a starting point, assess a stock's rolling beta over the past years. Note that this is just an example of use. Computationally it would be much faster to calculate the stock beta via the covariance/variance formula.
Can you write Python code for it?

def rollingBeta(df, window=60):

    '''Calculates the running beta of a stock.

    Parameters
    ==========
    df : [n x 2] pandas dataframe with log-returns for
         stock and market portfolio. Index should be
         datetime series.
    window : rolling window with default value 60 [optional]

    Returns
    =======
    rb : Pandas dataframe with (backward-looking) rolling beta.
    '''

    # Drop NA rows from df
    df = df.dropna()
    # Set up empty results array
    res = np.empty(len(df) - window + 1)

    # Loop through df
    for i in range(0, len(df)):

        # As long as remaining subset is >= window, we proceed
        if (len(df) - i) >= window:

            # Subset df
            sub = df.iloc[i:window+i, :]
            # Run Regression
            model = sm.OLS(endog=sub.iloc[:, 0],
                           exog=sm.add_constant(sub.iloc[:, 1])).fit()
            # Read out beta coefficient
            res[i] = model.params[1]

    # Format output to dataframe
    rb = pd.DataFrame(data=res, index=df.index[:(len(df)-window+1)])
    rb.columns = ['RollingBeta']

    return rb


I am working on the file "0206_LogisticRegression.ipynb".
The first task is:
Classic GLM

endog = endogenous variable = response = $y$
exog = exogenous variable(s) = features = $X$
Notice that a constant needs to be added manually with sm.add_constant(X)

Function details are described here.
Can you write Python code for it?

# 2. GLM

# Regress model
logReg = sm.GLM(endog=df.defaultFac, exog=sm.add_constant(df.balance),
                family=sm.families.Binomial()).fit()
print(logReg.summary())


I am working on the file "0206_LogisticRegression.ipynb".
The first task is:
Here is an alternative way of plotting it. Notice that for the logistic function I do not use the data I trained it with but 'any possible' outcome.
Can you write Python code for it?

# Create sorted resutls-df
res = pd.DataFrame()
res['balance'] = np.linspace(start=0, stop=5000, num=10000)
res['prob'] = logReg.predict(sm.add_constant(res['balance']))

# Plot scatter & logReg
plt.figure(figsize=(12, 8))
plt.scatter(df.balance, df.defaultFac, marker='.')
plt.plot(res.balance, res.prob, c='k')
plt.axhline(y=0, color='gray', linestyle='dashed')
plt.axhline(y=1, color='gray', linestyle='dashed')
plt.ylabel('Probability of default', fontsize=12)
plt.xlabel('Balance', fontsize=12);


I am working on the file "0206_LogisticRegression.ipynb".
The first task is:
Hypothesis Testing
The $z$-statistic in above summary and the corresponding p-values play the same role as the $t$-statistic in the linear regression output. For example, the $z$-statistic associated with $\hat{\beta}_1$ is equal to $\hat{\beta}_1 / SE(\hat{\beta}_1)$. You can check this by running logReg.params[1] / logReg.bse[1]. A large (absolute) $z$-value, and correspondingly a small p-value, provides evidence against the null hypothesis $H_0 : \beta_1 = 0$. Notice that this null hypothesis implies that $p(X) = \dfrac{e^{\beta_0}}{(1 + e^{\beta_0})}$.
The main purpose of the intercept is to adjust the average fitted probabilities to the proportion of ones in the data. Beyond that it is not of interest.
If we are interested in the coefficient's 99% CI (instead of 95% like in the summary) we can use the same code as in the linear regression case.
Can you write Python code for it?


I am working on the file "0206_LogisticRegression.ipynb".
The first task is:
Making Predictions
If we wish to make predictions of the default probability we can use the .predict() method. Given we leave the brackets empty, this method will calculate $p(X)$ on the basis of the previously used feature training sample. If we wish to get the probability of default for a balance of, let us say, USD 2'000 we can run the following line of code:
Can you write Python code for it?

# X must include 1 in first column for intercept
logReg.predict([1, 2000])


I am working on the file "0206_LogisticRegression.ipynb".
The first task is:
Multiple Logistic Regression
Regressing 'default' on multiple features follows the same procedure as for simple logistic regression. As a case of use we regress 'default' on 'balance', a credit card holder's 'income' and the dummy variable 'studentFac' (1 if applicant is a student, 0 otherwise).
Can you write Python code for it?

# Assign features to matrix X and response to y
X = sm.add_constant(df[['balance', 'income', 'studentFac']])
X.income = X.income / 1000  # Income in 1'000 dollars
y = df.defaultFac

# Run Logistic Regression
logReg = sm.Logit(endog=y, exog=X).fit()
print(logReg.summary().tables[1])


I am working on the file "0206_LogisticRegression.ipynb".
The first task is:
Plotting
Above we regressed default on balance. Thus this can still be displayed in a 2d-figure. In order to do so we need to sort the data first. Once this is done, plotting follows the usual routine.
Can you write Python code for it?

# Create sorted resutls-df
res = pd.DataFrame()
res['balance'] = df.balance
res['prob'] = logReg.predict()
res = res.sort_values('balance')

# Plot scatter & logReg
plt.figure(figsize=(12, 8))
plt.scatter(df.balance, df.defaultFac, marker='.')
plt.plot(res.balance, res.prob, c='k')
plt.axhline(y=0, color='gray', linestyle='dashed')
plt.axhline(y=1, color='gray', linestyle='dashed')
plt.ylabel('Probability of default', fontsize=12)
plt.xlabel('Balance', fontsize=12);


I am working on the file "0207_Vectors-1549598493596.ipynb".
The first task is:
$\vec{a_b} = a_b \hat{b} = a_b \frac{\vec{b}}{|\vec{b}|}$
Can you write Python code for it?


I am working on the file "0207_Vectors-1549598493596.ipynb".
The first task is:
$a_b = |\vec{a}| \cos(\theta) = |\vec{a}|\frac{\vec{a}\cdot\vec{b}}{|\vec{a}||\vec{b}|} = \frac{\vec{a}\cdot\vec{b}}{|\vec{b}|}$
Can you write Python code for it?

a_b = np.dot(a, b)/np.linalg.norm(b)

print(a_b)


I am working on the file "0207_Vectors-1549598493596.ipynb".
The first task is:
Linear combination
$\vec{c} = w_1 \vec{a} + w_2 \vec{b}$
Can you write Python code for it?

def plot_linear_combination(a, b, w1, w2):

  plt.quiver(0,0,a[0],a[1], scale_units='xy', angles='xy', scale=1, color='r')
  plt.quiver(0,0,b[0],b[1], scale_units='xy', angles='xy', scale=1, color='b')

  c = w1 * a + w2 * b

  plt.quiver(0,0,c[0],c[1], scale_units='xy', angles='xy', scale=1, color='g')

  plt.xlim(-10,10)
  plt.ylim(-10,10)
  plt.show()

a = np.asarray([3, 4])
b = np.asarray([1.5, 2])

plot_linear_combination(a, b, -1, 1)

def plot_span(a, b):

  for i in range(1000):
    w1 = (np.random.random(1) - 0.5) * 3
    w2 = (np.random.random(1) - 0.5) * 3
    c = w1 * a + w2 * b
    plt.quiver(0,0,c[0],c[1], scale_units='xy', angles='xy', scale=1, color='g')

  plt.quiver(0,0,a[0],a[1], scale_units='xy', angles='xy', scale=1, color='r')
  plt.quiver(0,0,b[0],b[1], scale_units='xy', angles='xy', scale=1, color='b')

  plt.xlim(-10,10)
  plt.ylim(-10,10)
  plt.show()

plot_span(a, b)


I am working on the file "0207_Vectors-1549598493596.ipynb".
The first task is:
Vector addition and subtraction
Can you write Python code for it?

vecs = [np.asarray([0,0,3,4]), np.asarray([0,0,-3,4]), np.asarray([0,0,-3,-2]), np.asarray([0,0,4,-1])]

plot_vectors([vecs[0], vecs[3]])

vecs[0] + vecs[3]

plot_vectors([vecs[0], vecs[3], vecs[0] + vecs[3]])

plot_vectors([vecs[0], vecs[0], vecs[0] + vecs[0]])

plot_vectors([vecs[0], vecs[3], vecs[0] - vecs[3]])

plot_vectors([-vecs[0], vecs[3], - vecs[0] + (vecs[3])])


I am working on the file "0207_Vectors-1549598493596.ipynb".
The first task is:
Vector plotting
Can you write Python code for it?

plt.quiver(0,0,3,4)
plt.show()

plt.quiver(0,0,3,4, scale_units='xy', angles='xy', scale=1)
plt.show()

plt.quiver(0,0,3,4, scale_units='xy', angles='xy', scale=1)
plt.xlim(-10,10)
plt.ylim(-10,10)
plt.show()

plt.quiver(0,0,3,4, scale_units='xy', angles='xy', scale=1, color='r')
plt.quiver(0,0,-3,4, scale_units='xy', angles='xy', scale=1, color='g')
plt.xlim(-10,10)
plt.ylim(-10,10)
plt.show()

def plot_vectors(vecs):
  colors = ['r', 'b', 'g', 'y']
  i = 0
  for vec in vecs:
    plt.quiver(vec[0], vec[1], vec[2], vec[3], scale_units='xy', angles='xy', scale=1, color=colors[i%len(colors)])
    i += 1
  plt.xlim(-10,10)
  plt.ylim(-10,10)
  plt.show()

plot_vectors([(0,0,3,4), (0,0,-3,4), (0,0,-3,-2), (0,0,4,-1)])


I am working on the file "0207_kNN.ipynb".
The first task is:
As mentioned before, rows represent the true outcome and columns show what class KNN predicted. In 31 cases, the test set's true response was 'down' (in our case represented by 0) and KNN correctly predicted 'down'. 120 times KNN was correct in predicting an 'up' (=1) movement. 19 returns in the test set were positive but KNN predicted a negative return. And in 83 out of 253 cases KNN predicted an 'up' movement whereas in reality the stock price decreased. The KNN score of 59.68% for ABB is the sum of true positive and negative (31 + 120) in relation to the total number of predictions (253 = 31 + 19 + 83 + 120). The error rate is 1 - score or (19 + 83)/253.
Class-specific performance is also helpful to better understand results. The related terms are sensitivity and specifity. In the above case, sensitivity is the percentage of true 'up' movements that are identified. A good 86.3% (= 120 / (19 + 120)). The specifity is the percentage of 'down' movements that are correctly identified, here a poor 27.2% (= 31 / (31 + 83)). More on this in the next chapter.
Because confusion matrices are important to analyze results, Scikit-learn has its own command to generate it. It is part of the metrics sublibrary. The difficulty is that in contrast to above (manually generated) table, the function's output provides no labels. Therefore one must be sure to know which value are where. Here's the code to generate the confusion matrix.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

# Confusion matrix
confusion_matrix(y_test, pred)


I am working on the file "0207_kNN.ipynb".
The first task is:
Following finance theory, returns should be distributed symmetrically. Thus the simplest guess would be to expect a share price to increase on 50% of the days and to decrease on the remaining 50%. Similar to guessing a coin flip, if we would guess an 'up' movement for every day, we obviously would - in the long run - be correct 50% of the times. This would make for a score of 50%.
Looking in that light at the above summary, we see some very interesting results. For 10 out of 20 stocks KNN produces test scores of > 50% for even the 0.25th percentile. Let's plot the ones with the highest test-scores (ABBN, ZURN, NOVN, SIK) to see at what value of $k$ the best test-score is achieved.
Can you write Python code for it?

nms = ['ABBN', 'ZURN', 'NOVN', 'SIK']

plt.figure(figsize=(12, 8))
for col in nms:
    scr[col].plot(legend=True)
plt.axhline(0.50, c='k', ls='--');


I am working on the file "0207_kNN.ipynb".
The first task is:
For GIVN the maximum score is reached where $k=145$. You can check this with the scr['ABBN'].idxmax() command, which provides the index of the maximum value of the selected column. In our case, the index is equivalent to the value of $k$. Thus we run KNN with $k=145$.
Can you write Python code for it?

scr['ABBN'].idxmax()

# 4) Run KNN
# Instantiate KNN class for GIVN with k=145
knn = neighbors.KNeighborsClassifier(n_neighbors=145)
# Fit KNN classifier using training set
knn = knn.fit(X_train, y_train)

# 5) Extract test score for ABB
scr_ABBN = knn.score(X_test, y_test)
scr_ABBN


I am working on the file "0207_kNN.ipynb".
The first task is:
Further Ressources
In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

Batista, Gustavo, and Diego Furtado Silva, 2009, How k-nearest neighbor parameters affect its performance, in Argentine Symposium on Artificial Intelligence, 1–12, sn.
Fortmann-Roe, Scott, 2012, Understanding the Bias-Variance Tradeoff from website, http://scott.fortmann-roe.com/docs/BiasVariance.html, 08/15/17.
Guggenbuehler, Jan P., 2015, Predicting net new money using machine learning algorithms and newspaper articles, Technical report, University of Zurich, Zurich.
James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, An Introduction to Statistical Learning: With Applications in R (Springer Science & Business Media, New York, NY).
Müller, Andreas C., and Sarah Guido, 2017, Introduction to Machine Learning with Python (O’Reilly Media, Sebastopol, CA).
Russell, Stuart, and Peter Norvig, 2009, Artificial Intelligence: A Modern Approach (Prentice Hall Press, Upper Saddle River, NJ).
Ting, Kai Ming, 2011, Confusion matrix, in Claude Sammut, and Geoffrey I. Webb, eds., Encyclopedia of Machine Learning (Springer Science & Business Media, New York, NY).
Can you write Python code for it?


I am working on the file "0207_kNN.ipynb".
The first task is:
Having the data in a proper dataframe, we are now in a position to create the features and response values.
Can you write Python code for it?

# Calculate log-returns and label responses:
# 'direction' equals 1 if stock closed above
# previous day and 0 if it fell.
today = np.log(shsPr / shsPr.shift(1))
direction = np.where(today >= 0, 1, 0)

# Convert 'direction' to dataframe
direction = pd.DataFrame(direction, index=today.index, columns=today.columns)

# Lag1, 2: t-1 and t-2 returns; excl. smi (in last column)
Lag1 = np.log(shsPr.iloc[:, :-1].shift(1) / shsPr.iloc[:, :-1].shift(2))
Lag2 = np.log(shsPr.iloc[:, :-1].shift(2) / shsPr.iloc[:, :-1].shift(3))

# Previous day return for SMI index
smi  = np.log(shsPr.iloc[:, -1].shift(1) / shsPr.iloc[:, -1].shift(2))


I am working on the file "0207_kNN.ipynb".
The first task is:
Results & Analysis
Now let's see the results in an overview.
Can you write Python code for it?


I am working on the file "0207_kNN.ipynb".
The first task is:
The score of 59.68% is the very same as what we have seen in above summary statistic. Nothing new so far. (Recall that the score is the total of correctly predicted outcomes.)
However, the alert reader should by now raise some questions regarding our assumption that 50% of the returns should have been positive. In the long run, this might be true. But our training sample contained only 1'017 records and of these 534 were positive.
Can you write Python code for it?

# Percentage of 'up' days in training set
y_train.sum() / y_train.size


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
Error rate

Error rate = 1 - Score or Error rate = (FP + FN) / (TN + TP + FP + FN)
Fraction of (overall) incorrectly predicted classes
Also known as "Misclassification Rate"
Can you write Python code for it?

print((FP + FN) / (TN + TP + FP + FN))
print(1 - metrics.accuracy_score(y_test, y_pred))
print(1 - lda.score(X_test, y_test))


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
In a Jupyter notebook, to see all options you can simply type lda. and hit tab.
LDA Performance
Here are some basic metrics on how the LDA classifier performed on the training data.
Can you write Python code for it?

print('default-rate: {0: .4f}'.format(np.sum(y_train)/len(y_train)))
print('score:        {0: .4f}'.format(lda.score(X_train, y_train)))
print('error-rate:   {0: .4f}'.format(1-lda.score(X_train, y_train)))


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
LDA Classifier Object & Fit
Now we are in a position to run the LDA classifier. This, as you can see from the three lines below, is as easy as it gets.
Can you write Python code for it?

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

# Create LDA object and run classifier
lda = LDA(solver='lsqr')
lda = lda.fit(X_train, y_train)
lda


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
Overall, 3.33% of all observations defaulted. If we would simply label each entry as 'non-default' we would have an error rate of this magnitude. So, in comparison to this naive classifier, LDA seems to have some skill in predicting the default.

IMPORTANT NOTE: In order to be in line with James et al. (2015), the textbook for this course, we have not performed any train/test split of the data. Therefore we will use the same full matrix X_train and response vector y_train as test data. Performance metrics might be applied to both test and training data but in the end the results on the test set are those that we are ultimately interested in. To drive this point home, I have relabeled the  X_train and y_train to X_test, y_test.  Nevertheless, be aware that in this unique case it is the same data!

Let us print the confusion matrix introduced in the previous chapter to see the class-wise performance. For reference the confusion matrix is also printed as DataFrame but moving forward be sure to know that row represent the true values and columns predicted labels.
Can you write Python code for it?

# Relabel variables as discussed
X_test = X_train
y_test = y_train

# Predict labels
y_pred = lda.predict(X_test)

# Sklearn's confusion matrix
print(metrics.confusion_matrix(y_test, y_pred))

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted default status': y_pred,
                      'True default status': y_test})
confm.replace(to_replace={0:'No', 1:'Yes'}, inplace=True)
print(confm.groupby(['True default status','Predicted default status']).size().unstack('Predicted default status'))


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
ROC & Precision-Recall Curve in Sklearn Version 0.22.1
Starting with Scikit-learn version 0.22.1 the plotting of the ROC and Precision-Recall Curve was integrated into Scikit-learn and there's now a function available to cut the plotting work a bit short. Below two code snippets that show how to do it.
Can you write Python code for it?

# Plot Precision-Recall Curve
disp = metrics.plot_precision_recall_curve(lda, X_test, y_test);

disp = metrics.plot_roc_curve(lda, X_test, y_test);


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
Sensitivity or Recall

Sensitivity = TP / (TP + FN)
Fraction of correctly predicted 'positives' (e.g. 'defaults'). Basically asks the question: "When the actual value is positive, how often is the prediction correct?"
Also known as True positive rate
Counterpart to Precision
Can you write Python code for it?

print(TP / (TP + FN))
print(metrics.recall_score(y_test, y_pred))


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
So far we've encountered the following performance metrics:

Score
Error rate
Sensitivity and
Specificity.

We briefly recapture their meaning, how they are calculated and how to call them in Scikit-learn. We will make use of the functions in the metrics sublibrary of sklearn
Score

Score = (TN + TP) / (TN + TP + FP + FN)
Fraction of (overall) correctly predicted classes
Can you write Python code for it?

print((TN + TP) / (TN + TP + FP + FN))
print(metrics.accuracy_score(y_test, y_pred))
print(lda.score(X_test, y_test))


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
Specificity

Specificity = TN / (TN + FP)
Fraction of correctly predicted negatives (e.g. 'non-defaults')
Can you write Python code for it?


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
The above four classification performance metrics we already encountered. There are two more metrics we want to cover: Precision and the F-Score.
Precision

Precision = TP / (TP + FP)
Refers to the accuracy of a positive ('default') prediction. Basically asks the question: "When a positive value is predicted, how often is the prediction correct?"
Counterpart to Recall
Can you write Python code for it?

print(TP / (TP + FP))
print(metrics.precision_score(y_test, y_pred))


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
The confusion matrix tells us that for the non-defaulters, LDA only misclassified 22 of them. This is an excellent rate. However, out of the 333 (=253 + 80) people who actually defaulted, LDA classified only 80 correctly. This means our classifier missed out on 76.0% of those who actually defaulted! For a credit card applicant with a bad credit score this is good news. For a credit card company, not so much.
Varying the Threshold Levels
Why does LDA miss all these 'defaulters'? Implicitly, Bayes classifier minimizes the overall error rate, meaning that it yields the smallest possible total number of misclassified observations - irrespective of the class-specific error rate. Bayes classifier works by assigning an observation to class 'default' for which the posterior probability $Pr(\text{default = Yes}|X=x) > 0.5$. For a credit card company who seeks to have as few defaults as possible, this threshold might be too large. Instead, such a company might decide to label any customer with a posterior probability of default above 20% to the 'default' class ($Pr(\text{default = Yes}|X=x) > 0.2$). Let us investigate how the results in such a case would look like.
Can you write Python code for it?

# Calculated posterior probabilities
posteriors = lda.predict_proba(X_test)
posteriors[:5, :]


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
The parameter solver='lsqr' specifies the method by which the covariance matrix is estimated. lsqr follows the approach introduced in the preceding subsection. Others such as svd or eigen are available. See Scikit-learn's guide or the function description.
Every function in sklearn has different attributes and methods. Sklearns convention is to store anything that is derived from the data in attributes that end with a trailing underscore. That is to separate them from parameters that are set by the user (Mueller and Guido (2017)). For example the estimated covariance matrix can be printed with this command.
Can you write Python code for it?


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
The performance seems to be slightly better than with LDA. Let's plot the ROC curve for both LDA and QDA.
Can you write Python code for it?

# Compute ROC curve and ROC area (AUC) for each class
fpr_qda, tpr_qda, _ = metrics.roc_curve(y_test, posteriors_qda)
roc_auc_qda = metrics.auc(fpr_qda, tpr_qda)

plt.figure(figsize=(6, 6))
plt.plot(fpr, tpr, lw=2, label='LDA ROC (AUC = {0: 0.2f})'.format(roc_auc))
plt.plot(fpr_qda, tpr_qda, lw=2, label='QDA ROC (AUC = {0: 0.2f})'.format(roc_auc_qda))
plt.plot([0, 1], [0, 1], lw=2, c = 'k', linestyle='--')
plt.xlim([-0.01, 1.0])
plt.ylim([-0.01, 1.01])
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.title('ROC Curve', fontweight='bold', fontsize=18)
plt.legend(loc="lower right");


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
This one can easily visualize - done in the next code snippet. We also add some more information to the plot by displaying the Average Precision (AP) and the Area under the Curve (AUC). The former summarizes the plot in that it calculates the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight see further description here. The latter calculates the area under the curve using the trapezoidal rule. Notice that ideally the function hugs the top-right corner.
Can you write Python code for it?

# Calculate the average precisions score
y_dec_bry = lda.decision_function(X_test)
average_precision = metrics.average_precision_score(y_test, y_dec_bry)

# Calculate AUC
prec_recall_auc = metrics.auc(recall, precision)

# Plot Precision/Recall variations given different
# levels of thresholds
plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('2-class Precision-Recall curve: \n AP={0:0.2f} / AUC={1:0.2f}'.format(
          average_precision, prec_recall_auc));


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
With respect to Sensitivity (Recall) and Specificity LDA and QDA perform virtually identical. Therefore, one might give the edge here to QDA because of its slighly better Recall and $F_1$-Score.
Reality and the Gaussian Assumption for LDA & QDA
Despite the rather strict assumptions regarding normal distribution, LDA and QDA perform well on an amazingly large and diverse set of classification tasks. Friedman et al. (2001, p. 111) put it this way:

"Both techniques are widely used, and entire books are devoted to LDA. It seems that whatever exotic tools are the rage of the day, we should always have available these two simple tools. The question arises why LDA and QDA have such a good track record. The reason is not likely to be that the data are approximately Gaussian, and in addition for LDA that the covariances are approximately equal. More likely a reason is that the data can only support simple decision boundaries such as linear or quadratic, and the estimates provided via the Gaussian models are stable. This is a bias variance tradeoff - we can put up with the bias of a linear decision boundary because it can be estimated with much lower variance than more exotic alternatives. This argument is less believable for QDA, since it can have many parameters itself - although perhaps fewer than the non-parametric alternatives."

Whether LDA or QDA should be applied to categorical/binary features warrants a separate note. It is true that discriminant analysis was designed for continuous features (Ronald A. Fisher (1936)) where the underlying assumption is that the values are normally distributed. However, as above quote shows, studies have proven the robustness of the model even in light of violations of the rather rigid normality assumption. This is not only true for continuous features but also for categorical/binary features. For more details see Huberty et al. (1986). It follows that applying LDA and QDA is possible, though the user should cautiously control the output. We will discuss appropriate cross...
Can you write Python code for it?


I am working on the file "0208_LDA-QDA.ipynb".
The first task is:
classifier.predict_proba() we have also introduced above: it provides probabilities of $\Pr(y = 0|X=x)$ in the first column and $\Pr(y = 1|X=x)$ in the second.
Can you write Python code for it?


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
A great feature available in all discussed CV functions is the possibility to set the number of CPUs to use to do the computation on. For this define n_jobs=n, where n is the number of CPU's you want to use. n_jobs=-1 will use all available cores. This parallelization is especially helpful if you work on large data sets and/or computationally expensive tasks such as CV. If you want to know how many CPUs your computer runs on you can type the following code:
Can you write Python code for it?


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
A stratified sample is one that maintains the proportion of values as in the original data set. If, for example, the response vector $y$ is a binary categorical variable with 25% zeros and 75% ones, stratify=y ensures that the random splits have 25% zeros and 75% ones too. Note that stratify=y does not mean stratify=yes but rather tells the function to take the categorical proportions from response vector y.
Feature Scaling
Feature scaling is a crucial step in preparing data for ML applications. While some of the algorithms are invariant to the feature's scale (e.g. decision trees and random forests, which we will discuss in the next chapter), the majority of machine learning and optimization algorithms perform much better if features are scaled. The reasons are two-fold: for one, most ML algorithm have optimization functions to find the optimal coefficients/hyperparameter and these functions work more efficient on scaled values; for another, algorithms such as KNN, which use a distance measure, will put much more weight on features that have a larger scale than others (Raschka (2015), Müller and Guido (2017)). A good example for the latter is a data set that contains 'age' as well as 'income' variables. For a KNN model a difference in $1'000 salary is enormous compared to a difference of 10 years in age. Thus such a model would be driven by 'income' - and this would be contrary to our intuition (James et al. (2013)).
There are two common approaches to bringing different features onto the same scale: normalization and standardization. Unfortunately these terms are often used quite loosely in different fields so that the meaning has to be derived from the context they are mentioned (Raschka (2015)).
Normalization
In general, normalization refers to the process of rescaling the features to a range of $[0, 1]$. This can be viewed as a special case of min-max scaling. In our context, we normalize a feature column $X_i$ by the following convention:
$$\begin{equation}
X_{i}^{\text{norm}} = \frac{X_i -...
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

# Get cols to scale
cols_scl = X.columns.values[:6]

# Apply MinMaxScaler on continuous columns only
mms = MinMaxScaler()
X_train_norm = mms.fit_transform(X_train[cols_scl])  # fit & transform
X_test_norm  = mms.transform(X_test[cols_scl])  # ONLY transform


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
Again we fit the StandardScaler() only on the train set. The parameters of this first step (stdsc.mean_, stdsc.var_) are then applied to the test set with the code stdsc.transform(X_test). Note that the variance in StandardScaler() is based on the population variance (division by $n$; degree of freedom of 0, biased estimator). This explains differences if you compare stdsc.var_ with X_train.var() for small data sets. Use X_train.var(ddof=1) instead to get the same result as with the StandardScaler.
Other Scaling Methods
For our purposes the above introduced two scaling methods are sufficient. However, this is not to say others are not helpful and unnecessary. Scikit-learn of course offers many more scaling procedures. For an overview see Scikit-learn's guide on the topic. For those who like to see the effects of the different scaling methods in 2D plots, see Scikit-learn's tutorial "Compare the effect of different scalers on data with outliers". Finally, Sebastian Raschka published a text "About Feature Scaling and Normalization" that not only discusses the different scaling methods but also shows the positive effect scaling can have on ML predictions/scores. Please give it a read to understand the importance of scaling.
Cross Validation
Model Evaluation
To evaluate our models, we have discussed the necessity to split our data set into training and test set. For this we introduced the train_test_split() function. A ML algorithm is calibrated on the training set and then, to see how well our model generalizes to new, previously unseen data, applied to the test data. In this section we expand on aspects of evaluating a model by discussing cross validation (CV). This is a statistical method of performance evaluation that is more stable and thorough than a simple split into training and test set (Müller and Guido (2017)).
$k$-Fold Cross Validation
In $k$-fold CV we randomly split the training data set into $k$ folds without replacement, where $k − 1$
folds are used for the model training and the remaining fold...
Can you write Python code for it?

# Import necessary functions
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression

# Create k-Fold CV and LogReg object
kFold = StratifiedKFold(n_splits=5)
logReg = LogisticRegression(max_iter=1000)

# Run CV and print results
scores = cross_val_score(logReg, X_train_std, y_train, cv=kFold)
print(scores)
print('CV accuracy on train set: {0: .3f} +/- {1: .3f}'.format(np.mean(scores), np.std(scores)))


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
As a comparison we can also run a test on the unscaled data set X_train. The results are indeed worse as literature suggests.
Can you write Python code for it?

# Run CV on unscaled values and print results
scores = cross_val_score(logReg, X_train, y_train, cv=kFold)
print('CV accuracy on train set: {0: .3f} +/- {1: .3f}'.format(np.mean(scores), np.std(scores)))