I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
Comparing categorical label and regression
Since baby weight is a positive real value, this is intuitively a regression problem. However, we can train the model as a multi-class classification by bucketizing the output label. At inference time, the model then predicts a collection of probabilities corresponding to these potential outputs.
Let's do both and see how they compare.
Can you write Python code for it?

import os

import numpy as np
import pandas as pd
import tensorflow as tf

import matplotlib.pyplot as plt
from tensorflow.keras.utils import to_categorical
from tensorflow import keras
from tensorflow import feature_column as fc
from tensorflow.keras import layers, models, Model
%matplotlib inline

df = pd.read_csv("./data/babyweight_train.csv")


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
Create tf.data datsets for both classification and regression.
Can you write Python code for it?

# train/validation dataset for classification model
cls_train_data = tf.data.Dataset.from_tensor_slices((X_train.to_dict('list'), y_train_cls))
cls_valid_data = tf.data.Dataset.from_tensor_slices((X_valid.to_dict('list'), y_valid_cls))

# train/validation dataset for regression model
reg_train_data = tf.data.Dataset.from_tensor_slices((X_train.to_dict('list'), y_train_reg.values))
reg_valid_data = tf.data.Dataset.from_tensor_slices((X_valid.to_dict('list'), y_valid_reg.values))

# Examine the two datasets. Notice the different label values.
for data_type in [cls_train_data, reg_train_data]:
    for dict_slice in data_type.take(1):
        print("{}\n".format(dict_slice))

# create feature columns to handle categorical variables
numeric_columns = [fc.numeric_column("mother_age"),
                  fc.numeric_column("gestation_weeks")]

CATEGORIES = {
    'plurality': list(df.plurality.unique()),
    'is_male' : list(df.is_male.unique()),
    'mother_race': list(df.mother_race.unique())
}

categorical_columns = []
for feature, vocab in CATEGORIES.items():
    cat_col = fc.categorical_column_with_vocabulary_list(
        key=feature, vocabulary_list=vocab, dtype=tf.string)
    categorical_columns.append(fc.indicator_column(cat_col))

# create Inputs for model
inputs = {colname: tf.keras.layers.Input(
    name=colname, shape=(), dtype="float32")
    for colname in ["mother_age", "gestation_weeks"]}
inputs.update({colname: tf.keras.layers.Input(
    name=colname, shape=(), dtype=tf.string)
    for colname in ["plurality", "is_male", "mother_race"]})

# build DenseFeatures for the model
dnn_inputs = layers.DenseFeatures(categorical_columns+numeric_columns)(inputs)

# create hidden layers
h1 = layers.Dense(20, activation="relu")(dnn_inputs)
h2 = layers.Dense(10, activation="relu")(h1)

# create classification model
cls_output = layers.Dense(4, activation="softmax")(h2)
cls_model = tf.keras.models.Model(inputs=inputs, outputs=cls_output)
cls_model.compile(optimizer='adam',
             ...


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
First, train the classification model and examine the validation accuracy.
Can you write Python code for it?

# train the classifcation model
cls_model.fit(cls_train_data.batch(50), epochs=1)

val_loss, val_accuracy = cls_model.evaluate(cls_valid_data.batch(X_valid.shape[0]))
print("Validation accuracy for classifcation model: {}".format(val_accuracy))


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
Increasing the number of  categorical labels
We'll generalize the code above to accommodate N label buckets, instead of just 4.
Can you write Python code for it?

# Read in the data and preprocess
df = pd.read_csv("./data/babyweight_train.csv")

# prepare inputs
df.is_male = df.is_male.astype(str)

df.mother_race.fillna(0, inplace = True)
df.mother_race = df.mother_race.astype(str)

# create categorical label
MIN = np.min(df.weight_pounds)
MAX = np.max(df.weight_pounds)
NBUCKETS = 50

def categorical_weight(weight_pounds, weight_min, weight_max, nbuckets=10):
    buckets = np.linspace(weight_min, weight_max, nbuckets)

    return np.digitize(weight_pounds, buckets) - 1

df["weight_category"] = df.weight_pounds.apply(lambda x: categorical_weight(x, MIN, MAX, NBUCKETS))

def encode_labels(classes):
    one_hots = to_categorical(classes)
    return one_hots

FEATURES = ['is_male', 'mother_age', 'plurality', 'gestation_weeks', 'mother_race']
LABEL_COLUMN = ['weight_category']

N_TRAIN = int(df.shape[0] * 0.80)

X_train, y_train = df[FEATURES][:N_TRAIN], encode_labels(df[LABEL_COLUMN][:N_TRAIN])
X_valid, y_valid = df[FEATURES][N_TRAIN:], encode_labels(df[LABEL_COLUMN][N_TRAIN:])

# create the training dataset
train_data = tf.data.Dataset.from_tensor_slices((X_train.to_dict('list'), y_train))
valid_data = tf.data.Dataset.from_tensor_slices((X_valid.to_dict('list'), y_valid))


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
Next, we'll train the regression model and examine the validation RMSE.
Can you write Python code for it?

# train the classifcation model
reg_model.fit(reg_train_data.batch(50), epochs=1)

val_loss, val_mse = reg_model.evaluate(reg_valid_data.batch(X_valid.shape[0]))
print("Validation RMSE for regression model: {}".format(val_mse**0.5))


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
Restricting the prediction range
One way to restrict the prediction range is to make the last-but-one activation function sigmoid instead, and add a lambda layer to scale the (0,1) values to the desired range. The drawback is that it will be difficult for the neural network to reach the extreme values.
Can you write Python code for it?

import numpy as np
import tensorflow as tf
from tensorflow import keras

MIN_Y =  3
MAX_Y = 20
input_size = 10
inputs = keras.layers.Input(shape=(input_size,))
h1 = keras.layers.Dense(20, 'relu')(inputs)
h2 = keras.layers.Dense(1, 'sigmoid')(h1)  # 0-1 range
output = keras.layers.Lambda(lambda y : (y*(MAX_Y-MIN_Y) + MIN_Y))(h2) # scaled
model = keras.Model(inputs, output)

# fit the model
model.compile(optimizer='adam', loss='mse')
batch_size = 2048
for i in range(0, 10):
    x = np.random.rand(batch_size, input_size)
    y = 0.5*(x[:,0] + x[:,1]) * (MAX_Y-MIN_Y) + MIN_Y
    model.fit(x, y)

# verify
min_y = np.finfo(np.float64).max
max_y = np.finfo(np.float64).min
for i in range(0, 10):
    x = np.random.randn(batch_size, input_size)
    y = model.predict(x)
    min_y = min(y.min(), min_y)
    max_y = max(y.max(), max_y)
print('min={} max={}'.format(min_y, max_y))


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
The classification model predicts a probability for each bucket of values.
Can you write Python code for it?

preds = cls_model.predict(x={"gestation_weeks": tf.convert_to_tensor([38]),
                             "is_male": tf.convert_to_tensor(["True"]),
                             "mother_age": tf.convert_to_tensor([28]),
                             "mother_race": tf.convert_to_tensor(["1.0"]),
                             "plurality": tf.convert_to_tensor(["Single(1)"])},
                          steps=1).squeeze()
preds

objects = ('very_low', 'low', 'average', 'high')
y_pos = np.arange(len(objects))
predictions = list(preds)

plt.bar(y_pos, predictions, align='center', alpha=0.5)
plt.xticks(y_pos, objects)
plt.title('Baby weight prediction')

plt.show()


I am working on the file "03_problem_representation/reframing.ipynb".
The first task is:
We'll use the same features for both models. But we need to create a categorical weight label for the classification model.
Can you write Python code for it?

# prepare inputs
df.is_male = df.is_male.astype(str)

df.mother_race.fillna(0, inplace = True)
df.mother_race = df.mother_race.astype(str)

# create categorical label
def categorical_weight(weight_pounds):
    if weight_pounds < 3.31:
        return 0
    elif weight_pounds >= 3.31 and weight_pounds < 5.5:
        return 1
    elif weight_pounds >= 5.5 and weight_pounds < 8.8:
        return 2
    else:
        return 3

df["weight_category"] = df.weight_pounds.apply(lambda x: categorical_weight(x))

df.head()

def encode_labels(classes):
    one_hots = to_categorical(classes)
    return one_hots

FEATURES = ['is_male', 'mother_age', 'plurality', 'gestation_weeks', 'mother_race']

LABEL_CLS = ['weight_category']
LABEL_REG = ['weight_pounds']

N_TRAIN = int(df.shape[0] * 0.80)

X_train = df[FEATURES][:N_TRAIN]
X_valid = df[FEATURES][N_TRAIN:]

y_train_cls = encode_labels(df[LABEL_CLS][:N_TRAIN])
y_train_reg = df[LABEL_REG][:N_TRAIN]

y_valid_cls = encode_labels(df[LABEL_CLS][N_TRAIN:])
y_valid_reg = df[LABEL_REG][N_TRAIN:]


I am working on the file "03_problems.ipynb".
The first task is:
11. Draw a ASCII chrismas tree and write it to a file
Merry christmas. Feel free to add some UNICODE decoration.
Solution:
Can you write Python code for it?


I am working on the file "03_problems.ipynb".
The first task is:
2. Create all products of the two lists
Can you write Python code for it?


I am working on the file "03_problems.ipynb".
The first task is:
4. Filter the list
Filter out all the negativ numbers.
Can you write Python code for it?

import random

random_data = [random.random() - 0.5 for _ in range(10000)]


I am working on the file "03_problems.ipynb".
The first task is:
5. Create a parabola function
Define your own parabola function $f(x) = a (x - b)^2 - c$.
Solution:
Can you write Python code for it?

def parabola(x, a = 1, b=0, c=0):
    return a * (x - b)**2 - c

parabola(0, a = 0.5, b=-2, c=3)


I am working on the file "03_problems.ipynb".
The first task is:
7. Differntiate a sin function numerical
Use the given dataset
Can you write Python code for it?

import math
import matplotlib.pyplot as plt
%matplotlib inline

xdata = [2 *  math.pi  * x / 1000 for x in range(1000)]
ydata = [math.sin(x) for x in xdata]


I am working on the file "03_problems.ipynb".
The first task is:
9. Create a Histogram
Create a histogram from the poisson data array. Do not use any build in numpy or histogram plot function.
Can you write Python code for it?

from numpy.random import poisson

data = poisson(lam = 50, size=int(10e6))

# Don't worry too much about the next line it just prints nicely
print(str(data[:5])[1:-1], '...', str(data[-5:])[1:-1])


I am working on the file "03_problems.ipynb".
The first task is:
Plot everything
Can you write Python code for it?

plt.figure(figsize=(12, 3))
plt.plot(xdata, ydata, label=r'$sin(x)$')
plt.plot(xdata[1:], dy_data, label=r'$\frac{\Delta sin(x)}{\Delta x} $')
plt.legend()
plt.xlim(0, 6.28)


I am working on the file "03_problems.ipynb".
The first task is:
Solution:
Can you write Python code for it?

s = 50
smoothed_data = [sum(noisy_data[i:i+s]) / s  for i in range(len(noisy_data) - s)]

plt.figure(figsize=(12,3))
plt.plot(xdata, noisy_data, ls='', marker='.', ms=3)
plt.plot(xdata[s//2:-s//2], smoothed_data, color='red', lw=3)

plt.xlim(0, 6.28)


from collections import Counter

def plot_histogram(data, normed=False, *args, **kwargs):

    histo = Counter(data)

    bins = list(histo.keys())
    freqs = list(histo.values())

    if normed:
        n = sum(freqs)
        freqs = [freq / n for freq in freqs]

    plt.plot(bins, freqs, *args, **kwargs)

plot_histogram(data, normed=True, ls='--', marker='o', ms=5, color='red')


neg_data = [i for i in random_data if i < 0]

neg_data[:10]


I am working on the file "03_problems.ipynb".
The first task is:
Solution:
Use slicing and zip() to differentiate
Can you write Python code for it?


I am working on the file "03_problems.ipynb".
The first task is:
Solution:
Using python build in dict and check for key with if in
Can you write Python code for it?

histo = {}

for value in data:
    if value in histo:
        histo[value] += 1
    else:
        histo[value] = 1


I am working on the file "03_problems.ipynb".
The first task is:
Using counter from collections
Can you write Python code for it?

from collections import Counter

histo = Counter(data)

histo


I am working on the file "03_problems.ipynb".
The first task is:
Using the defaultdict() from collections with int()
Can you write Python code for it?

from collections import defaultdict

histo = defaultdict(int)

for value in data:
    histo[value] += 1


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Can use Unicode strings with NLTK tokenizers
Can you write Python code for it?


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Dealing with HTML
Get an "urban legend" article called Blondes to die out in 200 years -- from the BBC
Can you write Python code for it?

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
html = request.urlopen(url).read().decode('utf8')
html[:60]

type(html)

from bs4 import BeautifulSoup
raw = BeautifulSoup(html, 'html.parser').get_text()
tokens = word_tokenize(raw)
tokens[:50]


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Doing More with Word Pieces
Remove internal vowels from words
Can you write Python code for it?

regexp = r'^[AEIOUaeiou]+|[AEIOUaeiou]+$|[^AEIOUaeiou]'

def compress(word):
    pieces = re.findall(regexp, word)
    return ''.join(pieces)

re.findall(regexp, 'Universal')

english_udhr = nltk.corpus.udhr.words('English-Latin1')
print(nltk.tokenwrap(compress(w) for w in english_udhr[:75]))


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Frequencies for sequences of 2+ vowels in the text
Can you write Python code for it?

wsj = sorted(set(nltk.corpus.treebank.words()))
len(wsj)

fd = nltk.FreqDist(vowels for word in wsj
                          for vowels in re.findall(r'[aeiou]{2,}', word))
len(fd)

fd.most_common(12)


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Lemmatization
WordNet lemmatizer only removes affixes for words in its dictionary
dictionary lookup process is much slower than Porter stemmer
Can you write Python code for it?

wnl = nltk.WordNetLemmatizer()
[wnl.lemmatize(t) for t in tokens]


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Normalizing Text
Can you write Python code for it?

raw = """DENNIS: Listen, strange women lying in ponds distributing swords
is no basis for a system of government.  Supreme executive power derives from
a mandate from the masses, not from some farcical aquatic ceremony."""
tokens = word_tokenize(raw)
tokens


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Project Gutenberg is a collocation for this text because it is included as a header and possibly footer for the raw text file
find start and end manually using find
Can you write Python code for it?


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Regular Expressions for Detecting Word Patterns
skipping this section
cheatsheet:
Operator    Behavior
.        Wildcard, matches any character
^abc     Matches some pattern abc at the start of a string
abc$     Matches some pattern abc at the end of a string
[abc]    Matches one of a set of characters
[A-Z0-9] Matches one of a range of characters
ed|ing|s Matches one of the specified strings (disjunction)
*        Zero or more of previous item, e.g. a*, [a-z]* (also known as Kleene Closure)
+        One or more of previous item, e.g. a+, [a-z]+
?        Zero or one of the previous item (i.e. optional), e.g. a?, [a-z]?
{n}      Exactly n repeats where n is a non-negative integer
{n,}     At least n repeats
{,n}     No more than n repeats
{m,n}    At least m and no more than n repeats
a(b|c)+  Parentheses that indicate the scope of the operators

Useful Applications of Regular Expressions
Extracting Word Pieces
find all vowels in a word and count them
Can you write Python code for it?

import re

word = 'supercalifragilisticexpialidocious'
vowel_matches = re.findall(r'[aeiou]', word)
vowel_matches

len(vowel_matches)


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Regular Expressions for Tokenizing Text
Simple Approaches to Tokenization
Can you write Python code for it?

raw = """'When I'M a Duchess,' she said to herself, (not in a very hopeful tone
though), 'I won't have any pepper in my kitchen AT ALL. Soup does very
well without--Maybe it's always pepper that makes people hot-tempered,'..."""


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Searching Tokenized Text
"<a> <man>" finds all instances of a man in the text
Can you write Python code for it?

from nltk.corpus import gutenberg, nps_chat
moby = nltk.Text(gutenberg.words('melville-moby_dick.txt'))
moby.findall(r"<a> (<.*>) <man>")

chat = nltk.Text(nps_chat.words())
chat.findall(r"<.*> <.*> <bro>")

chat.findall(r"<l.*>{3,}")


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Segmenting a stream of characters into sentences: sent_tokenize
Can you write Python code for it?

import pprint

text = nltk.corpus.gutenberg.raw('chesterton-thursday.txt')
sents = nltk.sent_tokenize(text)
pprint.pprint(sents[79:89])


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Stemmers
"off-the-shelf" stemmers included in NLTK
* Porter
* Lancaster
Can you write Python code for it?

porter = nltk.PorterStemmer()
lancaster = nltk.LancasterStemmer()
[porter.stem(t) for t in tokens]

[lancaster.stem(t) for t in tokens]


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
Using Python string methods and re with Unicode characters
Can you write Python code for it?

line

line.find('zosta\u0142y')

line = line.lower()
line

line.encode('unicode_escape')

import re
m = re.search('\u015b\w*', line)
m.group()

m.group().encode('unicode_escape')


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
allow internal hyphens and apostrophes in words
Can you write Python code for it?


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
downloading Crime and Punishment**
Can you write Python code for it?

from urllib import request
url = 'http://www.gutenberg.org/files/2554/2554.txt'
response = request.urlopen(url)
raw = response.read().decode('utf8')
type(raw)


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
find collocations (words that frequently appear together)
Can you write Python code for it?


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
find start and end indices of the content (manually) and create a Text object
Can you write Python code for it?

tokens = tokens[110:390]
text = nltk.Text(tokens)
text


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
re offers \w (word characters) and \W (all characters except letters, digits, _ )
split on nonword characters:
Can you write Python code for it?


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
skipping these sections...

Processing Search Engine Results

Processing RSS Feeds

Reading Local Files

Strings: Text Processing at the Lowest Level
skipping basic string and list operations

Text Processing with Unicode
Can you write Python code for it?

path = nltk.data.find('corpora/unicode_samples/polish-lat2.txt')
path

with open(path, encoding='latin2') as f:
    for line in f:
        line_strip = line.strip()
        print(line_strip)

with open(path, encoding='latin2') as f:
    for line in f:
        line_strip = line.strip()
        print(line_strip.encode('unicode_escape'))

import unicodedata
with open(path, encoding='latin2') as f:
    lines = f.readlines()

line = lines[2]
print(line.encode('unicode_escape'))

for c in line:
    if ord(c) > 127:
        print('{} U+{:04x} {}'.format(c.encode('utf8'), ord(c), unicodedata.name(c)))

for c in line:
    if ord(c) > 127:
        print('{} U+{:04x} {}'.format(c, ord(c), unicodedata.name(c)))


I am working on the file "03_processing_raw_text.ipynb".
The first task is:
split on whitespace
Can you write Python code for it?

re.split(r' ', raw)

re.split(r'[ \t\n]+', raw)


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Assign global constants
Can you write Python code for it?

# data-related constants
IN_FILE_PATH = '../data/loan_clean.csv'
Y            = 'STD_IMP_REP_loan_amnt'
DROPS        = ['id', 'GRP_REP_home_ownership', 'GRP_addr_state', 'GRP_home_ownership',
                'GRP_purpose', 'GRP_verification_status', '_WARN_']

# model-related constants
LEARN_RATE   = 0.05 # how much each gradient descent step impacts parameters
CONV         = 1e-10 # desired precision in parameters
MAX_ITERS    = 10000 # maximum number of gradient descent steps to allow


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Define function for executing gradient descent minimization
For each gradient descent step:
* Predictions are made using the current model parameters.
* The gradient is calculated for each model pararmeter.
* The gradient is used in combination with the learning rate to update each parameter.
Can you write Python code for it?

def grad_descent(X, y, learn_rate, max_iters, sgd_mini_batch_n=0, lambda_=0):

    """ Routine for executing simple gradient descent with stochastic gradient descent option.

    :param X: Matrix of numeric data.
    :param y: Vector of known target values.
    :param learn_rate: Learning rate.
    :param max_iters: Maximum number of gradient descent steps to perform.
    :param sgd_mini_batch_n: Minibatch size for sgd optimization.
                             If > 0 minibatch stochastic gradient descent is performed.
    :param lambda_: Scale factor for L2 regularization, default 0.

    """

    tic = time.time()               # start timer
    n_betas = X.shape[1]            # number of model parameters including bias
    betas = np.zeros(shape=n_betas) # parameters start with value of 0
    n = y.shape[0]                  # number of rows in X

    # Pandas dataframe for iteration history
    iteration_frame = pd.DataFrame(columns=['Iteration', 'Loss'])

    print('Iteration history:')

    # loop for gradient descent steps
    for i in range(max_iters):

        # stochastic gradient descent
        if sgd_mini_batch_n > 0:

            samp_idx = np.random.randint(n, size=sgd_mini_batch_n)
            X_samp = X[samp_idx, :]
            y_samp = y[samp_idx]
            n_samp = X_samp.shape[0]
            yhat_samp = X_samp.dot(betas) # model predictions for iteration

            # loop for column-wise parameter updates
            for j in range(n_betas):

                # select column
                # calculate column-wise gradient
                # update corresponding parameter based on negative gradient
                # calculate loss

                xj_samp = X_samp[:, j]
                beta_j = betas[j]
                xj_grad_samp = grad(n_samp, y_samp, yhat_samp, xj_samp, beta_j, lambda_)
                betas[j] = betas[j] - learn_rate * xj_grad_samp.sum()
                iter_loss = squared_loss(n_samp, X_samp, y_samp, betas, lambda_)
      ...


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Define gradient of loss function

The derivative of the loss function w.r.t the model parameters is used to update model parameters at each gradient descent step.
The gradient of our MSE loss function is trivial:
Can you write Python code for it?

def grad(n, y, yhat, x, beta_j, lambda_=0):

    """ Analytical gradient of scaled MSE loss function with L2 regularization.

    :param n: Number of rows in X.
    :param y: Vector of known target values.
    :param yhat: Vector of predicted target values.
    :param x: Vector of input values.
    :param beta_j: Model parameter for which to calculate gradient.
    :param lambda_: Scale factor for L2 regularization, default 0.
    :return: Vector of gradient values.

    """

    return (1 / n) * (x * (yhat - y) + (lambda_ * beta_j))


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Execute L2 penalized gradient descent
Can you write Python code for it?

grad_descent(X, y, LEARN_RATE, MAX_ITERS, lambda_=0.01)


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Execute L2 penalized stochastic gradient descent
Can you write Python code for it?

grad_descent(X, y, LEARN_RATE, MAX_ITERS, lambda_=0.01, sgd_mini_batch_n=1000)


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Execute gradient descent
Can you write Python code for it?

grad_descent(X, y, LEARN_RATE, MAX_ITERS)


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Execute stochastic gradient descent
Can you write Python code for it?

grad_descent(X, y, LEARN_RATE, MAX_ITERS, sgd_mini_batch_n=1000)


I am working on the file "03_regression/src/py_part_3_linear_regression_gradient_descent.ipynb".
The first task is:
Import clean data and convert to numpy matrices
Can you write Python code for it?

# import data using Pandas
raw = pd.read_csv(IN_FILE_PATH)
raw.describe()

# select target column
y = raw[Y].as_matrix()
print(y)

# create input matrix
# add an additional column of 1's for intercept
# by overlaying inputs onto matrix of 1's
numeric = raw.drop(DROPS + [Y], axis=1).as_matrix()
N, p = numeric.shape
X = np.ones(shape=(N, p + 1))
X[:,1:] = numeric


I am working on the file "03_reinforcement_learning_basic.ipynb".
The first task is:
'0' bin means the algorithm did not converge
Policy gradient algorithm
Can you write Python code for it?

import tensorflow as tf

tf.reset_default_graph()

n_state_dims = env.observation_space.shape[0]
n_action_dims = env.action_space.shape[0]
n_state_dims, n_action_dims


I am working on the file "03_reinforcement_learning_basic.ipynb".
The first task is:
Because the action space is discrete, we only need to take the sign of the linear regression output. In this case the weights can be sampled uniformly in [-1, 1]. However, if we want to use the bias term, it should be sampled according to the boundaries of the observation space (?)
Can you write Python code for it?

# generate model weights: [n_models, n_params]
models = 2 * np.random.random([N_MODELS, env.observation_space.shape[0]]) - 1 # uniform in [-1, +1]

class LinearAgent(object):
    def __init__(self, weights=None):
        self._weights = weights

    @property
    def weights(self):
        return self._weights

    @weights.setter
    def weights(self, new_weights):
        self._weights = new_weights

    def action(self, state):
        a = np.dot(self._weights, state)
        return int(a > 0)

def play_episode(env, agent, max_steps=MAX_STEPS):
    observations = []
    rewards = []
    o = env.reset()
    for t in range(max_steps):
        a = agent.action(o)
        o, r, done, info = env.step(a)
        rewards.append(r)
        observations.append(o)
        if done:
            break
    return observations, rewards

def evaluate_agent(env, agent, n_experiments=20, max_steps=MAX_STEPS):
    episode_rewards = []
    for e in range(n_experiments):
        observations, rewards = play_episode(env, agent, max_steps)

        avg_reward = np.array(rewards).sum() / max_steps
        episode_rewards.append(avg_reward)

    episode_rewards = np.array(episode_rewards)
    return episode_rewards.sum(), episode_rewards.mean()

model_metrics = np.zeros((models.shape[0], 2)) # total and mean reward for each model
for i, m in tqdm(enumerate(models)):
    agent = LinearAgent(m)
    model_metrics[i, :] = evaluate_agent(env, agent)

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('ggplot')

plt.hist(model_metrics[:,1], bins=50)
plt.title('Distribution of expected cumulative reward')
plt.xlabel('mean reward')
plt.show()

print('% of successful models', (model_metrics[:,1] == 1.0).mean())


I am working on the file "03_reinforcement_learning_basic.ipynb".
The first task is:
Check action and observation spaces dimensions
Can you write Python code for it?


I am working on the file "03_reinforcement_learning_basic.ipynb".
The first task is:
The hill-climbing algorithm
Can you write Python code for it?

n_experiments = 500
noise_scale = 1.0
experiment_steps = np.zeros(n_experiments)
for i in tqdm(range(n_experiments)):
    old_weights = 2 * np.random.random(env.observation_space.shape) - 1 # uniform in [-1, +1]
    agent = LinearAgent()
    r_best = 0.0
    for step in range(MAX_STEPS):
        noise = np.random.randn(*old_weights.shape)
        new_weights = old_weights + noise_scale * noise

        agent.weights = new_weights
        r_total, r_mean = evaluate_agent(env, agent)
        if r_mean == 1.0:
            experiment_steps[i] = step + 1
            break
        if r_mean > r_best:
            r_best = r_mean
            old_weights = new_weights

plt.hist(experiment_steps, bins=40)
plt.title('Distribution of steps for hill-climbing algorithm')
plt.xlabel('steps')
plt.show()


I am working on the file "03_reinforcement_learning_basic.ipynb".
The first task is:
Visualize the observation space
Sample observations using random policies
Can you write Python code for it?

observations = []
for m_id in np.random.randint(0, models.shape[0], size=200):
    agent = LinearAgent(models[m_id])
    os, rs = play_episode(env, agent)
    observations.extend(os)

observations = np.array(observations)


I am working on the file "03_reinforcement_learning_basic.ipynb".
The first task is:
build graph
Can you write Python code for it?

with tf.name_scope('policy'):
    states_in = tf.placeholder(tf.float32, [None, n_state_dims], name='states')
    W = tf.Variable(tf.truncated_normal([n_state_dims, n_action_dims], stddev=0.1), name='W')
    # b = tf.Variable(tf.constant(0.0, shape=[n_action_dims]), name='b')

    logits = tf.matmul(states_in, W)
    probs = tf.nn.softmax(logits, name='probabilities')

with tf.name_scope('policy_gradient'):
    actions_in = tf.placeholder(tf.int32, [None], name='actions')
    rewards_in = tf.placeholder(tf.float32, [None], name='cumulative_rewards')

    actions_one_hot = tf.one_hot(actions_in, n_action_dims)
    # because the actions are mutually exclusive it should be ok to use cross-entropy
    entropies = tf.nn.softmax_cross_entropy_with_logits(logits, actions_one_hot)
    weighted_entropies = tf.mul(entropies, rewards_in)
    loss = tf.reduce_sum(weighted_entropies, name='loss')

with tf.name_scope('optimizer'):
    global_step = tf.Variable(0, name='global_step', trainable=False)
    train_op = tf.train.AdamOptimizer(0.003).minimize(loss, global_step=global_step)


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Define section and perform first calculations
Can you write Python code for it?


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Define sympy parameters used for geometric description of sections
Can you write Python code for it?

A, A0, t, t0, a, b, h, L = sympy.symbols('A A_0 t t_0 a b h L', positive=True)


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Expression of Inertial properties wrt Center of Gravity in with original rotation
Can you write Python code for it?


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Import Section class, which contains all calculations
Can you write Python code for it?


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Multiconnected Section
Define graph describing the section:
1) stringers are nodes with parameters:
- x coordinate
- y coordinate
- Area
2) panels are oriented edges with parameters:
- thickness
- lenght which is automatically calculated
Can you write Python code for it?


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Plot of S3 section in inertial reference Frame
Section is plotted wrt center of gravity and rotated (if necessary) so that x and y are principal axes.
Center of Gravity and Shear Center are drawn
Can you write Python code for it?

positions={ii: [float(S3.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S3.g.nodes() }

x_ct, y_ct = S3.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S3.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
Resulting fluxes and coordinate
Can you write Python code for it?


I am working on the file "03a_Multiconnected_section.ipynb".
The first task is:
We also define numerical values for each symbol in order to plot scaled section and perform calculations
Can you write Python code for it?

values = [(A, 400 * ureg.millimeter**2),(A0, 250  * ureg.millimeter**2),(a, 400 * ureg.millimeter), \
          (b, 300 * ureg.millimeter),(h, 150 * ureg.millimeter),(L, 650 * ureg.millimeter),(t, 3 * ureg.millimeter)]
datav = [(v[0],v[1].magnitude) for v in values]


I am working on the file "03b - Some more advanced indexing.ipynb".
The first task is:
But you can also have multiple columns as the index, leading to a multi-index or hierarchical index:
Can you write Python code for it?

c = cast.set_index(['title', 'year'])

c.head()

%%time
c.loc[('Hamlet', 2000),:]

c2 = c.sort_index()

%%time
c2.loc[('Hamlet', 2000),:]


I am working on the file "03b - Some more advanced indexing.ipynb".
The first task is:
Instead of doing:
Can you write Python code for it?


I am working on the file "03b - Some more advanced indexing.ipynb".
The first task is:
Setting columns as the index
Why is it useful to have an index?

Giving meaningful labels to your data -> easier to remember which data are where
Unleash some powerful methods, eg with a DatetimeIndex for time series
Easier and faster selection of data

It is this last one we are going to explore here!
Setting the title column as the index:
Can you write Python code for it?


I am working on the file "03b - Some more advanced indexing.ipynb".
The first task is:
This dataset is borrowed from the PyCon tutorial of Brandon Rhodes (so all credit to him!). You can download these data from here: titles.csv and cast.csv and put them in the /data folder.
Can you write Python code for it?

cast = pd.read_csv('data/cast.csv')
cast.head()

titles = pd.read_csv('data/titles.csv')
titles.head()


I am working on the file "03b - Some more advanced indexing.ipynb".
The first task is:
we can now do:
Can you write Python code for it?


I am working on the file "03b_inverzna_funkcija.ipynb".
The first task is:
<< nazaj: funkcije naprej: bisekcija >>
Can you write Python code for it?

import disqus
%reload_ext disqus
%disqus matpy


I am working on the file "03b_inverzna_funkcija.ipynb".
The first task is:
Vrednost $\log_2(3)$ je rešitev enačbe $3=e^x$ in je približno enaka $1.6$.
Can you write Python code for it?


I am working on the file "03c_bisekcija.ipynb".
The first task is:
Vidimo, da sta $f(0)$ in $f(1)$ različnega predznaka, kar pomeni, da je na intervalu $(0,1)$ ničla.
Can you write Python code for it?

def bisekcija(f,a,b,n):
    """bisekcija(f,a,b,n) z bisekcijo izračuna interval širine (b-a)/2**n na katerem leži ničla funkcije f."""
    if n<=0:
        return (a,b)
    fa, fb = (f(a),f(b))
    assert (fa*fb)<=0, "Predznaka v krajiščih intervala [%f,%f] sta enaka" % (a,b)
    c = (a+b)/2 # središče intervala
    fc = f(c)
    if fc == 0:
        return (c,c)
    elif fc*fa<=0:
        return bisekcija(f,a,c,n-1)
    else:
        return bisekcija(f,c,b,n-1)

a,b = (0,1)
# 10 korakov bisekcije
for i in range(10):
    print(bisekcija(f,a,b,i))


I am working on the file "03d_iteracija.ipynb".
The first task is:
Odvod v rešitvi je približno -0.44, kar je po absolutni vrednosti manj od 1. To pomeni, da zaporedje približkov konvergira.
Hitrost konvergence
Ko iščemo rešitev enačbe z rekurzivnim zaporedjem, nas seveda zanima, koliko korakov je potrebnih za določeno število decimalk. To najlažje predstavimo z grafom napake v logaritemski skali.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
xp = sym.solve(sym.Eq(x,g(x)),x)[0].evalf() # točna rešitev

n = 30;
xz = [1] # zaporedje približkov
for i in range(n-1):
    xz.append(g(xz[-1]))
napaka = [x - xp for x in xz] # zadnji približek vzamemo za točno rešitev
plt.semilogy(range(n),napaka,'o')
plt.title("Napaka pri računanju rešitve z navadno iteracijo")


I am working on the file "03explore.ipynb".
The first task is:
First, a word about the below code.
In the accompanying func.py there is a function called parse_train that returns a pandas.Series object. For some reason, when it's returned from a map or apply, it seems to get cast as a string. When applied to a list or a dataframe, this string gets turned into a single field in the row, OR divided into several rows, throwing the count off.
To get around this, I return the results of the parse_train function and then CAST it back to a series. This adds a weird 0 index, which I delete. I then fill in the plethora of NaNs and recombine it with the primary dataframe.
For context, previous iterations included
df['topic_train'].apply(lambda x:parse_train(x))
which would return a pd.Series object with str versions of the returned pd.Series from parse_train
Can you write Python code for it?

ret = []

def parse_train(t):
# x should be a list with train codes eg 123
# {"id": "123", "type:" "bullet", direction: "south"}

    try:
        s = t['topic_train'].split(',')
    except:
        return t['topic_train']
    if s[0] == '':
#         print ""
        return np.nan
    for x in s:
#         print "Iter",x[1:-1]
        q = {}
        # Check train id
#         x = parse_train_id(x)
        x = str(x)
        x = re.sub('[^0-9]','', x)
        if len(x)<3: continue

        # 1 = north, 0 = south
        q["t_northbound"] = 1 if int(x[2]) in [1,3,5,7,9] else 0
        q['t_limited'] = 0
        q['t_bullet'] = 0

        if x[0] == '1':
            q['t_limited'] = 0
        elif x[0] == '2':
            q["t_limited"] = 1 # limited
        elif x[0] == '3':
            q["t_bullet"] = 1 # bullet
        else:
            q['t_limited'] = 0

        ret.append({'tweet_id': t['id'],
                    'timestamp': t['created_at'],
                    'train_id': int(x),
                    't_northbound':q["t_northbound"],
                    't_limited': q["t_limited"],
                    't_bullet': q['t_bullet']})
    return s

# Let's then filter those train topics into details
# Btw this is jank as fuck.

# red = df[['id','created_at','topic_train']]
red = df.apply(lambda x:parse_train(x),axis=1)
print "red return:",len(red)
print "ret return,",len(ret)
#red
tf = pd.DataFrame(ret)
tf.head(5)

#events = pd.DataFrame([pd.Series(x) for x in red.apply(parse_train)])
#events
#del new.iloc[0]
#new.fillna('')
#df.combine_first(new)

print df.loc[df['topic_train'] != '',['topic_train','text']]

len(tf)

len(tf)

df = df.merge(tf, left_on='id',right_on='tweet_id',how='right')

df.groupby(['time_of_day','month']).mean()

list(df.columns.values)

df.plot(x='time_of_day',y='day_of_week',kind='hist')

# pd.scatter_matrix(df,alpha=0.1,figsize=(15,15), diagonal='hist');

df.groupby('month').describe()

train = df[df['train_id'] >...


I am working on the file "03explore.ipynb".
The first task is:
Let's start getting some more detailed data from the trips as well
Can you write Python code for it?

# df['favorite_count'] = df['favorite_count'].astype(np.int64)
# df['retweet_count'] = df['retweet_count'].astype(np.int64)
# df['text'] = df['text'].astype(str)
# df['id'] = df['id'].astype(np.int64)
# df['day_of_week'] = df['day_of_week'].astype(np.int64)
# df['day_of_month'] = df['day_of_month'].astype(np.int64)
# df['month'] = df['month'].astype(np.int64)
# df['time_of_day'] = df['time_of_day'].astype(np.int64)
df.loc[df["hashtags"]=='on',"hashtags"] = np.nan
df.convert_objects(convert_numeric=True)
df.dtypes
len(df)

# Pull out potential trains from both hashtags and text
df["topic_train"] = df["text"].apply(lambda x: check_train_id(x))
df["topic_train"] = df["topic_train"].apply(lambda x: str(x)[1:-1])
df["topic_train"].fillna(np.nan)
df.head(5)

len(df)

# pd.pivot_table(
#   df,values='values',
#   index=['month'],
#   columns=['day_of_week'])


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
1.4) Balancing the dataset via matching
Can you write Python code for it?

#One dataframe per group
control_group = lalonde_data[lalonde_data['treat'] == 0]
treatment_group = lalonde_data[lalonde_data['treat'] == 1]

#Compute the distance matrix using the absolute difference of the propensity scores
cost_matrix = spatial.distance.cdist(
    treatment_group["propensity score"].values.reshape((treatment_group.shape[0], 1)),
    control_group["propensity score"].values.reshape((control_group.shape[0], 1)),
    metric=lambda a,b: np.abs(a - b)
)

#Solve the distance matrix to minimze the total cost function. Where the total cost function is the sum of the distances
#And get the indices of the pairs that minimze this total cost function
treatment_ind, control_ind = optimize.linear_sum_assignment(cost_matrix)

#We construct a dataframe whith the rows corresponding to the indices obtaiend above. Note we have the same number of sample in each group by construction
lalonde_ps_matched = pd.concat((treatment_group.iloc[treatment_ind], control_group.iloc[control_ind]))


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
1.6) A less naive analysis
Can you write Python code for it?

#We construct a dataframe whith the rows corresponding to the indices obtaiend above. Note we have the same number of sample in each group by construction
lalonde_ps_matched_2 = pd.concat((treatment_group.iloc[treatment_ind_2], control_group.iloc[control_ind_2]))

display_proportions(lalonde_ps_matched_2)


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
2.1) Loading, TF-IDF and Spliting
Data fetching
Can you write Python code for it?

#Loading data
all_news = fetch_20newsgroups(subset='all')
vectorizer = TfidfVectorizer(stop_words='english', max_df=0.5, sublinear_tf=True)

#Vectorizing
news_data = vectorizer.fit_transform(all_news.data)
news_target = all_news.target
news_target_names = all_news.target_names

feature_names = vectorizer.get_feature_names()


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
2.2) Random Forest
Grid search for parameters tuning
Can you write Python code for it?

# use a full grid over max_depth and n_estimators parameters
param_grid = {
    "max_depth": [3, 10, 20, None],
    "n_estimators": np.linspace(3, 200, num=5, dtype=int)
    #"max_features": [1, 3, 10],
    #"min_samples_split": [2, 3, 10],
    #"min_samples_leaf": [1, 3, 10],
    #"bootstrap": [True, False],
    #"criterion": ["gini", "entropy"]
}

# run grid search
grid_search = GridSearchCV(RandomForestClassifier(), param_grid=param_grid)
grid_search.fit(X_validation, y_validation)
None #No output cell


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
A naive analysis would claim that there are no clear differences between the two groups and thus would conclude that the "Job Training Program" (JTP) is useless. And if a difference exists, people in the treatment groups have a smaller revenue by 10%, hence the treatment would be worst than no treatment at all.
1.2) A closer look at the data
Can you write Python code for it?

#Features of each group
main_variables = ['black', 'hispan', 'age', 'married', 'nodegree', 'educ']

#Function that displays a bar plot of each group for every features
def display_proportions(data, variables=main_variables, n_cols=3):
    N = len(variables)
    f, axes = plt.subplots(nrows=int(np.ceil(N / n_cols)), ncols=n_cols)
    f.set_figheight(10)
    for idx, axis, var in zip(range(N), axes.flatten(), variables):
        sns.barplot(x='treat', y=var, data=data, ax=axis)
        axis.set_xticklabels(["Control Group", "Treatment Group"])
        axis.set_xlabel("")
        axis.set_title(idx+1)
        axis.set_ylabel("mean of {}".format(var))

display_proportions(lalonde_data)


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
Data splitting
Can you write Python code for it?

ratios = [0.8, 0.1] #Ratio is 0.8 for train and twice 0.1 for test and validation

X_train, X_test, X_validation, \
y_train, y_test, y_validation = split(news_data, news_target, ratios)


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
Now, lets compare the difference in the distribution for each feature in the two groups as done earlier in part 1.2
Can you write Python code for it?


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
Obervations
1: As we can see on the barplot above, the concentration of black people in the treatment group is 4 times as high as in the control group
2: The concentration of hispanic people in the control group is more than twice as high as in the treatment group
3: Treatment group is on average 2 years younger that control group
4: People in the control group are more than twice as likely to be married than the ones in the treatment group
5: The proportion of people without a degree in the treatment group is higher by 20% than in the control group
6: The mean and the variance of the of years of education is more or less the same in both groups
With these 6 observations, we can say that that two group are not uniformly separated and that for this reason, it is dangerous to draw a conclusion from a superficial analysis.
Let's see whether each group has a similar number of sample:
Can you write Python code for it?

lalond_count = lalonde_data.groupby("treat").agg("count")
print("Number of people in the control group: {}".format(lalond_count["re78"].loc[0]))
print("Number of people in the treatment group: {}".format(lalond_count["re78"].loc[1]))


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
Question 2
Can you write Python code for it?

from sklearn import metrics
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

from time import time


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
Random forest classification
We reuse the optimal parameters computed above to produce prediction with a random forest classifier
Can you write Python code for it?

random_forest_clf = RandomForestClassifier(**best_params)

pred = predict(random_forest_clf, X_train, y_train, X_test)

#Choose the average type
average_type = "weighted"

#Get the different scores of the predicion computed above
accuracy = metrics.accuracy_score(y_test, pred)
precision = metrics.precision_score(y_test, pred, average=average_type)
recall = metrics.recall_score(y_test, pred, average=average_type)
f1_score = metrics.f1_score(y_test, pred, average=average_type)

print("accuracy  = {:.4f}".format(accuracy))
print("precision = {:.4f}".format(precision))
print("recall    = {:.4f}".format(recall))
print("f1_score  = {:.4f}".format(f1_score))


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
The difference in the salaries we perceived in part 1.4 increased, but not significantly.
Based on this difference, we could say that the "Job Training Program" (JTP) is slightly useful and has a positive effect on average on the salary of the people who took the program. We still have a selection bias by having way more black people in the treatment group and hence any conclusion drawn from these data will be biased. Shrinking the number of samples taken in each group so that we only match hispan with hispan and black with black in each group would result in such a small set that it would not be possible to draw any conclusion.
However it is good to point how far we are from the naive analysis realised in point 1. We had that the mean of the treatment group real earnings in 1978 was 10% lower than the one of the control group. However after we refined the way to analyse the data using propensity score and then late one with matching hispan people with hispan people only, we see that the mean of the treatment group real earnings in 1978 is 10% higher than the one of the control group. This example perfectly shows how a naive analyse could show wrong result. Indeed we go from "the treatment is worst" to "The treatment is worth"
Below you can find a barplot summary of the ratio of the means
Can you write Python code for it?

#Plot the means we recorded after each improvement
sns.barplot(y=means_ratio_over_improvement, x = ["Naive", "Propensity score", "Propensity score + hispan matching"])


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
Utility functions
For the following part of the exercise, we created some utility functions that we use here and could be reused for other tasks.
Can you write Python code for it?

# this could have been done in a simpler way for this homework,
# but it might be useful to have such a powerful function for other uses,
# hence we decide to keep it here so that other could use it too :)

def split(X, y, ratios):
    """
    Split X and y given some ratios

    Parameters
    ----------
    X : ndarray
    train matrix

    y : ndarray
    test matrix

    ratios : list(int)
    ratios on how to split X and y

    Returns
    -------
    out : tuple(ndarray)
    Output one tuple of first, the splits of X and then, the splits of y
    """
    assert np.sum(ratios) < 1, "sum of ratios cannot be greater than 1"
    assert len(ratios) >= 1, "at least one ratio required to split"

    def inner_split(X, y, ratios, acc_X, acc_y):
        ratio, *ratios_remaining = ratios
        X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=ratio)

        if len(ratios_remaining) == 0:
            acc_X.extend([X_train, X_test])
            acc_y.extend([y_train, y_test])
            acc_X.extend(acc_y)
            return tuple(acc_X)
        else:
            acc_X.append(X_train)
            acc_y.append(y_train)
            return inner_split(X_test, y_test, [r/(1.0 - ratio) for r in ratios_remaining], acc_X, acc_y)

    return inner_split(X, y, ratios, [], [])

def predict(clf, X_train, y_train, X_test):
    """
    Using a classifier, train with training data it using to fit testing labels
    and then predict some labels of testing data.

    It also times the different steps.

    Parameters
    ----------
    clf: sklearn classifier
    classifier

    X_train: ndarray
    training data

    y_train: ndarray
    training labels

    X_test: ndarray
    testing data

    Returns
    -------
    out : ndarray
    Output the prediction of labels
    """
    start_time = time()
    print("Prediction computations started...")
    clf.fit(X_train, y_train)
    train_time = time() - start_time
    pred = clf.predict(X_test)
    prediction_time =...


I am working on the file "04 - Applied ML/homework04.ipynb".
The first task is:
We can now see that the mean in the treatment group is slightly higher than in the control group, where it was slightly below before. Also the maximum, median and quartiles are all bigger than their counterpart in the control group. This is a complete different information from what we had before, but let's improve it even more.
1.5) Balancing the groups further
The main difference in the two groups resides in the proportion of hispanic and black people:
For this reason, we will add the condition when matching two subjects that they have the same value for the hispanic feature. Doing so for the black feature is not possible because 156 people out of the 185 people are black in the treatment group where for the control group there are 87 black people out of the 429 people.
Can you write Python code for it?

additionnal_feature_matched = 'hispan'

#Compute the distance matrix where a value is 0 if both the row and the colum is hispan, 1 otherwise
add_cost_matrix = spatial.distance.cdist(
    treatment_group[additionnal_feature_matched].values.reshape((treatment_group.shape[0], 1)),
    control_group[additionnal_feature_matched].values.reshape((control_group.shape[0], 1)),
    metric=lambda a,b: int(a != b)
)

#Solve the distance matrix (obtained by adding the propensity score distance matrix to the hispan distance matrix) to minimze the total cost function.
#Where the total cost function is the sum of the distances
#And get the indices of the pairs that minimze this total cost function
treatment_ind_2, control_ind_2 = optimize.linear_sum_assignment(cost_matrix + add_cost_matrix)


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Arrays as Python objects
Python can be used as an object oriented language, and numpy arrays have lots of properties. There are many functions we can use as numpy.<function>(<array>) and array.<function>
For example, the transpose above:
Can you write Python code for it?

print("The original array is: ", test_array)
print("The transposed array is: ", np.transpose(test_array) )

# Alternatively, using test_array as an opject:
print("The transposed array is: ", test_array.transpose() )


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Building arrays from other arrays (appending)
We can build arrays from other array via Python stacking in a horizontal or vertical way:
Can you write Python code for it?

test_array_Vstacked = np.vstack((test_array, [1,2,3,4]))
print(test_array_Vstacked)

test_array_Hstacked = np.hstack((test_array, test_array))
print(test_array_Hstacked)


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Caution: appending to numpy arrays frequently is memory intensive. Every time this happens, an entirely new chunk of memory needs to be used, so the old array is moved in memory to a new location.
It's faster to 'preallocate' an array with empty values, and simply populate as the computation progresses.
Operations on arrays of different sizes (broadcasting)
Python automatically handles arithmetic operations with arrays of different dimensions. In other words, when arrays have different (but compatible) shapes, the smaller is 'broadcast' across the larger.
Can you write Python code for it?

test_array

print("The broadcasted array is: ", test_array[0,:])
test_array[0,:] * test_array


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Fancy ways of indexing
Slicing Arrays:
Slicing arrays refers to indexing >1 elements in a previous array. Slicing is often used when parallelizing computations using arrays. Indexing is array[row, column].
Can you write Python code for it?

test_array[:,1]     # Use the ':' to index along one dimension fully

test_array[1,1:]    # Adding a colon indexes the rest of the values
                    #    (includes the numbered index)

test_array[1,1:-1]  # We can index relative to the first and last elements

test_array[1,::2]   # We can specify the indexing order

test_array[1,1::-1] # We can get pretty fancy about it
                    # Index second row, second from first to second from
                    #     last in reverse order.


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
However, if the dimensions don't match, it won't work:
Can you write Python code for it?

print("The broadcasted array is: ", test_array[:,0])
#test_array[:,0] * test_array # Uncomment the line to see that the
                              #     dimensions don't match

# Make use of the matrix transpose (also can use array.T)
np.transpose( test_array[:,0]*np.transpose(test_array) )


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
If statement says "10 loops, best of 3: [time]" it means the fastest of 10 repeated runs was recorded - then the 10 runs were repeated twice more, resulting in an overall fastest time.
Creating and accessing (indexing) arrays
We can create arrays from scratch:
Can you write Python code for it?

test_array = np.array([[1,2,3,4], [6,7,8,9]])
print(test_array)


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Index arrays using square brackets, starting from zero and specifying row, column:
Can you write Python code for it?


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
One of the most frequenly used properties of arrays is the dimension:
Can you write Python code for it?

print("The original array dimensions are: ", test_array.shape)
print("The array transpose dimensions are: ", test_array.transpose().shape)


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Sorting:
Sorting arrays happens in-place, so once the function is called on an array, the sorting happens to the original array:
Can you write Python code for it?

test_array2 = np.array([1,5,4,0,1])
print("The original array is: ", test_array2)

test_array3 = test_array2.sort() # Run the sort - note that the new variable isn't assigned
print("The reassigned array should be sorted: ", test_array3)
print("test_array2 after sort: ", test_array2)


I am working on the file "04 - Arrays - Numpy.ipynb".
The first task is:
Using the isnan function in numpy:
Can you write Python code for it?

nans = np.isnan(test_array)
print(nans)

test_array[nans] = 4
print(test_array)


I am working on the file "04 - Classification Models.ipynb".
The first task is:
Linear Methods
Can you write Python code for it?

from sklearn.svm import SVC

model = SVC()
model.fit(X_train, y_train)
yhat = model.predict(X_test)

print(classification_report(yhat, y_test))

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

model = LinearDiscriminantAnalysis()
model.fit(X_train, y_train)
yhat = model.predict(X_test)

print(classification_report(yhat, y_test))


I am working on the file "04 - Classification Models.ipynb".
The first task is:
Probabalistic Methods
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB

model = GaussianNB()
model.fit(X_train, y_train)
yhat = model.predict(X_test)

print(classification_report(yhat, y_test))

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)
yhat = model.predict(X_test)

print(classification_report(yhat, y_test))


I am working on the file "04 - Data 2.ipynb".
The first task is:
Each marker has details associated with it.  Those details are called a "listing".  The same Pandas loading method is used for the listings data.
Can you write Python code for it?

listings = pd.io.json.read_json('./listings.json')

listings[:5]


I am working on the file "04 - Data 2.ipynb".
The first task is:
Importing into numpy
When working with numerical data, I tend to use numpy.  Converting from a Pandas data frame to a numpy array is easy.
Can you write Python code for it?


I am working on the file "04 - Data 2.ipynb".
The first task is:
Parsing data using Pandas
Pandas has a built in module for loading json files directly.  Here it's used to load the markers data.
Can you write Python code for it?

import pandas as pd
markers = pd.io.json.read_json('./markers.json')

markers[:5]


I am working on the file "04 - Data 2.ipynb".
The first task is:
Plot the markers above the map image.
Can you write Python code for it?

plt.imshow(cropped_map)
plt.scatter(array[:, 1], array[:, 0], s=4., c='r', linewidths=0)
plt.show()


I am working on the file "04 - Data 2.ipynb".
The first task is:
Ploting with Matplotlib
Import matplotlib and set the necessary config options.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'


I am working on the file "04 - Data 2.ipynb".
The first task is:
Since the map image size and the longitude and latitude window is known, the longitudes and latitudes can be converted to pixel coordinates, which is necessary for plotting on top of the map image.
Can you write Python code for it?

long2px = lambda x: (x - coords['westLong']) / (coords['eastLong'] - coords['westLong']) * image_size[0]
lat2px = lambda y: (1 - (y - coords['southLat']) / (coords['northLat'] - coords['southLat'])) * image_size[1]
array[:, 0] = lat2px(array[:, 0])
array[:, 1] = long2px(array[:, 1])
array


I am working on the file "04 - Grid Searches for Hyper Parameters.ipynb".
The first task is:
A GridSearchCV object behaves just like a normal classifier.
Can you write Python code for it?

grid_search.fit(X_train, y_train)

grid_search.predict(X_test)

grid_search.score(X_test, y_test)

grid_search.best_params_

# We extract just the scores
%matplotlib notebook
import matplotlib.pyplot as plt

scores = [x[1] for x in grid_search.grid_scores_]
scores = np.array(scores).reshape(6, 5)

plt.matshow(scores)
plt.xlabel('gamma')
plt.ylabel('C')
plt.colorbar()
plt.xticks(np.arange(5), param_grid['gamma'])
plt.yticks(np.arange(6), param_grid['C']);


I am working on the file "04 - Grid Searches for Hyper Parameters.ipynb".
The first task is:
Define parameter grid:
Can you write Python code for it?

import numpy as np

param_grid = {'C': 10. ** np.arange(-3, 3),
              'gamma' : 10. ** np.arange(-5, 0)}

np.set_printoptions(suppress=True)
print(param_grid)

grid_search = GridSearchCV(SVC(), param_grid, verbose=3, cv=5)


I am working on the file "04 - Grid Searches for Hyper Parameters.ipynb".
The first task is:
Exercises
Use GridSearchCV to adjust n_neighbors of KNeighborsClassifier.
Can you write Python code for it?


I am working on the file "04 - Groupby operations.ipynb".
The first task is:

    EXERCISE: Using groupby(), plot the number of films that have been released each **decade** in the history of cinema.

    EXERCISE: Use groupby() to plot the number of "Hamlet" films made each decade.

Multiple groups
Note that you can also groupby on multiple keys:
Can you write Python code for it?

df['type'] = np.where(df['data'] % 2, 'odd', 'even')
print(df)

df.groupby(['type', 'key']).sum()

df['type'] = np.where(df['data'] % 2, 'odd', 'even')
print(df)
df['data']


I am working on the file "04 - Groupby operations.ipynb".
The first task is:

    EXERCISE: Using groupby(), plot the number of films that have been released each decade in the history of cinema.

    EXERCISE: Use groupby() to plot the number of "Hamlet" films made each decade.

    EXERCISE: How many leading (n=1) roles were available to actors, and how many to actresses, in each year of the 1950s?

    EXERCISE: Use groupby() to determine how many roles are listed for each of The Pink Panther movies.

    EXERCISE: List, in order by year, each of the films in which Frank Oz has played more than 1 role.

    EXERCISE: List each of the characters that Frank Oz has portrayed at least twice.

Transforms
Sometimes you don't want to aggregate the groups, but transform the values in each group. This can be achieved with transform:£
Can you write Python code for it?

df

def normalize(group):
    return (group - group.mean()) / group.std()

df.groupby('key').transform(normalize)


I am working on the file "04 - Groupby operations.ipynb".
The first task is:

    EXERCISE: Using groupby(), plot the number of films that have been released each year in the history of cinema.

    EXERCISE: Use groupby() to determine how many roles are listed for each of The Pink Panther movies.

Custom  grouping criteria
You can also group by the values on another array under the condition that this array has the length equal to the number of rows:
Can you write Python code for it?


I am working on the file "04 - Groupby operations.ipynb".
The first task is:

    EXERCISE: What are the 11 most common character names in movie history?

Custom aggregate functions
Aggregate function could be any function accepting the Series object.
For example, let's calculate most frequent apperances in each year of last decade:
Can you write Python code for it?

def most_frequent(x):
    return x.value_counts().index[0]

cast.loc[(cast['year'] >= 2010) & (cast['year'] < 2020), ['year', 'name']].groupby('year').agg(most_frequent)


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
And many more methods are available.
And now applying this on some real data
We go back to the titanic survival data:
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.head()


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
And now applying this on some real data
These exercises are based on the PyCon tutorial of Brandon Rhodes (so all credit to him!) and the datasets he prepared for that. You can download these data from here: titles.csv and cast.csv and put them in the /data folder.
Can you write Python code for it?

cast = pd.read_csv('data/cast.csv')
cast.head()

titles = pd.read_csv('data/titles.csv')
titles.head()


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
Automatic alignment on labels
Can you write Python code for it?

data = {'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']}
countries = pd.DataFrame(data).set_index('country')
countries


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
Movie database
These exercises are based on the PyCon tutorial of Brandon Rhodes (so all credit to him!) and the datasets he prepared for that. You can download these data from here: titles.csv and cast.csv and put them in the /data folder.
Can you write Python code for it?

cast = pd.read_csv('data/cast.csv')
cast[10:15]

titles = pd.read_csv('data/titles.csv')
titles.head()


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
Note that it creates a hierarchical index. More on that later.

    EXERCISE: List each of the characters that Frank Oz has portrayed at least twice.

    EXERCISE: List, in order by year, each of the films in which Frank Oz has played more than 1 role.

    EXERCISE: How many leading (n=1) roles were available to actors, and how many to actresses, in each year of the 1950s?

Value counts
A useful shortcut to calculate the number of occurences of certain values is value_counts (this is somewhat equivalent to df.groupby(key).size()))
For example, what are the most occuring movie titles?
Can you write Python code for it?


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
Now we calculate the natural increae by subtracting death rate from birth rate:
Can you write Python code for it?

natural_increase = birth_rate - death_rate
print(natural_increase)


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
Some 'theory': the groupby operation (split-apply-combine)
The "group by" concept: we want to apply the same function on subsets of your dataframe, based on some key to split the dataframe in subsets
This operation is also referred to as the "split-apply-combine" operation, involving the following steps:

Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure

<img src="img/splitApplyCombine.png">
Similar to SQL GROUP BY
The example of the image in pandas syntax:
Can you write Python code for it?


I am working on the file "04 - Groupby operations.ipynb".
The first task is:
You can also simply count members of each split:
Can you write Python code for it?


I am working on the file "04 - interactions.ipynb".
The first task is:
Bokeh also provides the vplot and hplot functions to arrange plot objects in vertical or horizontal layouts.
Can you write Python code for it?

# EXERCISE: use vplot to arrange a few plots vertically


I am working on the file "04 - interactions.ipynb".
The first task is:
Callbacks
Can you write Python code for it?

from bokeh.models import TapTool, CustomJS, ColumnDataSource

callback = CustomJS(code="alert('hello world')")
tap = TapTool(callback=callback)

p = figure(plot_width=600, plot_height=300, tools=[tap])

p.circle('x', 'y', size=20, source=ColumnDataSource(data=dict(x=[1, 2, 3, 4, 5], y=[2, 5, 8, 2, 7])))

show(p)


I am working on the file "04 - interactions.ipynb".
The first task is:
Hover Tools
Bokeh has a Hover Tool that allows additional information to be displayed in a popup whenever the uer howevers over a specific glyph. Basic hover tool configuration amounts to providing a list of (name, format) tuples. The full details can be found in the User's Guide here.
The example below shows some basic usage of the Hover tool with a circle glyph:
Can you write Python code for it?

from bokeh.models import HoverTool

source = ColumnDataSource(
        data=dict(
            x=[1, 2, 3, 4, 5],
            y=[2, 5, 8, 2, 7],
            desc=['A', 'b', 'C', 'd', 'E'],
        )
    )

hover = HoverTool(
        tooltips=[
            ("index", "$index"),
            ("(x,y)", "($x, $y)"),
            ("desc", "@desc"),
        ]
    )

p = figure(plot_width=300, plot_height=300, tools=[hover], title="Mouse over the dots")

p.circle('x', 'y', size=20, source=source)

# Also show custom hover
from utils import get_custom_hover

show(gridplot([[p, get_custom_hover()]]))


I am working on the file "04 - interactions.ipynb".
The first task is:
Ipython Interactors
It is possible to use native IPython notebook interactors together with Bokeh. In the interactor update function, the push_notebook method can be used to update a data source (presumably based on the iteractor widget values) to cause a plot to update.
Warning: The current implementation of push_notebook leaks memory. It is suitable for interactive exploration but not for long-running or streaming use cases. The problem will be resolved in future releases.
The example below shows a "trig function" exporer using IPython interactors:
Can you write Python code for it?

import numpy as np
from bokeh.models import Line

x = np.linspace(0, 2*np.pi, 2000)
y = np.sin(x)

source = ColumnDataSource(data=dict(x=x, y=y))

p = figure(title="simple line example", plot_height=300, plot_width=600, y_range=(-5, 5))
p.line(x, y, color="#2222aa", alpha=0.5, line_width=2, source=source, name="foo")

def update(f, w=1, A=1, phi=0):
    if   f == "sin": func = np.sin
    elif f == "cos": func = np.cos
    elif f == "tan": func = np.tan
    source.data['y'] = A * func(w * x + phi)
    source.push_notebook()

show(p)

from ipywidgets import interact
interact(update, f=["sin", "cos", "tan"], w=(0,10, 0.1), A=(0,5, 0.1), phi=(0, 10, 0.1))


I am working on the file "04 - interactions.ipynb".
The first task is:
Linked Interactions
It is possible to link various interactions between different Bokeh plots. For instance, the ranges of two (or more) plots can be linked, so that when one of the plots is panned (or zoomed, or otherwise has its range changed) the other plots will update in unison. It is also possible to link selections between two plots, so that when items are selected on one plot, the corresponding items on the second plot also become selected.
Linked panning
Linked panning (when mulitple plots have ranges that stay in sync) is simple to spell with Bokeh. You simply share the approrpate range objects between two (or more) plots. The example below shows how to accomplish this by linking the ranges of three plots in various ways:
Can you write Python code for it?

plot_options = dict(width=250, plot_height=250, title=None, tools='pan')

# create a new plot
s1 = figure(**plot_options)
s1.circle(x, y0, size=10, color="navy")

# create a new plot and share both ranges
s2 = figure(x_range=s1.x_range, y_range=s1.y_range, **plot_options)
s2.triangle(x, y1, size=10, color="firebrick")

# create a new plot and share only one range
s3 = figure(x_range=s1.x_range, **plot_options)
s3.square(x, y2, size=10, color="olive")

p = gridplot([[s1, s2, s3]])

# show the results
show(p)

# EXERCISE: create two plots in a gridplot, and link their ranges


I am working on the file "04 - interactions.ipynb".
The first task is:
Linked brushing
Linking selections is accomplished in a similar way, by sharing data sources between plots. Note that normally with bokeh.plotting and bokeh.charts creating a default data source for simple plots is handled automatically. However to share a data source, we must create them by hand and pass them explicitly. This is illustrated in the example below:
Can you write Python code for it?

from bokeh.models import ColumnDataSource

x = list(range(-20, 21))
y0, y1 = [abs(xx) for xx in x], [xx**2 for xx in x]

# create a column data source for the plots to share
source = ColumnDataSource(data=dict(x=x, y0=y0, y1=y1))

TOOLS = "box_select,lasso_select,help"

# create a new plot and add a renderer
left = figure(tools=TOOLS, width=300, height=300)
left.circle('x', 'y0', source=source)

# create another new plot and add a renderer
right = figure(tools=TOOLS, width=300, height=300)
right.circle('x', 'y1', source=source)

p = gridplot([[left, right]])

show(p)

# EXERCISE: create two plots in a gridplot, and link their data sources


I am working on the file "04 - interactions.ipynb".
The first task is:
Widgets
Bokeh supports direct integration with a small basic widget set. Thse can be used in conjunction with a Bokeh Server, or with CustomJS models to add more interactive capability to your documents. You can see a complete list, with example code in the Adding Widgets section of the User's Guide.
To use the widgets, include them in a layout like you would a plot object:
Can you write Python code for it?

from bokeh.models.widgets import Slider
from bokeh.io import vform

slider = Slider(start=0, end=10, value=1, step=.1, title="foo")

show(vform(slider))

# EXERCISE: create and show a Select widget


I am working on the file "04 If Statements.ipynb".
The first task is:
Exercises
True and False

Write a program that consists of at least ten lines, each of which has a logical statement on it. The output of your program should be 5 Trues and 5 Falses.
Note: You will probably need to write print(5 > 3), not just 5 > 3.
Can you write Python code for it?

# Ex 5.1 : True and False

# put your code here


I am working on the file "04 If Statements.ipynb".
The first task is:
As you can see, the if-elif-else chain lets you respond in very specific ways to any given situation.
Exercises
Three is a Crowd

Make a list of names that includes at least four people.
Write an if test that prints a message about the room being crowded, if there are more than three people in your list.
Modify your list so that there are only two people in it. Use one of the methods for removing people from the list, don't just redefine the list.
Run your if test again. There should be no output this time, because there are less than three people in the list.
Bonus: Store your if test in a function called something like crowd_test.

Three is a Crowd - Part 2

Save your program from Three is a Crowd under a new name.
Add an else statement to your if tests. If the else statement is run, have it print a message that the room is not very crowded.

Six is a Mob

Save your program from Three is a Crowd - Part 2 under a new name.
Add some names to your list, so that there are at least six people in the list.
Modify your tests so that
If there are more than 5 people, a message is printed about there being a mob in the room.
If there are 3-5 people, a message is printed about the room being crowded.
If there are 1 or 2 people, a message is printed about the room not being crowded.
If there are no people in the room, a message is printed abou the room being empty.
Can you write Python code for it?

# Ex 5.2 : Three is a Crowd

# put your code here

# Ex 5.3 : Three is a Crowd - Part 2

# put your code here

# Ex 5.4 : Six is a Mob

# put your code here


I am working on the file "04 If Statements.ipynb".
The first task is:
If we had done this using an if-elif-else chain, only the first dog that is present would be greeted:
Can you write Python code for it?

###highlight=[6,7,8,9,10,11]
dogs = ['willie', 'hootz']

if 'willie' in dogs:
    print("Hello, Willie!")
elif 'hootz' in dogs:
    print("Hello, Hootz!")
elif 'peso' in dogs:
    print("Hello, Peso!")
elif 'monty' in dogs:
    print("Hello, Monty!")


I am working on the file "04 If Statements.ipynb".
The first task is:
In this case, we only print a message if there is at least one dog present. Of course, you could add a new else clause to respond to the situation in which there are no dogs at all:
Can you write Python code for it?

###highlight=[10,11]
dogs = []

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
elif len(dogs) >= 1:
    print("Okay, this is a reasonable number of dogs.")
else:
    print("I wish we had a dog here.")


I am working on the file "04 If Statements.ipynb".
The first task is:
In this situation, nothing happens if the test does not pass.
Can you write Python code for it?

dogs = ['willie', 'hootz']

if len(dogs) > 3:
    print("Wow, we have a lot of dogs here!")


I am working on the file "04 If Statements.ipynb".
The first task is:
In this situation, the first two tests fail, so the statement in the else clause is executed. Note that this statement would be executed even if there are no dogs at all:
Can you write Python code for it?

dogs = []

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file "04 If Statements.ipynb".
The first task is:
It is important to note that in situations like this, only the first test is evaluated. In an if-elif-else chain, once a test passes the rest of the conditions are ignored.
Can you write Python code for it?

dogs = ['willie', 'hootz', 'peso', 'monty']

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file "04 If Statements.ipynb".
The first task is:
Of course, this could be written much more cleanly using lists and for loops. See if you can follow this code.
Can you write Python code for it?

dogs_we_know = ['willie', 'hootz', 'peso', 'monty', 'juno', 'turkey']
dogs_present = ['willie', 'hootz']

# Go through all the dogs that are present, and greet the dogs we know.
for dog in dogs_present:
    if dog in dogs_we_know:
        print("Hello, %s!" % dog.title())


I am working on the file "04 If Statements.ipynb".
The first task is:
Our results have not changed in this case, because if the test evaluates to True only the statements under the if statement are executed. The statements under else area only executed if the test fails:
Can you write Python code for it?

dogs = ['willie', 'hootz']

if len(dogs) > 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file "04 If Statements.ipynb".
The first task is:
The test evaluated to False, so only the statement under else is run.
if-elif...else chains
Many times, you will want to test a series of conditions, rather than just an either-or situation. You can do this with a series of if-elif-else statements
There is no limit to how many conditions you can test. You always need one if statement to start the chain, and you can never have more than one else statement. But you can have as many elif statements as you want.
Can you write Python code for it?

dogs = ['willie', 'hootz', 'peso', 'monty', 'juno', 'turkey']

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file "04 If Statements.ipynb".
The first task is:
This is the kind of code you should be aiming to write. It is fine to come up with code that is less efficient at first. When you notice yourself writing the same kind of code repeatedly in one program, look to see if you can use a loop or a function to make your code more efficient.
ADD LOGICAL OPERATORS
top
True and False values
Every value can be evaluated as True or False. The general rule is that any non-zero or non-empty value will evaluate to True. If you are ever unsure, you can open a Python terminal and write two lines to find out if the value you are considering is True or False.
Take a look at the following examples, keep them in mind, and test any value you are curious about.
I am using a slightly longer test just to make sure something gets printed each time.
Can you write Python code for it?

if 0:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

if 1:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

# Arbitrary non-zero numbers evaluate to True.
if 1253756:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

# Negative numbers are not zero, so they evaluate to True.
if -1:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

# An empty string evaluates to False.
if '':
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

# Any other string, including a space, evaluates to True.
if ' ':
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

# Any other string, including a space, evaluates to True.
if 'hello':
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

# None is a special object in Python. It evaluates to False.
if None:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")


I am working on the file "04 If Statements.ipynb".
The first task is:
greater than or equal to
Can you write Python code for it?


I am working on the file "04 If Statements.ipynb".
The first task is:
less than
Can you write Python code for it?


I am working on the file "04 If Statements.ipynb".
The first task is:
top
The if-elif...else chain
You can test whatever series of conditions you want to, and you can test your conditions in any combination you want.
Simple if statements
The simplest test has a single if statement, and a single statement to execute if the condition is True.
Can you write Python code for it?

dogs = ['willie', 'hootz', 'peso', 'juno']

if len(dogs) > 3:
    print("Wow, we have a lot of dogs here!")


I am working on the file "04 If Statements.ipynb".
The first task is:
top
Checking if an item is in a list
You can check if an item is in a list using the in keyword.
Can you write Python code for it?

vowels = ['a', 'e', 'i', 'o', 'u']
'a' in vowels

vowels = ['a', 'e', 'i', 'o', 'u']
'b' in vowels


I am working on the file "04 If Statements.ipynb".
The first task is:
top
Inequality
Two items are inequal if they do not have the same value. In Python, we test for inequality using the exclamation point and one equals sign.
Sometimes you want to test for equality and if that fails, assume inequality. Sometimes it makes more sense to test for inequality directly.
Can you write Python code for it?


I am working on the file "04 If Statements.ipynb".
The first task is:
top
More than one passing test
In all of the examples we have seen so far, only one test can pass. As soon as the first test passes, the rest of the tests are ignored. This is really good, because it allows our code to run more efficiently. Many times only one condition can be true, so testing every condition after one passes would be meaningless.
There are situations in which you want to run a series of tests, where every single test runs. These are situations where any or all of the tests could pass, and you want to respond to each passing test. Consider the following example, where we want to greet each dog that is present:
Can you write Python code for it?

dogs = ['willie', 'hootz']

if 'willie' in dogs:
    print("Hello, Willie!")
if 'hootz' in dogs:
    print("Hello, Hootz!")
if 'peso' in dogs:
    print("Hello, Peso!")
if 'monty' in dogs:
    print("Hello, Monty!")


I am working on the file "04 If Statements.ipynb".
The first task is:
top
Other Inequalities
greater than
Can you write Python code for it?


I am working on the file "04 If Statements.ipynb".
The first task is:
top
Overall Challenges
Alien Points

Make a list of ten aliens, each of which is one color: 'red', 'green', or 'blue'.
You can shorten this to 'r', 'g', and 'b' if you want, but if you choose this option you have to include a comment explaining what r, g, and b stand for.

Red aliens are worth 5 points, green aliens are worth 10 points, and blue aliens are worth 20 points.
Use a for loop to determine the number of points a player would earn for destroying all of the aliens in your list.
hint
Can you write Python code for it?

# Overall Challenge: Alien Points

# Put your code here


I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
Challenge Exercise
Can you compare the following algorithms to see which one performs best?

RandomForestRegressor
GradientBoostingRegressor
AdaBoostRegressor
ExtraTreesRegressor

Statistical Practices
Here, I will show you how to use the ShuffleSplit iterator, alongside the cross_val_score method, to evaluate the performance of different regressor models.
The methods here aren't limited to regression, though. You can easily implement these for classification as well.
Can you write Python code for it?

cv = ShuffleSplit(n=len(X), n_iter=10, test_size=0.3)

models = dict()
models['rf'] = RandomForestRegressor()
models['gb'] = GradientBoostingRegressor()
models['ad'] = AdaBoostRegressor()
models['ex'] = ExtraTreesRegressor()

scores = dict()

for abbr, model in models.items():
    print(abbr, model)
    score = cross_val_score(model, X, Y, cv=cv, scoring='mean_squared_error')
    scores[abbr] = -score  # a known issue in the scikit-learn package; we have to take the negative of the result.

score_summary = pd.DataFrame(scores)
score_summary = pd.DataFrame(score_summary.unstack()).reset_index()
# sns.violinplot(x=)
score_summary.columns = ['model', 'idx', 'error']
sns.violinplot(x='model', y='error', data=score_summary)


I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
Discussion
What does the distribution of values look like? Where is its skew? How could we tell?
What would be a better way of transforming the data prior to doing ML?
Live-Coding
Let's try log10-transforming the values to be predicted.
Can you write Python code for it?

X = data_matrix[[i for i in range(99)]]
Y = data_matrix[drug_name].apply(np.log10)  # the log10 transformation is applied here.

X_train, X_test, Y_train, Y_test = train_test_split(X, Y)

mdl = RandomForestRegressor()
mdl.fit(X_train, Y_train)
preds = mdl.predict(X_test)


I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
Evaluating the Model
Just as with classification tasks, we also need metrics to help evaluate how good a trained model is, given the input features.
Exercise
Look through the sklearn.metrics module. What might be a suitable metric to use?
Justify the use of two of them, and write the code that computes the evaluation metric.
Can you write Python code for it?

# Metric 1: correlation coefficient.

r2_score(preds, Y_test)

# Metric 2: mean squared error

mean_squared_error(preds, Y_test)


I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
Make a plot of what the predictions (y-axis) against the actual values (x-axis)
Can you write Python code for it?


I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
Note: the MSE goes down because of the log10 transform. However, that's exactly what we would have expected by definition.
Can you write Python code for it?


I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
We will train a model to predict drug resistance values from sequence.
This is the other general variant of supervised learning - where instead of predicting a "label" for a class (classification), we are predicting a "number" or a "value". This is called regression, analogous to, say, linear regression or logistic regression.
Can you write Python code for it?

# Load the sequence data as a Pandas dataframe.

seqids = [s.id for s in SeqIO.parse('data/hiv-protease-sequences-expanded.fasta', 'fasta')]

sequences = [s for s in SeqIO.parse('data/hiv-protease-sequences-expanded.fasta', 'fasta')]
sequences = MultipleSeqAlignment(sequences)
sequences = pd.DataFrame(np.array(sequences))
sequences.index = seqids
# Ensure that all of the letters are upper-case, otherwise the replace function in the next cell won't work.
for col in sequences.columns:
    sequences[col] = sequences[col].apply(lambda x: x.upper())
    sequences[col] = sequences[col].replace('*', np.nan)
sequences.head()

seqdf = sequences.replace(isoelectric_points.keys(), isoelectric_points.values())
seqdf.head()

# Load the drug resistance values
dr_vals = pd.read_csv('data/hiv-protease-data-expanded.csv', index_col=0)
dr_vals.set_index('seqid', inplace=True)
dr_vals.head()

# Join the sequence data together with that of one drug of interest.
drug_name = 'FPV'

data_matrix = seqdf.join(dr_vals[drug_name]).dropna()  # we have to drop NaN values because scikit-learn algorithms are not designed to accept them.
data_matrix.head()


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Dieser Wulst lässt sich aber ziemlich schwer lesen. Deshalb müssen wir das zuerst umformatieren und dann verstehen, wie das ganze strukturiert ist.
Can you write Python code for it?

response.json()#json strukturiert die daten. Ich sehe es ist ein dictionyry von zwei listen, die wiederum dictionaries enthält mit keys  und values - wo beginnt sie; es nur zwei keys: bbox und features. features ist interessanter

dct = response.json()


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Erste Erfahrung mit Pandas
Can you write Python code for it?

pd.DataFrame(earthquakes)#es muss aber eine dictionary-Liste sein; kann auch nur eine Liste sein, dann gibt es nur eine Spalte

df = pd.DataFrame(earthquakes)

df.sort_values(by='Mag', ascending=False)#startet von 'Mag'; default Einstellung: beginnt beim kleinsten, wenn ich beim stärksten beginnen will, muss ich ascending=false eingeben


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Features interessiert uns.
Can you write Python code for it?

type(dct['features'][0])#type zeigt mir den Datentyp; mit (0) sage ich, dass er mir erstes der Features

len(dct['features'])


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Grosse, komibinierte Abfrage
Details zu allen Erbeben in den über drei Tage, 17-9-2017 bis 21-9-2017.
Can you write Python code for it?

url1 = 'https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson'#query heisst wir suchen bestimmte erdbeben

urlzeit = '&starttime=2017-09-17T24:00:00&endtime=2017-09-21T08:00:00'

response = requests.get(url1+urlzeit)#Ich hänge die beiden Variablen zusammen
response.text#Alle Erdbeben auf der ganzen Welt in diesem Zeitraum


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Lesen wir folgendes aus:
1. Type
2. Magnitude
4. Örtlichkeit
5. Time
Can you write Python code for it?

earthquakes = []#Leere Liste kreieren

for elem in dct['features']:
    t = elem['properties']['type']#in den properties (die ich ja vorhin als element gefunden habe) will ich
    m = elem['properties']['mag']
    p = elem['properties']['place']
    tm = elem['properties']['time']

    mini_dict = {'Type': t,
                 'Mag': m,
                 'Place': p,
                 'Time': tm}#Hier baue ich mein eigenes Dictionary auf; die Begriffe kann ich selbst wählen

    earthquakes.append(mini_dict)


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Schauen wir uns alle keys und values an
Can you write Python code for it?

for key, value in dct.items() :# mit .items angehängt an dictionnary-name druckt value und keys aus
    print (key, value)


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Speichern wir das ab.
Can you write Python code for it?


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Um sicher zu gehen, prüfen wir nochmals den Datentyp.
Can you write Python code for it?


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Unsere Imports
Can you write Python code for it?

import requests
import pandas as pd
import calendar, datetime, time


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Wieviele hat es letzte Nacht von 22 bis 6 Uhr morgens gegeben? Dafür müssen wir in der Dokumentation Time nachfragen.
Can you write Python code for it?

url = 'https://earthquake.usgs.gov/fdsnws/event/1/count?starttime=2017-09-20T22:00:00&endtime=2017-09-21T06:00:00'#Funktion bis ? Dann Parameter mit & verbunden


I am working on the file "04 modules, requests/01 APIs und Pyhton Modules, with a Focus on Requests.ipynb".
The first task is:
Zur Dokumentation von requestst geht es hier lang.
Notes
In einer ersten Übung arbeiten wir mit der API des USGS arbeiten. Für die meisten API muss man sich heutezutage eine eigene API-Key kreieren. Das werden wir zu einem späteren Zeitpunkt tun. Diese API braucht keine Keys. Deshalb ist es ein gutes Kennenlernbeispiel.
Falls Probleme mit dem Output auftauchen, hier ein Work around für die Meldung, dass der Output limitiert wurde. Reconfigure line output and opening stuff from the command line
Dokumentation
Bevor man mit deiner API arbeitet, lohnt es sich immer zuerst, die Dokumentation anzuschauen.
Erste Abfragen
Kleinere Abfrage. Wieviele Erdbeben hat es in einem bestimmten Zeitraum gegeben?#APIs die genutzt werden wollen, dokumentieren ihren Aufbau auf der Website - zB ws-old.parlament
Can you write Python code for it?

url = 'https://earthquake.usgs.gov/fdsnws/event/1/count?starttime=2017-09-20&endtime=2017-09-21'

response = requests.get(url)

response

response.text


I am working on the file "04 modules, requests/02 module - homework.ipynb".
The first task is:
2. Lese eine dieser APIs mit requests ein und rufe die Ergebnisse auf.
Can you write Python code for it?

url1 = 'http://api.openhazards.com/GetEarthquakeProbability?q=San+Francisco,+CA&m=6.8&r=100'

response = requests.get(url1)

response

response.text

response.json


I am working on the file "04 modules, requests/02 module - homework.ipynb".
The first task is:
3. Zurück zum Erdbeben API. Wie viele Erdbeben wurden im August 2017  verzeichnet?
Can you write Python code for it?

url_august = 'https://earthquake.usgs.gov/fdsnws/event/1/count?starttime=2017-08-01&endtime=2017-08-31'

response = requests.get(url_august)

response


I am working on the file "04 modules, requests/02 module - homework.ipynb".
The first task is:
4. Und wie viele Erdbeben im Juli?
Can you write Python code for it?

url_juli = 'https://earthquake.usgs.gov/fdsnws/event/1/count?starttime=2017-07-01&endtime=2017-07-31'

response

response.text


I am working on the file "04--comps-gens-sparse.ipynb".
The first task is:
Default dictionaries: collections.defaultdict
The next new concept you will explore is a twist on the usual dictionary object, which is a default dictionary.
To motivate it, consider the following common pattern.
Suppose you are given a string and you wish to count the number of occurrences of alphabetic characters. A first and natural way might be to scan the string letter by letter, using a dictionary to store the counts. This approach would then involve a common idiom: for each letter, check whether we created a dictionary entry already; if so, then update the entry; otherwise, create a new entry.

Inspect this example and try it.
Can you write Python code for it?

def alpha_chars (text):
    """
    (Generator) Yields each of the alphabetic characters in a string.
    """
    for letter in text:
        if letter.isalpha ():
            yield letter

# Example: Count letter frequencies (case-insensitive), take 1
text = """How much wood could a woodchuck chuck
if a woodchuck could chuck wood?"""

freqs1 = {} # Frequency table for method 1

for letter in alpha_chars (text.lower ()):
    freqs1[letter] += 1

print (freqs1)

# Quick check
assert freqs1['o'] == 11 and freqs1['h'] == 6
print ("\n(Passed a quick, partial test.)")