I am working on the file "07_tensorboard_ex6.ipynb".
The first task is:
TensorBoard
Exercise 6

In this exercise, you will learn to use TensorBoard. It is a great visualization tool that comes with TensorFlow. It works by parsing special TensorFlow logs, called summaries, and displaying them nicely.
6.1) Starting the TensorBoard server. Open a Terminal and type the following commands.
Move to the tensorflow-safari-course directory:
~$ cd tensorflow-safari-course
Create the tf_logs directory that will hold the TensorFlow data that we will want TensorBoard to display:
~/tensorflow-safari-course$ mkdir tf_logs
Activate the virtual environment:
~/tensorflow-safari-course$ source env/bin/activate
Start the TensorBoard server:
(env) ~/tensorflow-safari-course$ tensorboard --logdir=tf_logs
Starting TensorBoard b'41' on port 6006
(You can navigate to http://127.0.1.1:6006 )
Now visit the URL given by TensorBoard. You should see the TensorBoard interface.
6.2) Now create a tf.summary.FileWriter, with the parameters: logdir="tf_logs/run_number_1/" and graph=graph where graph is the one we built just before this exercise. This will automatically:
* create the run_number_1 directory inside the tf_logs directory,
* create an events.out.tfevents.* file in that subdirectory that will contain the data that TensorBoard will display,
* write the graph's definition to this file.
Next, try refreshing the TensorBoard page in your browser (you may need to wait a couple minutes for it to detect the change, or else you can just restart the TensorBoard server). Visit the Graph tab: you should be able to visualize the graph.
6.3) As you can see, the graph looks really messy in TensorBoard. We need to organize it a bit. For this, name scopes come in handy. An operation can be placed inside a name scope in one of two ways:

Add the scope as a prefix to the operation's name, for example:

python
a = tf.constant(0.0, name="my_name_scope/a")

Or (generally clearer) use a tf.name_scope() block, for example:

python
with tf.name_scope("my_name_scope"):
    a = tf.constant(0.0, name="a")
Add name scopes to the...
Can you write Python code for it?

filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5

graph = tf.Graph()
with graph.as_default():
    reader = tf.TextLineReader(skip_header_lines=1)

    filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
    record_id, record = reader.read(filename_queue)

    record_defaults = [[''], [0.0], [0.0]]
    country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)

    X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
    X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
    y_batch_reshaped = tf.reshape(y_batch, [-1, 1])

    X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
    y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")

    b = tf.Variable(0.0, name="b")
    w = tf.Variable(tf.zeros([1, 1]), name="w")
    y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")  # X @ w + b

    mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
    global_step = tf.Variable(0, trainable=False, name='global_step')
    optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
    training_op = optimizer.minimize(mse, global_step=global_step)

    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()


I am working on the file "07_training/07a_ingest.ipynb".
The first task is:
Efficient Ingest
In this notebook, we speed the ingest of training/evaluation data into the model.
Enable GPU and set up helper functions
This notebook and pretty much every other notebook in this repository
will run faster if you are using a GPU.
On Colab:
- Navigate to Edit→Notebook Settings
- Select GPU from the Hardware Accelerator drop-down
On Cloud AI Platform Notebooks:
- Navigate to https://console.cloud.google.com/ai-platform/notebooks
- Create an instance with a GPU or select your instance and add a GPU
Next, we'll confirm that we can connect to the GPU with tensorflow:
Can you write Python code for it?

import tensorflow as tf
print('TensorFlow version' + tf.version.VERSION)
print('Built with GPU support? ' + ('Yes!' if tf.test.is_built_with_cuda() else 'Noooo!'))
print('There are {} GPUs'.format(len(tf.config.experimental.list_physical_devices("GPU"))))
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))


I am working on the file "07_training/07a_ingest.ipynb".
The first task is:
Speeding up the reading of data
To try it out, we'll simply read through the data several times and compute some quantity on the images.
Can you write Python code for it?

def loop_through_dataset(ds, nepochs):
    lowest_mean = tf.constant(1.)
    for epoch in range(nepochs):
        thresh = np.random.uniform(0.3, 0.7) # random threshold
        count = 0
        sumsofar = tf.constant(0.)
        for (img, label) in ds:
            # mean of channel values > thresh
            mean = tf.reduce_mean(tf.where(img > thresh, img, 0))
            sumsofar = sumsofar + mean
            count = count + 1
            if count%100 == 0:
                print('.', end='')
        mean = sumsofar/count
        print(mean)
        if mean < lowest_mean:
            lowest_mean = mean
    return lowest_mean

PATTERN_SUFFIX, NUM_EPOCHS = '-0000[01]-*', 2 # 2 files, 2 epochs
#PATTERN_SUFFIX, NUM_EPOCHS = '-*', 20 # 16 files, 20 epochs

%%time
ds = create_preproc_dataset_plain(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
)
loop_through_dataset(ds, NUM_EPOCHS)

%%time
# parallel map
ds = create_preproc_dataset_parallelmap(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
)
loop_through_dataset(ds, NUM_EPOCHS)

%%time
# with interleave
ds = create_preproc_dataset_interleave(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX,
    num_parallel=None
)
loop_through_dataset(ds, NUM_EPOCHS)

%%time
# with interleave and parallel mpas
ds = create_preproc_dataset_interleave(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX,
    num_parallel=AUTOTUNE
)
loop_through_dataset(ds, NUM_EPOCHS)


I am working on the file "07_training/07a_ingest.ipynb".
The first task is:
We note that the improvement remains:
| Method                 | CPU time    | Wall time    |
| -----------------------| ----------- | ------------ |
| Plain                  | 9.91s        | 9.39s        |
| Parallel Map           | 10.7s        | 8.17s        |
| Interleave             | 10.5s        | 7.54s        |
| Interleave+Parallel Map| 10.3s        | 7.17s        |
Speeding up the handling of data
Can you write Python code for it?

# alias to the more efficient one
def create_preproc_dataset(pattern):
    return create_preproc_dataset_interleave(pattern, num_parallel=AUTOTUNE)

%%time
# add prefetching
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).prefetch(AUTOTUNE).batch(1)
train_simple_model(ds, NUM_EPOCHS)

%%time
# Add batching of different sizes
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).prefetch(AUTOTUNE).batch(8)
train_simple_model(ds, NUM_EPOCHS)

%%time
# Add batching of different sizes
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).prefetch(AUTOTUNE).batch(16)
train_simple_model(ds, NUM_EPOCHS)

%%time
# Add batching of different sizes
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).prefetch(AUTOTUNE).batch(32)
train_simple_model(ds, NUM_EPOCHS)

%%time
# add caching: always do this optimization last.
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).cache().batch(32)
train_simple_model(ds, NUM_EPOCHS)

%%time
# add caching: always do this optimization last.
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).prefetch(AUTOTUNE).cache().batch(32)
train_simple_model(ds, NUM_EPOCHS)

%%time
# add caching: always do this optimization last.
ds = create_preproc_dataset(
    'gs://practical-ml-vision-book/flowers_tfr/train' + PATTERN_SUFFIX
).cache().prefetch(AUTOTUNE).batch(32)
train_simple_model(ds, NUM_EPOCHS)


I am working on the file "07_training/07b_gpumax.ipynb".
The first task is:
2. Pyfunc
If you absolutely need to iterate or call Python-only
functionality (like time/json/etc.) and still need to use map(),
you can use a py_func
Data is still moved out of the graph, the job done, and data moved back into
the graph. So, efficiency-wise it's not a gain.
Can you write Python code for it?

def to_grayscale_numpy(img):
    # the numpy happens here
    img = img.numpy()
    rows, cols, _ = img.shape
    result = np.zeros([rows, cols], dtype=np.float32)
    for row in range(rows):
        for col in range(cols):
            red = img[row][col][0]
            green = img[row][col][1]
            blue = img[row][col][2]
            c_linear = 0.2126 * red + 0.7152 * green + 0.0722 * blue
            if c_linear > 0.0031308:
                result[row][col] = 1.055 * pow(c_linear, 1/2.4) - 0.055
            else:
                result[row][col] = 12.92*c_linear
    # the convert back happens here
    return tf.convert_to_tensor(result)

def to_grayscale(img):
    return tf.py_function(to_grayscale_numpy, [img], tf.float32)

%%time
ds = tf.data.TextLineDataset('input.txt').repeat(10).map(read_jpeg).map(to_grayscale)

overall = tf.constant([0.], dtype=tf.float32)
count = 0
for gray in ds:
    m = tf.reduce_mean(gray, axis=[0, 1])
    overall += m
    count += 1
print(overall/count)


I am working on the file "07_training/07b_gpumax.ipynb".
The first task is:
3. Use TensorFlow slicing and tf.where
This is 10x faster than iterating.
Can you write Python code for it?

# All in GPU
def to_grayscale(img):
    # TensorFlow slicing functionality
    red = img[:, :, 0]
    green = img[:, :, 1]
    blue = img[:, :, 2]
    # All these are actually tf.mul(), tf.add(), etc.
    c_linear = 0.2126 * red + 0.7152 * green + 0.0722 * blue
    # Use tf.cond and tf.where for if-then statements
    gray = tf.where(c_linear > 0.0031308,
                    1.055 * tf.pow(c_linear, 1/2.4) - 0.055,
                    12.92*c_linear)
    return gray

%%time
ds = tf.data.TextLineDataset('input.txt').repeat(10).map(read_jpeg).map(to_grayscale)

overall = tf.constant([0.])
count = 0
for gray in ds:
    m = tf.reduce_mean(gray, axis=[0, 1])
    overall += m
    count += 1
print(overall/count)


I am working on the file "07_training/07b_gpumax.ipynb".
The first task is:
4. Use Matrix math and tf.where
This is 3x as fast as the slicing.
Can you write Python code for it?

def to_grayscale(img):
    wt = tf.constant([[0.2126], [0.7152], [0.0722]]) # 3x1 matrix
    c_linear = tf.matmul(img, wt) # (ht,wd,3) x (3x1) -> (ht, wd)
    gray = tf.where(c_linear > 0.0031308,
                    1.055 * tf.pow(c_linear, 1/2.4) - 0.055,
                    12.92*c_linear)
    return gray

%%time
ds = tf.data.TextLineDataset('input.txt').repeat(10).map(read_jpeg).map(to_grayscale)

overall = tf.constant([0.])
count = 0
for gray in ds:
    m = tf.reduce_mean(gray, axis=[0, 1])
    overall += m
    count += 1
print(overall/count)


I am working on the file "07_training/07b_gpumax.ipynb".
The first task is:
Ingest code
Can you write Python code for it?

%%writefile input.txt
gs://practical-ml-vision-book/images/california_fire1.jpg
gs://practical-ml-vision-book/images/california_fire2.jpg

import matplotlib.pylab as plt
import numpy as np
import tensorflow as tf

def read_jpeg(filename):
    img = tf.io.read_file(filename)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = tf.reshape(img, [338, 600, 3])
    return img

ds = tf.data.TextLineDataset('input.txt').map(read_jpeg)
f, ax = plt.subplots(1, 2, figsize=(15,10))
for idx, img in enumerate(ds):
    ax[idx].imshow( img.numpy() );


I am working on the file "07_training/07b_gpumax.ipynb".
The first task is:
Results
When we did it, these were the timings we got:
| Method                 | CPU time    | Wall time    |
| ---------------------- | ----------- | ------------ |
| Iterate                | 39.6s       | 41.1s        |
| Pyfunc                 | 39.7s       | 41.1s        |
| Slicing                | 4.44s       | 3.07s        |
| Matmul                 | 1.22s       | 2.29s        |
| Batch                  | 1.11s       | 2.13s        |
Signature
Playing around with signature
Can you write Python code for it?

from inspect import signature

def myfunc(a, b):
    return (a + b)

print(myfunc(3,5))
print(myfunc('foo', 'bar'))
print(signature(myfunc).parameters)
print(signature(myfunc).return_annotation)

from inspect import signature

def myfunc(a: int, b: float) -> float:
    return (a + b)

print(myfunc(3,5))
print(myfunc('foo', 'bar')) # runtime doesn't check
print(signature(myfunc).parameters)
print(signature(myfunc).return_annotation)

from inspect import signature
import tensorflow as tf

@tf.function(input_signature=[
    tf.TensorSpec([3,5], name='a'),
    tf.TensorSpec([5,8], name='b')
])
def myfunc(a, b):
    return (tf.matmul(a,b))

print(myfunc.get_concrete_function(tf.ones((3,5)), tf.ones((5,8))))


I am working on the file "07_training/07c_export.ipynb".
The first task is:
Save model, then load it to make predictions
This way, we don't have to have the model in memory
Can you write Python code for it?

import os, shutil
shutil.rmtree('export', ignore_errors=True)
os.mkdir('export')
model.save('export/flowers_model')

!ls export/flowers_model

!saved_model_cli show --tag_set serve --signature_def serving_default --dir export/flowers_model

## for prediction, we won't have TensorFlow Records.
## this is how we'd predict for individual images
filenames = [
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9158041313_7a6a102f7a_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
]
serving_model = tf.keras.models.load_model('export/flowers_model')
input_images = [create_preproc_image(f) for f in filenames]
f, ax = plt.subplots(1, 6, figsize=(15,15))
for idx, img in enumerate(input_images):
    ax[idx].imshow((img.numpy()));
    batch_image = tf.reshape(img, [1, IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS])
    batch_pred = serving_model.predict(batch_image)
    pred = batch_pred[0]
    pred_label_index = tf.math.argmax(pred).numpy()
    pred_label = CLASS_NAMES[pred_label_index]
    prob = pred[pred_label_index]
    ax[idx].set_title('{} ({:.2f})'.format(pred_label, prob))


I am working on the file "07_training/07c_export.ipynb".
The first task is:
Save the model to Google Cloud Storage
Save the model so that it is available even after this Notebook instance is shutdown.
Change the BUCKET to be a bucket that you own.
Can you write Python code for it?

%%bash
PROJECT=$(gcloud config get-value project)
BUCKET=${PROJECT}  # create a bucket with this name, or change to a bucket that you own.

gsutil cp -r export/flowers_model gs://${BUCKET}/flowers_5_trained


I am working on the file "07_training/07c_export.ipynb".
The first task is:
Serving signature
Expecting clients to do all this reshaping is not realistic.
Let's do two things:
(1) Define a serving signature that simply takes the name of a file
(2) Vectorize the code to accept a batch of filenames, so that we can do it all in one go.
Can you write Python code for it?

## it's better to vectorize the prediction
filenames = tf.convert_to_tensor([
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9158041313_7a6a102f7a_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
])
print(filenames)
input_images = tf.map_fn(create_preproc_image,
                         filenames, fn_output_signature=tf.float32)
batch_pred = serving_model.predict(input_images)
print('full probs:\n', batch_pred)
top_prob = tf.math.reduce_max(batch_pred, axis=[1])
print('top prob:\n', top_prob)
pred_label_index = tf.math.argmax(batch_pred, axis=1)
print('top cls:\n', pred_label_index)
pred_label = tf.gather(tf.convert_to_tensor(CLASS_NAMES), pred_label_index)
print(pred_label)

@tf.function(input_signature=[tf.TensorSpec([None,], dtype=tf.string)])
def predict_flower_type(filenames):
    input_images = tf.map_fn(
        create_preproc_image,
        filenames,
        fn_output_signature=tf.float32
    )
    batch_pred = model(input_images) # same as model.predict()
    top_prob = tf.math.reduce_max(batch_pred, axis=[1])
    pred_label_index = tf.math.argmax(batch_pred, axis=1)
    pred_label = tf.gather(tf.convert_to_tensor(CLASS_NAMES), pred_label_index)
    return {
        'probability': top_prob,
        'flower_type_int': pred_label_index,
        'flower_type_str': pred_label
    }

shutil.rmtree('export', ignore_errors=True)
os.mkdir('export')
model.save('export/flowers_model',
          signatures={
              'serving_default': predict_flower_type
          })

serving_fn =...


I am working on the file "07_training/07c_export.ipynb".
The first task is:
Training code
This is the original code, from ../06_preprocessing/06e_colordistortion.ipynb
modified to use the most efficient ingest as determined in ./07a_ingest.ipynb.
Can you write Python code for it?

import matplotlib.pylab as plt
import numpy as np
import tensorflow as tf
import tensorflow_hub as hub
import os
# Load compressed models from tensorflow_hub
os.environ['TFHUB_MODEL_LOAD_FORMAT'] = 'COMPRESSED'

from tensorflow.data.experimental import AUTOTUNE

IMG_HEIGHT = 448 # note *twice* what we used to have
IMG_WIDTH = 448
IMG_CHANNELS = 3
CLASS_NAMES = 'daisy dandelion roses sunflowers tulips'.split()

def training_plot(metrics, history):
  f, ax = plt.subplots(1, len(metrics), figsize=(5*len(metrics), 5))
  for idx, metric in enumerate(metrics):
    ax[idx].plot(history.history[metric], ls='dashed')
    ax[idx].set_xlabel("Epochs")
    ax[idx].set_ylabel(metric)
    ax[idx].plot(history.history['val_' + metric]);
    ax[idx].legend([metric, 'val_' + metric])

class _Preprocessor:
    def __init__(self):
        # nothing to initialize
        pass

    def read_from_tfr(self, proto):
        feature_description = {
            'image': tf.io.VarLenFeature(tf.float32),
            'shape': tf.io.VarLenFeature(tf.int64),
            'label': tf.io.FixedLenFeature([], tf.string, default_value=''),
            'label_int': tf.io.FixedLenFeature([], tf.int64, default_value=0),
        }
        rec = tf.io.parse_single_example(
            proto, feature_description
        )
        shape = tf.sparse.to_dense(rec['shape'])
        img = tf.reshape(tf.sparse.to_dense(rec['image']), shape)
        label_int = rec['label_int']
        return img, label_int

    def read_from_jpegfile(self, filename):
        # same code as in 05_create_dataset/jpeg_to_tfrecord.py
        img = tf.io.read_file(filename)
        img = tf.image.decode_jpeg(img, channels=IMG_CHANNELS)
        img = tf.image.convert_image_dtype(img, tf.float32)
        return img

    def preprocess(self, img):
        return tf.image.resize_with_pad(img, IMG_HEIGHT, IMG_WIDTH)

# most efficient way to read the data
# as determined in 07a_ingest.ipynb
# splits the files into two halves and interleaves datasets
def...


I am working on the file "07_ubiquitous_numpy.ipynb".
The first task is:
Clustering
Clustering example on iris dataset data using sklearn.cluster.KMeans
Can you write Python code for it?

from sklearn.cluster import KMeans

kmean = KMeans(n_clusters=3)
kmean.fit(iris.data)
kmean.cluster_centers_

kmean.cluster_centers_.shape


I am working on the file "07_ubiquitous_numpy.ipynb".
The first task is:
Features in the Iris dataset:

sepal length in cm

sepal width in cm
petal length in cm

petal width in cm

Target classes to predict:

Iris Setosa

Iris Versicolour
Iris Virginica
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()


I am working on the file "07_ubiquitous_numpy.ipynb".
The first task is:
Plotting using matplotlib
Matplotlib is one of the most popular and widely used plotting library in Python. Matplotlib is tightly integrated with NumPy as all the functions expect ndarray in input.
Can you write Python code for it?

from itertools import combinations
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline

rgb = np.empty(shape=y.shape, dtype='<U1')
rgb[y==0] = 'r'
rgb[y==1] = 'g'
rgb[y==2] = 'b'

for cols in combinations(range(4), 2):
    f, ax = plt.subplots(figsize=(7.5, 7.5))
    ax.scatter(X[:, cols[0]], X[:, cols[1]], c=rgb)
    ax.scatter(kmean.cluster_centers_[:, cols[0]],
               kmean.cluster_centers_[:, cols[1]], marker='*', s=250,
               color='black', label='Centers')
    feature_x = iris.feature_names[cols[0]]
    feature_y = iris.feature_names[cols[1]]
    ax.set_title("Features: {} vs {}".format(feature_x.title(),
                                            feature_y.title()))
    ax.set_xlabel(feature_x)
    ax.set_ylabel(feature_y)
    ax.legend(loc='best')

    plt.show()


I am working on the file "07_ubiquitous_numpy.ipynb".
The first task is:
Try by yourself one of the following commands where 'd' is the variable containing the dataset:
print(iris.keys())           # Structure of the contained data
print(iris.DESCR)            # A complete description of the dataset
print(iris.data.shape)       # [n_samples, n_features]
print(iris.target.shape)     # [n_samples,]
print(iris.feature_names)
datasets.get_data_home() # This is where the datasets are stored
Can you write Python code for it?

print(iris.keys())

print(iris.DESCR)

print(type(iris.data))

X = iris.data
print(X.size, X.shape)

y = iris.target
type(y)


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
And the effect of the FIR digital filter:
Can you write Python code for it?

# Apply FIR filter
sig_filtered = signal.filtfilt(b, 1, Y_noise)

# Determine approx PSD
f, Pxx_den_f = signal.periodogram(sig_filtered, sr)

# Plot
G = gridspec.GridSpec(2, 1)
axis1 = plt.subplot(G[0, 0])
axis1.plot(f, Pxx_den)
axis1.set_title('Approx PSD of Original Signal')

axis2 = plt.subplot(G[1, 0])
axis2.plot(f, Pxx_den_f)
axis2.set_title('Approx PSD of Filtered Signal')

plt.tight_layout()
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Correlate the single signal with the repeating, noisy one:
Can you write Python code for it?

# Correlate
corr = signal.correlate(hrtbt_noise, heartbeat, mode='same')

# Plot
plt.plot(corr/max(corr), label='Corelogram')
plt.plot(cent_x,cent_y,'r^', label='Centroid')
plt.title('Correlogram')
plt.xlabel('Delay')
plt.ylabel('Normalized Volts $^2$')
plt.legend(loc='best')
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Correlation
Correlations are a measure of the product of two signals as a function of the x-axis shift between them. They are often used to determine similarity between the two signals, e.g. is there some structure or repeating feature that is present in both signals?
Can you write Python code for it?

# Create a signal
npts = 200
heartbeat = np.array([0,1,0,0,4,8,2,-4,0,4,0,1,2,1,0,0,0,0])/8
xvals  = np.linspace(0,len(heartbeat),npts)
heartbeat = np.interp(xvals,np.arange(0,len(heartbeat)),heartbeat) # Use interpolation to spread the signal out

# Repeat the signal ten times, add some noise:
hrtbt = np.tile(heartbeat,10)
hrtbt_noise = hrtbt + np.random.rand(len(hrtbt))

# Plot
G = gridspec.GridSpec(2, 1)
axis1 = plt.subplot(G[0, 0])
axis1.plot(heartbeat)
axis1.set_title('Single Heartbeat Electrocardiogram')

axis2 = plt.subplot(G[1, 0])
axis2.plot(hrtbt_noise)
axis2.set_title('Noisy Electrocardiogram for repeating Heartbeat')

plt.tight_layout()
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Deconvolution
Deconvolution can be thought of as removing the filter or instrument response. This is pretty common when reconstructing real signals if the response is known.
In the microscope example, this would be deconvolving image with a known response of the instrument to a point source. If it is known how much the entire image is spread out, the original image can be recovered.
Can you write Python code for it?

# Deconvolve
recovered, remainder = signal.deconvolve(convolved, psf)

# Plot
G = gridspec.GridSpec(3, 1)
axis1 = plt.subplot(G[0, 0])
axis1.plot(convolved)
axis1.set_xlim(0, len(convolved))
axis1.set_title('Convolved Signal')

axis2 = plt.subplot(G[1, 0])
axis2.plot(psf)
axis2.set_xlim(0, len(convolved))
axis2.set_title('Known Impulse Response')

axis3 = plt.subplot(G[2, 0])
axis3.plot(recovered)
axis3.set_xlim(0, len(convolved))
axis3.set_title('Recovered Pulse')

plt.tight_layout()
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Filtering
Filters recieve a signal input and selectively reduce the amplitude of certain frequencies. Working with digital signals, they can broadly be divided into infinitie impulse response (IIR) and finite impulse response (FIR).
IIR filters that receive an impulse response (signal of value 1 followed by many zeros) yield a (theoretically) infinite number of non-zero values.
This is in contrast with the the finite impulse response (FIR) filter that receives an impulse response and does become exactly zero beyond the duration of the impulse.
IIR and FIR filters are also different in that they have different filter coefficients (b, a) that represent the feed forward coefficients and feedback coefficients, respectively. Feed forward coefficients (b) are applied to input (x) values, and feedback coefficients (a) are applied to output (y) values - i.e:
$y(n) = b_1x(n) + b_2x(n) - a_1y(n) - a_2y(n) $
where:
$b_jx(n)$ are the feed forward coefficients, using $x$ values
$a_jy(n)$ are the feedback coefficients (notice the $y(n)$!)
Generate Signal
First, we generate a signal and approximate the power spectral density (PSD):
Can you write Python code for it?

frq1 = 250                    # Frequency 1(hz)
amp1 = 3                      # Amplitude 1
sr = 2000                     # Sample rate
dur = 1                       # Duration (s) (increasing/decreasing this changes S/N)

# Create timesteps, signal and noise
X = np.linspace(0, dur-1/sr, int(dur*sr))  # Time
Y = amp1*np.sin(X*2*np.pi*frq1)            # Signal
Y_noise = Y + 40*np.random.rand(len(X))    # Add noise

# Approx PSD
f, Pxx_den = signal.periodogram(Y_noise, sr)

# Plot
G = gridspec.GridSpec(2, 1)
axis1 = plt.subplot(G[0, 0])
axis1.plot(X, Y_noise)
axis1.set_title('Plot of Signal with Noise')

axis2 = plt.subplot(G[1, 0])
axis2.plot(f, Pxx_den)
axis2.set_title('Approx PSD')

plt.tight_layout()
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Infinite Impulse Response (IIR) filters
Digital filters inherently account for digital signal limitations, i.e. the sampling frequency. The Nyquist theorem asserts that we can't measure frequencies that are higher than 1/2 the sampling frequency, and the digital filter operates on this principle.
Next, we create the digital filter and plot the response, using both feedforward (b) and feedback (a) coefficeints.:
Can you write Python code for it?

f_order = 10.0             # Filter order
f_pass = 'low'           # Filter is low pass
f_freq = 210.0             # Frequency to pass
f_cutoff = f_freq/(sr/2) # Convert frequency into

# Create the filter
b, a = signal.iirfilter(f_order, f_cutoff, btype=f_pass, ftype='butter')

# Test the filter
w, h = signal.freqz(b, a, 1000)  # Test response of filter across
                                 #    frequencies (Use 'freqz' for digital)
freqz_hz = w * sr / (2 * np.pi)  # Convert frequency to Hz
resp_db = 20 * np.log10(abs(h))  # Convert response to decibels

# Plot filter response
plt.semilogx(freqz_hz, resp_db )
plt.title('Butterworth Bandpass Frequency Response')
plt.xlabel('Frequency (Hz)')
plt.ylabel('Amplitude (dB)')
plt.axis((100, 500, -200, 10))
plt.grid(which='both', axis='both')
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Inverse Fourier transform back, and plot the original filtered signal:
Can you write Python code for it?

# Inverse FFT the original, non-absolute, full spectrum
Y2 = np.fft.ifft(Y_filt)
Y2 = np.real(Y2) # Use the real values to plot the filtered signal

# Plot
plt.plot(X[:100],Y_s[:100], label='Original')
plt.plot(X[:100],Y2[:100], label='Filtered')
plt.title('Two Signals')
plt.xlabel('Time (s)')
plt.ylabel('Signal Amplutude')
plt.legend(loc='best')
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
Phase Example
Phase is shift of a periodic signal 'left' or 'right'. it is the $\phi_x$ in the following equation:
$$ Y = \frac{1}{2} a_0 \sum_{n=1}^{\infty} a_n cos (n x + \phi_x) $$
Can you write Python code for it?

# We can use the previous signal to get the phase:

# Set a tolerance limit - phase is sensitive to floating point errors
# (see Gotchas and Optimization for more info):
FT_trun = FT
tol = 1*10**-6 # Truncate signal below tolerance level
FT_trun[np.abs(FT_trun)<tol] = 0

# Use the angle function (arc tangent of imaginary over real)
phase = np.angle(FT_trun[len(X)//2:])
phase_rad = 1/np.pi * phase # Convert to radians

# Plot
plt.plot(freqs[1:], phase_rad[1:])
plt.title('Single-Sided Spectrum of Signal')
plt.xlabel('freq (Hz)')
plt.ylabel('Freq Amplitude')
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
The amplitudes don't seem quite right - longer duration increases the signal to noise and gives a better result:
Can you write Python code for it?

# Create signal
sr = 2000                      # Sample rate
dur = 10  # Increased duration (s) (increasing/decreasing this changes S/N)
X = np.linspace(0, dur-1/sr, int(dur*sr))  # Time
Y_s = amp1*np.sin(X*2*np.pi*frq1 - np.pi/4) + amp2*np.sin(X*2*np.pi*frq2 + np.pi/2)
Y_sn = Y_s + 40*np.random.rand(len(X))

# Determine Single Sided FT Spectrum
Y_sn_fft = np.fft.fft(Y_sn)

# Update ft output
FT = np.roll(Y_sn_fft, len(X)//2)     # Shift zero freq component to center of spectrum
SSFT_amp = np.abs(FT)[len(X)//2:]     # Use the absolute value for amplitude; spectrum is symmetric - start from zero freq component
SSFT_amp = 2/len(X) * SSFT_amp        # Scale by 2 (using half the spectrum) / number points

# Determine frequencies
freqs = sr/len(X)*np.arange(0,len(SSFT_amp))

# Plot
plt.plot(freqs[1:], SSFT_amp[1:])
plt.title('Single-Sided Spectrum of Signal')
plt.xlabel('freq (Hz)')
plt.ylabel('Freq Amplitude')
plt.show()

# Create signal
sr = 2000                      # Sample rate
dur = 10  # Increased duration (s) (increasing/decreasing this changes S/N)
X = np.linspace(0, dur-1/sr, int(dur*sr))  # Time
Y_s = amp1*np.cos(X*2*np.pi*frq1 - np.pi/4) + amp2*np.cos(X*2*np.pi*frq2 + np.pi/2)
Y_sn = Y_s + 40*np.random.rand(len(X))

# Determine Single Sided FT Spectrum
Y_sn_fft = np.fft.fft(Y_sn)

# Update ft output
FT = np.roll(Y_sn_fft, len(X)//2)     # Shift zero freq component to center of spectrum
SSFT_amp = np.abs(FT)[len(X)//2:]     # Use the absolute value for amplitude; spectrum is symmetric - start from zero freq component
SSFT_amp = 2/len(X) * SSFT_amp        # Scale by 2 (using half the spectrum) / number points

# Determine frequencies
freqs = sr/len(X)*np.arange(0,len(SSFT_amp))

# Plot
plt.plot(freqs[1:], SSFT_amp[1:])
plt.title('Single-Sided Spectrum of Signal')
plt.xlabel('freq (Hz)')
plt.ylabel('Freq Amplitude')
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
This is better visualized without noise:
Can you write Python code for it?

# Determine Single Sided FT Spectrum
Y_s_fft = np.fft.fft(Y_s)

# Update ft output
FT = np.roll(Y_s_fft, len(X)//2)

# Set a tolerance limit - phase is sensitive to floating point errors
# (see Gotchas and Optimization for more info):
FT_trun = FT
tol = 1*10**-6 # Truncate signal below tolerance level
FT_trun[np.abs(FT_trun)<tol] = 0

# Use the angle function (arc tangent of imaginary over real)
phase = np.angle(FT_trun[len(X)//2:])
phase_rad = 1/np.pi * phase # Convert to radians

# Plot
plt.plot(freqs[1:], phase_rad[1:])
plt.title('Single-Sided Spectrum of Signal')
plt.xlabel('freq (Hz)')
plt.ylabel('Freq Amplitude')
plt.show()

# To streamline, we can create a function in Pandas (see Pandas Crash Course for more info):
import pandas as pd

def fft_norm(signal, sr=1):
    '''Return a normalized fft single sided spectrum.'''
    signal = signal[: signal.shape[0]//2*2] # Use an even number of data points so can divide FFT in half evenly
    N = signal.shape[0]
    freqs = sr*np.arange(0, N//2)/N

    # FFT
    fft = np.fft.fft(signal)
    fft = np.roll(fft, N//2) # Shift zero freq component to center of spectrum

    # Normalized Amplitude
    amp_norm = 2/N*np.abs(fft[N//2:])

    # Phase
    tol = 1*10**-6 # Truncate signal below tolerance so phase isn't weird
    fft[np.abs(fft)<tol] = 0
    phase_rad = np.angle(fft[N//2:])/(np.pi)

    # To convert the phase, use (fft_norm(Phase (Radians)+np.pi)) * conversion factor/(2*np.pi)
    # I.e. add Pi to the output before converting from radians
    return  pd.DataFrame({'Frequency':freqs, 'Amplitude':amp_norm, 'Phase (Radians)':phase_rad, 'Phase (Degrees)':phase_rad*180}).set_index('Frequency')

Y_ms = Y_s-Y_s.mean() # Mean subtract to remove the offset (0 freq component)
fft_norm(Y_ms, sr=2000).plot(subplots=True, layout=(3,1)) # mean subtract to
plt.show()


I am working on the file "08 - Signal Processing - Scipy.ipynb".
The first task is:
This shows the phase for every single frequency, but we really only care about the nonzero frequencies with minimum amplitude:
Can you write Python code for it?

nonzero_freqs = freqs[SSFT_amp > 1][1:]
print('Notable frequencies are: {}'.format(nonzero_freqs))

inds = [list(freqs).index(x) for x in nonzero_freqs] # Return index of nonzero frequencies
print('Phase shifts for notable frequencies are: {}'.format(phase_rad[inds]))


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
A method is just a function that is part of a class. Since it is just a function, you can do anything with a method that you learned about with functions. You can accept positional arguments, keyword arguments, an arbitrary list of argument values, an arbitrary dictionary of arguments, or any combination of these. Your arguments can return a value or a set of values if you want, or they can just do some work without returning any values.
Each method has to accept one argument by default, the value self. This is a reference to the particular object that is calling the method. This self argument gives you access to the calling object's attributes. In this example, the self argument is used to access a Rocket object's y-value. That value is increased by 1, every time the method move_up() is called by a particular Rocket object. This is probably still somewhat confusing, but it should start to make sense as you work through your own examples.
If you take a second look at what happens when a method is called, things might make a little more sense:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self):
        # Each rocket has an (x,y) position.
        self.x = 0
        self.y = 0

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1

# Create a Rocket object, and have it start to move up.
my_rocket = Rocket()
print("Rocket altitude:", my_rocket.y)

my_rocket.move_up()
print("Rocket altitude:", my_rocket.y)

my_rocket.move_up()
print("Rocket altitude:", my_rocket.y)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
After this, classes are accessed using dot notation:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
All the __init__() method does so far is set the x and y values for the rocket to 0. We can easily add a couple keyword arguments so that new rockets can be initialized at any position:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
Hopefully these short refinements show that you can extend a class' attributes and behavior to model the phenomena you are interested in as closely as you want. The rocket could have a name, a crew capacity, a payload, a certain amount of fuel, and any number of other attributes. You could define any behavior you want for the rocket, including interactions with other rockets and launch facilities, gravitational fields, and whatever you need it to! There are techniques for managing these more complex interactions, but what you have just seen is the core of object-oriented programming.
At this point you should try your hand at writing some classes of your own. After trying some exercises, we will look at object inheritance, and then you will be ready to move on for now.
top
Exercises
Your Own Rocket 2

There are enough new concepts here that you might want to try re-creating the Rocket class as it has been developed so far, looking at the examples as little as possible. Once you have your own version, regardless of how much you needed to look at the example, you can modify the class and explore the possibilities of what you have already learned.
Re-create the Rocket class as it has been developed so far:
Define the Rocket() class.
Define the __init__() method. Let your __init__() method accept x and y values for the initial position of the rocket. Make sure the default behavior is to position the rocket at (0,0).
Define the move_rocket() method. The method should accept an amount to move left or right, and an amount to move up or down.
Create a Rocket object. Move the rocket around, printing its position after each move.
Create a small fleet of rockets. Move several of them around, and print their final positions to prove that each rocket can move independently of the other rockets.
Define the get_distance() method. The method should accept a Rocket object, and calculate the distance between the current rocket and the rocket that is passed into the method.
Use the...
Can you write Python code for it?

# Ex 9.2 : Your Own Rocket 2

# put your code here

# Ex 9.3 : Rocket Attributes

# put your code here

# Ex 9.4 : Rocket Methods

# put your code here

# Ex 9.5 : Person Class

# put your code here

# Ex 9.6 : Car Class

# put your code here


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
If you are comfortable using list comprehensions, go ahead and use those as much as you can. I'd rather not assume at this point that everyone is comfortable with comprehensions, so I will use the slightly longer approach of declaring an empty list, and then using a for loop to fill that list. That can be done slightly more efficiently than the previous example, by eliminating the temporary variable new_rocket:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self):
        # Each rocket has an (x,y) position.
        self.x = 0
        self.y = 0

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1

# Create a fleet of 5 rockets, and store them in a list.
my_rockets = []
for x in range(0,5):
    my_rockets.append(Rocket())

# Show that each rocket is a separate object.
for rocket in my_rockets:
    print(rocket)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
Now when you create a new Rocket object you have the choice of passing in arbitrary initial values for x and y:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1

# Make a series of rockets at different starting places.
rockets = []
rockets.append(Rocket())
rockets.append(Rocket(0,10))
rockets.append(Rocket(100,0))

# Show where each rocket is.
for index, rocket in enumerate(rockets):
    print("Rocket %d is at (%d, %d)." % (index, rocket.x, rocket.y))


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
Now you can import the Rocket and the Shuttle class, and use them both in a clean uncluttered program file:
Can you write Python code for it?

# Save as rocket_game.py
from rocket import Rocket, Shuttle

rocket = Rocket()
print("The rocket is at (%d, %d)." % (rocket.x, rocket.y))

shuttle = Shuttle()
print("\nThe shuttle is at (%d, %d)." % (shuttle.x, shuttle.y))
print("The shuttle has completed %d flights." % shuttle.flights_completed)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
Now you can import the file multiplying.py, and use these functions. Using the from module_name import function_name syntax:
Can you write Python code for it?

from multiplying import double, triple, quadruple

print(double(5))
print(triple(5))
print(quadruple(5))


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
The Rocket class can now store some information, and it can do something. But this code has not actually created a rocket yet. Here is how you actually make a rocket:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self):
        # Each rocket has an (x,y) position.
        self.x = 0
        self.y = 0

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1

# Create a Rocket object.
my_rocket = Rocket()
print(my_rocket)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
The __init__() function of the new class needs to call the __init__() function of the parent class. The __init__() function of the new class needs to accept all of the parameters required to build an object from the parent class, and these parameters need to be passed to the __init__() function of the parent class. The super().__init__() function takes care of this:
Can you write Python code for it?

class NewClass(ParentClass):

    def __init__(self, arguments_new_class, arguments_parent_class):
        super().__init__(arguments_parent_class)
        # Code for initializing an object of the new class.


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
The first line shows how a class is created in Python. The keyword class tells Python that you are about to define a class. The rules for naming a class are the same rules you learned about naming variables, but there is a strong convention among Python programmers that classes should be named using CamelCase. If you are unfamiliar with CamelCase, it is a convention where each letter that starts a word is capitalized, with no underscores in the name. The name of the class is followed by a set of parentheses. These parentheses will be empty for now, but later they may contain a class upon which the new class is based.
It is good practice to write a comment at the beginning of your class, describing the class. There is a more formal syntax for documenting your classes, but you can wait a little bit to get that formal. For now, just write a comment at the beginning of your class summarizing what you intend the class to do. Writing more formal documentation for your classes will be easy later if you start by writing simple comments now.
Function names that start and end with two underscores are special built-in functions that Python uses in certain ways. The __init()__ method is one of these special functions. It is called automatically when you create an object from your class. The __init()__ method lets you make sure that all relevant attributes are set to their proper values when an object is created from the class, before the object is used. In this case, The __init__() method initializes the x and y values of the Rocket to 0.
The self keyword often takes people a little while to understand. The word "self" refers to the current object that you are working with. When you are writing a class, it lets you refer to certain attributes from any other part of the class. Basically, all methods in a class need the self object as their first argument, so they can access any attribute that is part of the class.
Now let's take a closer look at a method.
A simple method
Here is the method that...
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self):
        # Each rocket has an (x,y) position.
        self.x = 0
        self.y = 0

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
The first line tells Python to import both the Rocket and the Shuttle classes from the rocket module. You don't have to import every class in a module; you can pick and choose the classes you care to use, and Python will only spend time processing those particular classes.
A number of ways to import modules and classes
There are several ways to import modules and classes, and each has its own merits.
import module_name
The syntax for importing classes that was just shown:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
The general syntax for this kind of import is:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
This approach is often used to shorten the name of the module, so you don't have to type a long module name before each class name that you want to use. But it is easy to shorten a name so much that you force people reading your code to scroll to the top of your file and see what the shortened name stands for. In this example,
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
This is a really clean and uncluttered file. A rocket is now something you can define in your programs, without the details of the rocket's implementation cluttering up your file. You don't have to include all the class code for a rocket in each of your files that deals with rockets; the code defining rocket attributes and behavior lives in one file, and can be used anywhere.
The first line tells Python to look for a file called rocket.py. It looks for that file in the same directory as your current program. You can put your classes in other directories, but we will get to that convention a bit later. Notice that you do not.
When Python finds the file rocket.py, it looks for a class called Rocket. When it finds that class, it imports that code into the current file, without you ever seeing that code. You are then free to use the class Rocket as you have seen it used in previous examples.
top
Storing multiple classes in a module
A module is simply a file that contains one or more classes or functions, so the Shuttle class actually belongs in the rocket module as well:
Can you write Python code for it?

# Save as rocket.py
from math import sqrt

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_rocket(self, x_increment=0, y_increment=1):
        # Move the rocket according to the paremeters given.
        #  Default behavior is to move the rocket up one unit.
        self.x += x_increment
        self.y += y_increment

    def get_distance(self, other_rocket):
        # Calculates the distance from this rocket to another rocket,
        #  and returns that value.
        distance = sqrt((self.x-other_rocket.x)**2+(self.y-other_rocket.y)**2)
        return distance

class Shuttle(Rocket):
    # Shuttle simulates a space shuttle, which is really
    #  just a reusable rocket.

    def __init__(self, x=0, y=0, flights_completed=0):
        super().__init__(x, y)
        self.flights_completed = flights_completed


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
This is not recommended, for a couple reasons. First of all, you may have no idea what all the names of the classes and functions in a module are. If you accidentally give one of your variables the same name as a name from the module, you will have naming conflicts. Also, you may be importing way more code into your program than you need.
If you really need all the functions and classes from a module, just import the module and use the module_name.ClassName syntax in your program.
You will get a sense of how to write your imports as you read more Python code, and as you write and share some of your own code.
top
A module of functions
You can use modules to store a set of functions you want available in different programs as well, even if those functions are not attached to any one class. To do this, you save the functions into a file, and then import that file just as you saw in the last section. Here is a really simple example; save this is multiplying.py:
Can you write Python code for it?

# Save as multiplying.py
def double(x):
    return 2*x

def triple(x):
    return 3*x

def quadruple(x):
    return 4*x


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
This might seem a little easier to read, but it is preferable to use the super() syntax. When you use super(), you don't need to explicitly name the parent class, so your code is more resilient to later changes. As you learn more about classes, you will be able to write child classes that inherit from multiple parent classes, and the super() function will call the parent classes' __init__() functions for you, in one line. This explicit approach to calling the parent class' __init__() function is included so that you will be less confused if you see it in someone else's code.
The output above shows that a new Shuttle object was created. This new Shuttle object can store the number of flights completed, but it also has all of the functionality of the Rocket class: it has a position that can be changed, and it can calculate the distance between itself and other rockets or shuttles. This can be demonstrated by creating several rockets and shuttles, and then finding the distance between one shuttle and all the other shuttles and rockets.
This example uses a simple function called randint, which generates a random integer between a lower and upper bound, to determine the position of each rocket and shuttle:
Can you write Python code for it?

from math import sqrt
from random import randint

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_rocket(self, x_increment=0, y_increment=1):
        # Move the rocket according to the paremeters given.
        #  Default behavior is to move the rocket up one unit.
        self.x += x_increment
        self.y += y_increment

    def get_distance(self, other_rocket):
        # Calculates the distance from this rocket to another rocket,
        #  and returns that value.
        distance = sqrt((self.x-other_rocket.x)**2+(self.y-other_rocket.y)**2)
        return distance

class Shuttle(Rocket):
    # Shuttle simulates a space shuttle, which is really
    #  just a reusable rocket.

    def __init__(self, x=0, y=0, flights_completed=0):
        super().__init__(x, y)
        self.flights_completed = flights_completed

# Create several shuttles and rockets, with random positions.
#  Shuttles have a random number of flights completed.
shuttles = []
for x in range(0,3):
    x = randint(0,100)
    y = randint(1,100)
    flights_completed = randint(0,10)
    shuttles.append(Shuttle(x, y, flights_completed))

rockets = []
for x in range(0,3):
    x = randint(0,100)
    y = randint(1,100)
    rockets.append(Rocket(x, y))

# Show the number of flights completed for each shuttle.
for index, shuttle in enumerate(shuttles):
    print("Shuttle %d has completed %d flights." % (index, shuttle.flights_completed))

print("\n")
# Show the distance from the first shuttle to all other shuttles.
first_shuttle = shuttles[0]
for index, shuttle in enumerate(shuttles):
    distance = first_shuttle.get_distance(shuttle)
    print("The first shuttle is %f units away from shuttle %d." % (distance, index))

print("\n")
# Show the distance from the first shuttle to all other rockets.
for index, rocket in enumerate(rockets):
   ...


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
This prevents some name conflicts. If you were reading carefully however, you might have noticed that the variable name rocket in the previous example had to be changed because it has the same name as the module itself. This is not good, because in a longer program that could mean a lot of renaming.
import module_name as local_module_name
There is another syntax for imports that is quite useful:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
Using the from module_name import * syntax:
Can you write Python code for it?

from multiplying import *

print(double(5))
print(triple(5))
print(quadruple(5))


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
When a new class is based on an existing class, you write the name of the parent class in parentheses when you define the new class:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
When you are importing a module into one of your projects, you are free to choose any name you want for the module in your project. So the last example could be rewritten in a way that the variable name rocket would not need to be changed:
Can you write Python code for it?

# Save as rocket_game.py
import rocket as rocket_module

rocket = rocket_module.Rocket()
print("The rocket is at (%d, %d)." % (rocket.x, rocket.y))

shuttle = rocket_module.Shuttle()
print("\nThe shuttle is at (%d, %d)." % (shuttle.x, shuttle.y))
print("The shuttle has completed %d flights." % shuttle.flights_completed)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
You can prove that each rocket has its own x and y values by moving just one of the rockets:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self):
        # Each rocket has an (x,y) position.
        self.x = 0
        self.y = 0

    def move_up(self):
        # Increment the y-position of the rocket.
        self.y += 1

# Create a fleet of 5 rockets, and store them in a list.
my_rockets = [Rocket() for x in range(0,5)]

# Move the first rocket up.
my_rockets[0].move_up()

# Show that only the first rocket has moved.
for rocket in my_rockets:
    print("Rocket altitude:", rocket.y)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
from module_name import *
There is one more import syntax that you should be aware of, but you should probably avoid using. This syntax imports all of the available classes and functions in a module:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
leads to much more readable code than something like:
Can you write Python code for it?


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
top
Method Resolution Order (mro)
Can you write Python code for it?

class A:
    def __init__(self, a):
        self.a = a

class GreatB:

    def greetings(self):
        print('Greetings from Type: ', self.__class__)

class B(GreatB):
    def __init__(self, b):
        self.b = b

class C(A,B):
    def __init__(self, a, b):
        A.__init__(self, a)
        B.__init__(self, b)

print('MRO: ', C.mro())
c = C('A', 'B')
print('c.a: ', c.a)
print('c.b: ', c.b)

c.greetings()
super(C, c).greetings()
super(B, c).greetings()


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
top
Adding a new method
One of the strengths of object-oriented programming is the ability to closely model real-world phenomena by adding appropriate attributes and behaviors to classes. One of the jobs of a team piloting a rocket is to make sure the rocket does not get too close to any other rockets. Let's add a method that will report the distance from one rocket to any other rocket.
If you are not familiar with distance calculations, there is a fairly simple formula to tell the distance between two points if you know the x and y values of each point.
This new method performs that calculation, and then returns the resulting distance.
Can you write Python code for it?

from math import sqrt

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_rocket(self, x_increment=0, y_increment=1):
        # Move the rocket according to the paremeters given.
        #  Default behavior is to move the rocket up one unit.
        self.x += x_increment
        self.y += y_increment

    def get_distance(self, other_rocket):
        # Calculates the distance from this rocket to another rocket,
        #  and returns that value.
        distance = sqrt((self.x-other_rocket.x)**2+(self.y-other_rocket.y)**2)
        return distance

# Make two rockets, at different places.
rocket_0 = Rocket()
rocket_1 = Rocket(10,5)

# Show the distance between them.
distance = rocket_0.get_distance(rocket_1)
print("The rockets are %f units apart." % distance)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
top
Inheritance
One of the most important goals of the object-oriented approach to programming is the creation of stable, reliable, reusable code. If you had to create a new class for every kind of object you wanted to model, you would hardly have any reusable code.
In Python and any other language that supports OOP, one class can inherit from another class. This means you can base a new class on an existing class; the new class inherits all of the attributes and behavior of the class it is based on.
A new class can override any undesirable attributes or behavior of the class it inherits from, and it can add any new attributes or behavior that are appropriate. The original class is called the parent class, and the new class is a child of the parent class. The parent class is also called a superclass, and the child class is also called a subclass.
The child class inherits all attributes and behavior from the parent class, but any attributes that are defined in the child class are not available to the parent class. This may be obvious to many people, but it is worth stating.
This also means a child class can override behavior of the parent class. If a child class defines a method that also appears in the parent class, objects of the child class will use the new method rather than the parent class method.
To better understand inheritance, let's look at an example of a class that can be based on the Rocket class.
The SpaceShuttle class
If you wanted to model a space shuttle, you could write an entirely new class. But a space shuttle is just a special kind of rocket. Instead of writing an entirely new class, you can inherit all of the attributes and behavior of a Rocket, and then add a few appropriate attributes and behavior for a Shuttle.
One of the most significant characteristics of a space shuttle is that it can be reused. So the only difference we will add at this point is to record the number of flights the shutttle has completed. Everything else you need to...
Can you write Python code for it?

from math import sqrt

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_rocket(self, x_increment=0, y_increment=1):
        # Move the rocket according to the paremeters given.
        #  Default behavior is to move the rocket up one unit.
        self.x += x_increment
        self.y += y_increment

    def get_distance(self, other_rocket):
        # Calculates the distance from this rocket to another rocket,
        #  and returns that value.
        distance = sqrt((self.x-other_rocket.x)**2+(self.y-other_rocket.y)**2)
        return distance

class Shuttle(Rocket):
    # Shuttle simulates a space shuttle, which is really
    #  just a reusable rocket.

    def __init__(self, x=0, y=0, flights_completed=0):
        super().__init__(x, y)
        self.flights_completed = flights_completed

shuttle = Shuttle(10,0,3)
print(shuttle)


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
top
Accepting paremeters in a method
The __init__ method is just a special method that serves a particular purpose, which is to help create new objects from a class. Any method in a class can accept parameters of any kind. With this in mind, the move_up() method can be made much more flexible. By accepting keyword arguments, the move_up() method can be rewritten as a more general move_rocket() method.
This new method will allow the rocket to be moved any amount, in any direction:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_rocket(self, x_increment=0, y_increment=1):
        # Move the rocket according to the paremeters given.
        #  Default behavior is to move the rocket up one unit.
        self.x += x_increment
        self.y += y_increment


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
top
Modules and classes
Now that you are starting to work with classes, your files are going to grow longer. This is good, because it means your programs are probably doing more interesting things. But it is bad, because longer files can be more difficult to work with. Python allows you to save your classes in another file and then import them into the program you are working on. This has the added advantage of isolating your classes into files that can be used in any number of different programs. As you use your classes repeatedly, the classes become more reliable and complete overall.
Storing a single class in a module
When you save a class into a separate file, that file is called a module. You can have any number of classes in a single module. There are a number of ways you can then import the class you are interested in.
Start out by saving just the Rocket class into a file called rocket.py. Notice the naming convention being used here: the module is saved with a lowercase name, and the class starts with an uppercase letter.
This convention is pretty important for a number of reasons, and it is a really good idea to follow the convention.
Can you write Python code for it?

# Save as rocket.py
from math import sqrt

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self, x=0, y=0):
        # Each rocket has an (x,y) position.
        self.x = x
        self.y = y

    def move_rocket(self, x_increment=0, y_increment=1):
        # Move the rocket according to the paremeters given.
        #  Default behavior is to move the rocket up one unit.
        self.x += x_increment
        self.y += y_increment

    def get_distance(self, other_rocket):
        # Calculates the distance from this rocket to another rocket,
        #  and returns that value.
        distance = sqrt((self.x-other_rocket.x)**2+(self.y-other_rocket.y)**2)
        return distance


I am working on the file "08 Classes and OOP.ipynb".
The first task is:
top
Object-Oriented terminology
Classes are part of a programming paradigm called object-oriented programming. Object-oriented programming, or OOP for short, focuses on building reusable blocks of code called classes. When you want to use a class in one of your programs, you make an object from that class, which is where the phrase "object-oriented" comes from. Python itself is not tied to object-oriented programming, but you will be using objects in most or all of your Python projects. In order to understand classes, you have to understand some of the language that is used in OOP.
General terminology
A class is a body of code that defines the attributes and behaviors required to accurately model something you need for your program. You can model something from the real world, such as a rocket ship or a guitar string, or you can model something from a virtual world such as a rocket in a game, or a set of physical laws for a game engine.
An attribute is a piece of information. In code, an attribute is just a variable that is part of a class.
A behavior is an action that is defined within a class. These are made up of methods, which are just functions that are defined for the class.
An object is a particular instance of a class. An object has a certain set of values for all of the attributes (variables) in the class. You can have as many objects as you want for any one class.
There is much more to know, but these words will help you get started. They will make more sense as you see more examples, and start to use classes on your own.
top
A closer look at the Rocket class
Now that you have seen a simple example of a class, and have learned some basic OOP terminology, it will be helpful to take a closer look at the Rocket class.
The __init()__ method
Here is the initial code block that defined the Rocket class:
Can you write Python code for it?

class Rocket():
    # Rocket simulates a rocket ship for a game,
    #  or a physics simulation.

    def __init__(self):
        # Each rocket has an (x,y) position.
        self.x = 0
        self.y = 0


I am working on the file "08 Overfitting Tutorial.ipynb".
The first task is:
Overfitting
You should notice that in the logs above, after around Epoch 15, the model begins to overfit. Even though the cost on the training set continues to decrease, the validation loss flattens (even increasing slightly). We can visualize these effects using the code below.
Note: The same plots can be created using our nvis command line utility (see: http://neon.nervanasys.com/docs/latest/tools.html)
Can you write Python code for it?

from neon.visualizations.figure import cost_fig, hist_fig, deconv_summary_page
from neon.visualizations.data import h5_cost_data, h5_hist_data, h5_deconv_data
from bokeh.plotting import output_notebook, show

cost_data = h5_cost_data('data.h5', False)
output_notebook()
show(cost_fig(cost_data, 400, 800, epoch_axis=False))


I am working on the file "08 Overfitting Tutorial.ipynb".
The first task is:
This situation illustrates the importance of plotting the validation loss (blue) in addition to the training cost (red). The training cost may mislead the user into thinking that model is continuing to perform well, but we can see from the validation loss that the model has begun to overfit.
Dropout layers
To correct overfitting, we introduce Dropout layers to the model, as shown below. Dropout layers randomly silence a subset of units for each minibatch, and are an effective means of preventing overfitting.
Can you write Python code for it?

layers = [
          Conv((3, 3, 64), **convp1),
          Conv((3, 3, 64), **convp1s2),
          Dropout(keep=.5),   # Added Dropout
          Conv((3, 3, 128), **convp1),
          Conv((3, 3, 128), **convp1s2),
          Dropout(keep=.5),   # Added Dropout
          Conv((3, 3, 128), **convp1),
          Conv((1, 1, 128), **conv),
          Conv((1, 1, 16), **conv),
          Pooling(8, op="avg"),
          Activation(Softmax())]

cost = GeneralizedCost(costfunc=CrossEntropyMulti())

mlp = Model(layers=layers)

# configure callbacks
callbacks = Callbacks(mlp, output_file='data.h5', eval_set=valid_set, eval_freq=1)

print "Training"
mlp.fit(train_set, optimizer=opt_gdm, num_epochs=num_epochs, cost=cost, callbacks=callbacks)

print('Misclassification error = %.1f%%' % (mlp.eval(valid_set, metric=Misclassification())*100))


I am working on the file "08 Overfitting Tutorial.ipynb".
The first task is:
We then plot the results of the training run below.
Can you write Python code for it?


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Analyze Data
Can you write Python code for it?

# show basic statistics
df.describe()

# Select a column
df["sepal length"].head()

# select columns
df[["sepal length", "petal width"]].head()

# select rows by name
df.loc[5:10]

# select rows by position
df.iloc[5:10]

# select rows by condition
df[df["sepal length"] > 5.0]


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Bar plots
Can you write Python code for it?

titanic = sns.load_dataset("titanic")
sns.barplot(x="sex", y="survived", hue="class", data=titanic)


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Boxplots
Can you write Python code for it?

sns.boxplot(x="day", y="total_bill", hue="time", data=tips)


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Create Data
The data set will consist of 5 baby names and the number of births recorded for that year (1880).
Can you write Python code for it?

# The inital set of baby names and bith rates
names = ['Bob','Jessica','Mary','John','Mel']
births = [968, 155, 77, 578, 973]


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Get Data
To pull in the csv file, we will use the pandas function read_csv. Let us take a look at this function and what inputs it takes.
Can you write Python code for it?


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Hexbin plot
Can you write Python code for it?

sns.jointplot(x="x", y="y", data=df, kind="hex")


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
If we wanted to give the columns specific names, we would have to pass another paramter called names. We can also omit the header parameter.
Can you write Python code for it?

df = pd.read_csv("births1880.csv", names=['Names','Births'])
df


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
It is also possible to read in a csv file by passing an url address
Here we use the famous Iris dataset.
The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals.
Can you write Python code for it?

df = pd.read_csv("https://raw.githubusercontent.com/bigmlcom/bigmler/master/data/iris.csv")
df.head(10)


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Kernel density estimation
Can you write Python code for it?

sns.jointplot(x="x", y="y", data=df, kind="kde")


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Plotting bivariate distributions
Can you write Python code for it?

mean, cov = [0, 1], [(1, .5), (.5, 1)]
data = np.random.multivariate_normal(mean, cov, 200)
df = pd.DataFrame(data, columns=["x", "y"])


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Plotting univariate distributions
distplot() function will draw a histogram and fit a kernel density estimate
Can you write Python code for it?

import numpy as np
x = np.random.normal(size=100)
sns.distplot(x)

import random
x = [random.normalvariate (0, 1) for i in range (0, 1000)]
sns.distplot (x)


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
The plot clearly shows that this is not a good model.
Let's try to fit a polynomial regression model with degree 2.
Can you write Python code for it?

sns.regplot(x="x", y="y", data=anscombe[anscombe["dataset"] == "II"], order=2)


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
To merge these two lists together we will use the zip function.
Can you write Python code for it?

BabyDataSet = zip(names,births)
BabyDataSet


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
Visualizing pairwise relationships in a dataset
To plot multiple pairwise bivariate distributions in a dataset, you can use the pairplot() function. This creates a matrix of axes and shows the relationship for each pair of columns in a DataFrame. by default, it also draws the univariate distribution of each variable on the diagonal Axes:
Can you write Python code for it?

iris = sns.load_dataset("iris")
sns.pairplot(iris)

# we can add colors to different species
sns.pairplot(iris, hue="species")


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
We are basically done creating the data set. We now will use the pandas library to export this data set into a csv file.
We will create a DataFrame object. You can think of this object holding the contents of the BabyDataSet in a format similar to an excel spreadsheet.
Can you write Python code for it?

df = pd.DataFrame(data = BabyDataSet, columns=['Names', 'Births'])
df


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
We can get the maximum sepal length by
Can you write Python code for it?


I am working on the file "08--pandas-seaborn.ipynb".
The first task is:
We can use the function regplot to show the linear relationship between total_bill and tip.
It also shows the 95% confidence interval.
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
And here is the assigment category:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
As you might expect, this is a corpus of IMDB movie reviews. Someone went through and read each review, labeling it as either "positive" or "negative". The task we have before us is to create a model that can accurately predict whether a never-before-seen review is positive or negative. This is analogous to Underwood and Sellers looking at whether a poem volume was reviewed or randomly selected.
From the movie_reviews object let's take out the reviews and the judgement:
Can you write Python code for it?

reviews = [movie_reviews.raw(fileid) for fileid in movie_reviews.fileids()]
judgements = [movie_reviews.categories(fileid)[0] for fileid in movie_reviews.fileids()]


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Before we go any further, sklearn likes our data in a very specific format. The X must be in an array of arrays, each sub array is an observation. Because we only have one independent variable, we'll have sub arrays of len 1. We can do that with the reshape method:
Can you write Python code for it?

X = demo_tb['Study_Hours'].reshape(-1,1)
X


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Classification of Textual Data
How can we translate this simple model of binary classification to text? I'm going to leave the more complicated model that Underwood and Sellers use for the next notebook if you're interested, today we're just going to work through the basic classfication pipeline. We'll download a pre-made corpus from nltk:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Do you consider this a positive or negative review? Let's see what the human annotator said:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
For the Linear Regression, sklearn returns an R-squared from the score method. The R-squared tells us how much of the variation in the data can be explained by our model, .559 isn't that bad, but obviously more goes into your Grade than just Study_Hours.
Nevertheless we can still predict a grade just like we did above to create that line, let's say I studied for 5 hours:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
How many training documents are there?
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
How might this curve be used for a binary classification task?
Logistic Classification
That's great, so we can begin to see how we might use such a model to conduct binary classification. In this task, we want to get a number of study hours as an observation, and place it in one of two bins: pass or fail.
To create the model though, we have to train it on the data we have. In machine learning, we also need to put some data aside as "testing data" so that we don't bias our model by using it in the training process. In Python we often see X_train, y_train and X_test, y_test:
Can you write Python code for it?

X_train = demo_tb.column('Study_Hours')[:-2]
y_train = demo_tb.column('Pass')[:-2]

X_test = demo_tb.column('Study_Hours')[-2:]
y_test = demo_tb.column('Pass')[-2:]


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
I've copied the cell from above below. Try playing with the parameters to these objects and see if you can improve the cross_val_score for the model.
Can you write Python code for it?

text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))),
                    ('tfidf', TfidfTransformer()),
                    ('clf', LogisticRegression(random_state=0))
                     ])

scores = cross_val_score(text_clf, X, y, cv=5)

print(scores, np.mean(scores))


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
If you don't believe me that all we did is reassign and shuffle:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
In the example above, we're interested in Study_Hours and Grade. This is a natural "input" "output" situation. To plot the regression line, or best-fit, we can feed in fit_line=True to the scatter method:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Intro to Regression
A popular classification model is logistic regression. This is what Underwood and Sellers use in their article to classify whether a text was reviewed or randomly selected from HathiTrust. Today we'll look at the difference between regression and classification tasks, and how we can use a logistic regression model to classify text like Underwood and Sellers. We won't have time to go through their full code, but if you're interested I've provided a walk-through in the second notebook.
To explore the regression model let's first create some dummy data:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Intuiting the Linear Regression Model
You may have encountered linear regression in previous coursework of yours. Linear regression, in its simple form, tries to model the relationship between two continous variables as a straight line. It interprets one variable as the input, and the other as the output.:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Let's see the observations we're setting aside for later:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Let's see what categories they have:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Maybe I should study more?
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Now we'll fit our model again but only on the _train data, and get out the $\beta$ coefficients:
Can you write Python code for it?

lr.fit(X_train.reshape(-1,1),y_train.reshape(len(y_train),))
B0, B1 = lr.intercept_[0], lr.coef_[0]


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Prediction
We can also use our model to classify new reviews, all we have to do is extract the tfidf features from the raw text and send them to the model as our features (independent variables):
Can you write Python code for it?

new_bad_review = "This movie really sucked. I can't believe how long it dragged on. The actors are absolutely terrible. They should rethink their career paths"

features = tfidf.transform([new_bad_review])

model.predict(features)

new_good_review = "I loved this film! The cinematography was incredible, and Leonardo Dicarpio is flawless. Super cute BTW."

features = tfidf.transform([new_good_review])

model.predict(features)


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
So this means:
$GRADE_i= 42.897229302892598 + 5.9331153718275509 * STUDYHOURS + \epsilon_i$
As a linear regression this is simple to interpret. To get our grade score, we take the number of study hours and multipy it by 5.9331153718275509 then we add 42.897229302892598 and that's our prediction.
If we look at our chart again but using the model we just made, that looks about right:
Can you write Python code for it?

y_pred = linreg.predict(X)
print(X)
print(y_pred)
plt.scatter(X, y)
plt.plot(X, y_pred)


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
The list of documents (strings) is in the .data property, we can access the first one like so:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
The sklearn built-in methods can make this predict process faster:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
The subset parameter will give you training and testing data. You can also use the categories parameter to choose only certain categories.
If we wanted to get the training data for sci.electronics and rec.autos we would write this:
Can you write Python code for it?

train = fetch_20newsgroups(subset="train", categories=['sci.electronics', 'rec.autos'])


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Then we can plot the curve just like we did earlier, and we'll add our points:
Can you write Python code for it?

xmin, xmax = 0,10
xlist = [float(x)/int(1e4) for x in range(xmin*int(1e4), xmax*int(1e4))]
ylist = [logistic(B0 + B1*x) for x in xlist]
plt.plot(xlist,ylist)

# add our "observed" data points
plt.scatter(demo_tb['Study_Hours'],demo_tb['Pass'])


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
To get meaningful independent variables (words) we have to do some processing too (think DTM!). With sklearn's text pipelines, we can quickly build a text a classifier in only a few lines of Python:
Can you write Python code for it?

LogisticRegression?

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split

text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))),
                    ('tfidf', TfidfTransformer()),
                    ('clf', LogisticRegression(random_state=0, penalty='l2', C=1000))
                     ])

scores = cross_val_score(text_clf, X, y, cv=5)

print(scores, np.mean(scores))


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Trigram:
Can you write Python code for it?

ngs = ngrams("Text analysis is so cool. I can really see why classification can be a valuable tool.".split(), 3)
list(ngs)


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We can do the same for the testing data:
Can you write Python code for it?

test = fetch_20newsgroups(subset="test", categories=['sci.electronics', 'rec.autos'])
test.data[0]

test.target[0]

len(test.data)


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We can evaluate how great our model is with the score method. We need to give it the X and observed y values, and it will predict its own y values and compare:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We can get back the intercept_ and $\beta$ coef_ with attributes of the linreg object:
Can you write Python code for it?

B0, B1 = linreg.intercept_, linreg.coef_[0]
B0, B1


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We can get those $\beta$ coefficients back out from sklearn for our grade data:
Can you write Python code for it?

B0, B1 = lr.intercept_[0], lr.coef_[0][0]
B0, B1


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We can send these coefficients back into the logistic function we wrote earlier to get the probability that a student would pass given our X_test values:
Can you write Python code for it?

fitted = [logistic(B1*th + B0) for th in X_test]
fitted


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We'll also need to assign a couple $\beta$ coefficients for the intercept and variable just like we saw in linear regression:
Can you write Python code for it?


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
We'll reshape our arrays again too, since we know how sklearn likes them:
Can you write Python code for it?

X = demo_tb['Study_Hours'].reshape(-1,1)
y = demo_tb['Pass'].reshape(len(demo_tb['Pass']),)
X, y


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Why do you think your score improved (or didn't)?

BONUS (not assigned)
We're going to download the 20 Newsgroups, a widely used corpus for demos of general texts:

The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.

First we'll import the data from sklearn:
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
You get the point. This helps us combat this "bag of words" idea, but doesn't completely save us. For our purposes here, just as we counted the frequency of individual words, we've added counting the frequency of groups of 2s and 3s.

Important Features
After we train the model we can then index the tfidf matrix for the words with the most significant coefficients (remember independent variables!) to get the most helpful features:
Can you write Python code for it?

feature_names = tfidf.get_feature_names()
top10pos = np.argsort(model.coef_[0])[-10:]
print("Top features for positive reviews:")
print(list(feature_names[j] for j in top10pos))
print()
print("Top features for negative reviews:")
top10neg = np.argsort(model.coef_[0])[:10]
print(list(feature_names[j] for j in top10neg))


I am working on the file "08-Classification/01-Classification.ipynb".
The first task is:
Your output, or dependent variable, is just one array with no sub arrays.
Can you write Python code for it?

y = demo_tb['Grade'].reshape(len(demo_tb['Grade']),)
y


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Classification
Can you write Python code for it?

# Train model using full set of 'reviewed' and 'random' texts

# Use this to predict the probability that other prestigious texts
# (i.e. ones that we haven't trained on) might have been reviewed
# ...if they had been published! The new texts include, for example,
# Emily Dickinson and Gerard Manley Hopkins.

# Re-run script from scratch

%pylab inline
matplotlib.style.use('ggplot')

from datascience import *
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction import DictVectorizer

corpus_path = 'poems/'

# Read metadata from spreadsheet
metadata_tb = Table.read_table('poemeta.csv', keep_default_na=False)

# We'll copy just our new texts into a separate table as well, for later
canon_tb = metadata_tb.where('recept','addcanon')

# Read Term Frequencies from files

freqdict_list = []

# Iterate through texts in our spreadsheet
for _id in metadata_tb['docid']:

    # Each text will have its own dictionary
    # Keys are terms and values are frequencies
    termfreq_dict = {}

    # Open the given text's spreadsheet
    with open(corpus_path+_id+'.poe.tsv', encoding='utf-8') as file_in:
        filelines = file_in.readlines()

        # Each line in the spreadsheet contains a unique term and its frequency
        for line in filelines:
            termfreq = line.split('\t')

            # 'If' conditions throw out junk lines in the spreadsheet
            if len(termfreq) > 2 or len(termfreq) > 2:
                continue
            term, freq = termfreq[0], int(termfreq[1])
            if len(term)>0 and term[0].isalpha():

                # Create new entry in text's dictionary for the term
                termfreq_dict[term] = freq

    freqdict_list.append(termfreq_dict)

# Create the Document-Term-Matrix

dv = DictVectorizer()
dtm = dv.fit_transform(freqdict_list)
term_list = dv.feature_names_

# Place the DTM into a Pandas DataFrame for further manipulation

dtm_df = pd.DataFrame(dtm.toarray(),...


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Document Frequency
Can you write Python code for it?

# Create new DataFrame that simply lists whether a term appears in
# each document, so that we don't have to repeat this process evey iteration

term_in_doc_df = dtm_df>0

term_in_doc_df

# Re-write the model-building function

def set_author_aside(author, tb, dtm_df_, dfreq_df_):
    train_ids = tb.where(tb['author']!=author).column('docid')
    test_ids = tb.where(tb['author']==author).column('docid')

    train_df_ = dtm_df_.loc[train_ids]
    dfreq_df_ = dfreq_df_.loc[train_ids] # Include only term_in_doc values for texts in training set
    test_df_ = dtm_df_.loc[test_ids]

    train_targets_ = tb.where(tb['author']!=author)['recept']=='reviewed'

    return train_df_, test_df_, train_targets_, dfreq_df_

# Re-write our vocabulary selection function

def top_vocab_by_docfreq(df, num_words):
    # Removed the test of whether a term is in a given document (i.e. df>0)
    wordcolumn_sums = sum(df)
    words_by_freq = wordcolumn_sums.sort_values(ascending=False)
    top_words = words_by_freq[:num_words]
    top_words_list = top_words.index.tolist()

    return top_words_list


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Evaluation
Can you write Python code for it?

# Visualize the probability each text was reviewed

colors = ['r' if recept=='reviewed' else 'b'  for recept in clf_tb['recept']]

clf_tb.scatter('firstpub', 'P(reviewed)', c=colors, fit_line=True)

# Does the Logistic Regression Model think its likely each book was reviewed?
predictions = probabilities>0.5
predictions

from sklearn.metrics import accuracy_score

# Creates array where '1' indicates a reviewed book and '0' indicates not
targets = clf_tb['recept']=='reviewed'

print(accuracy_score(predictions, targets))

# Note: Often we prefer to evaluate accuracy based on the F1-score, which
# weighs the number of times we correctly predicted reviewed texts against
# the number of times we incorrectly predicted them as 'random'.

from sklearn.metrics import f1_score

print(f1_score(predictions, targets))

## EX. Change the regularization parameter ('C') in our Logistic Regression function.
##     How does this change the classifier's accuracy?

## EX. Reduce the size of the vocabulary used for classification. How does accuracy change?

##  Q. Are there cases when we might not want to set the classification threshold
##     to 50% likelihood? How certain are we that 51% is different from a 49% probability?


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Inputs and Outputs
Underwood and Sellers create a unique model for each author in the corpus. They set aside a given author from the training set and then use the model to predict whether she was likely to be reviewed or not. Create a list of authors and an "empty" array in which to record probabilities.
Can you write Python code for it?

authors = list(set(clf_tb['author']))
probabilities = np.zeros([len(clf_tb['docid'])])


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Let's add in the docid as the index instead of just a counter for the row:
Can you write Python code for it?


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Now let's select only columns for words in our pre-processed vocabulary. This will make our computation more efficient later:
Can you write Python code for it?


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Parallel Processing
Can you write Python code for it?

# Parallel Processing means running our script on multiple cores simultaneously
# This can be used in situations where we might otherwise use a 'FOR' loop
# (when it doesn't matter what order we go through the list of values!)

clf = LogisticRegression(C = 0.00007)

def master_function(author):
    # Note: Our only input is the name of the author.
    # Remember that we had iterated over the list of authors previously.
    train_df, test_df, train_targets, dfreq_df = set_author_aside(author, clf_tb, dtm_df, term_in_doc_df)
    vocab_list = top_vocab_by_docfreq(dfreq_df, 3200)
    train_df, test_df = normalize_model(train_df, test_df, vocab_list)
    clf.fit(train_df, train_targets)

    # Create a list of each text's probability of review AND its index in the metadata table
    index_probability_tuples = []
    for _id in test_df.index.tolist():
        text = test_df.loc[_id]
        probability = clf.predict_proba([text])[0][1]
        _index = list(clf_tb.column('docid')).index(_id)
        index_probability_tuples.append( (_index, probability) )
    return index_probability_tuples

# Multiprocessing enables Python to parallelize

import multiprocessing

# Return number of cores
multiprocessing.cpu_count()

# By default, the Pool contains one worker for each core

# Since we are working on a shared server, we'll set the number
# of workers to 4.

pool = multiprocessing.Pool(4, maxtasksperchild=1)

# Efficiently applies the master_function() to our list of authors
# Returns a list where each entry is an item returned by the function

# Timing the process again
start = time.time()

output = pool.map(master_function, authors)

end = time.time()
print(end-start)

output[:10]

# In this case, each element in output is itself a list ('index_probability_tuples'),
# the length of which is the number of texts by a given author. We'll flatten it for
# ease of use.

flat_output = [tup  for lst in output  for tup in lst]
flat_output[:10]

# Use the indices returned with the output to arrange probabilities...


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
So how long is this going to take us?
Can you write Python code for it?


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
Training and Prediction
The cell below will build a model for just 1 author, let's see how long it takes:
Can you write Python code for it?

import time
start = time.time()

for author in authors[:1]:

    # Set aside each author's texts from training set
    train_df, test_df, train_targets = set_author_aside(author, clf_tb, dtm_df)

    # Retrieve the most common words (by document frequency) for a given model
    vocab_list = top_vocab_by_docfreq(train_df, 3200)

    # Normalize the model's term frequencies and put them into standard units
    train_df, test_df = normalize_model(train_df, test_df, vocab_list)

    # Learn the Logistic Regression over our model
    clf.fit(train_df, train_targets)

    # Some authors have more than one text in the corpus, so we retrieve all
    for _id in test_df.index.tolist():

        # Make prediction whether text was reviewed
        text = test_df.loc[_id]
        probability = clf.predict_proba([text])[0][1]

        # Record predictions in same order as the metadata spreadsheet
        _index = list(clf_tb.column('docid')).index(_id)
        probabilities[_index] = probability

end = time.time()
print(end - start)