I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Even better using name scopes:
Can you write Python code for it?

reset_graph()

def relu(X):
    with tf.name_scope("relu"):
        w_shape = (int(X.get_shape()[1]), 1)                          # not shown in the book
        w = tf.Variable(tf.random_normal(w_shape), name="weights")    # not shown
        b = tf.Variable(0.0, name="bias")                             # not shown
        z = tf.add(tf.matmul(X, w), b, name="z")                      # not shown
        return tf.maximum(z, 0., name="max")                          # not shown

n_features = 3
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X) for i in range(5)]
output = tf.add_n(relus, name="output")

save_graph(run_id="relu2")

%tensorboard --logdir {root_logdir}


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Exercise solutions
1. to 11.
See appendix A.
12. Logistic Regression with Mini-Batch Gradient Descent using TensorFlow
First, let's create the moons dataset using Scikit-Learn's make_moons() function:
Can you write Python code for it?

from sklearn.datasets import make_moons

m = 1000
X_moons, y_moons = make_moons(m, noise=0.1, random_state=42)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Extra material
Can you write Python code for it?

reset_graph()

with tf.variable_scope("my_scope"):
    x0 = tf.get_variable("x", shape=(), initializer=tf.constant_initializer(0.))
    x1 = tf.Variable(0., name="x")
    x2 = tf.Variable(0., name="x")

with tf.variable_scope("my_scope", reuse=True):
    x3 = tf.get_variable("x")
    x4 = tf.Variable(0., name="x")

with tf.variable_scope("", default_name="", reuse=True):
    x5 = tf.get_variable("my_scope/x")

print("x0:", x0.op.name)
print("x1:", x1.op.name)
print("x2:", x2.op.name)
print("x3:", x3.op.name)
print("x4:", x4.op.name)
print("x5:", x5.op.name)
print(x0 is x3 and x3 is x5)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Feeding data to the training algorithm
Placeholder nodes
Can you write Python code for it?

reset_graph()

A = tf.placeholder(tf.float32, shape=(None, 3))
B = A + 5
with tf.Session() as sess:
    B_val_1 = B.eval(feed_dict={A: [[1, 2, 3]]})
    B_val_2 = B.eval(feed_dict={A: [[4, 5, 6], [7, 8, 9]]})

print(B_val_1)

print(B_val_2)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Great! Now that the data is ready to be fed to the model, we need to build that model. Let's start with a simple implementation, then we will add all the bells and whistles.
First let's reset the default graph.
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
How could you find the partial derivatives of the following function with regards to a and b?
Can you write Python code for it?

def my_func(a, b):
    z = 0
    for i in range(100):
        z = a * np.cos(z + i) + z * np.sin(b - i)
    return z

my_func(0.2, 0.3)

reset_graph()

a = tf.Variable(0.2, name="a")
b = tf.Variable(0.3, name="b")
z = tf.constant(0.0, name="z0")
for i in range(100):
    z = a * tf.cos(z + i) + z * tf.sin(b - i)

grads = tf.gradients(z, [a, b])
init = tf.global_variables_initializer()


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
However, the actual graph data may still be in the OS's file cache, so we need to flush() or close() the FileWriter to be sure that it's well written to disk:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
If you want to have a saver that loads and restores theta with a different name, such as "weights":
Can you write Python code for it?

saver = tf.train.Saver({"weights": theta})


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
In fact, let's create a function that will generate such a subdirectory path every time we need one:
Can you write Python code for it?

def make_log_subdir(run_id=None):
    if run_id is None:
        run_id = datetime.utcnow().strftime("%Y%m%d%H%M%S")
    return "{}/run-{}/".format(root_logdir, run_id)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Let's check:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Let's create a little function to get the name of the log directory to save the summaries for Tensorboard:
Can you write Python code for it?

from datetime import datetime

def log_dir(prefix=""):
    now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
    root_logdir = "tf_logs"
    if prefix:
        prefix += "-"
    name = prefix + "run-" + now
    return "{}/{}/".format(root_logdir, name)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Let's plot these predictions to see what they look like:
Can you write Python code for it?

y_pred_idx = y_pred.reshape(-1) # a 1D array rather than a column vector
plt.plot(X_test[y_pred_idx, 1], X_test[y_pred_idx, 2], 'go', label="Positive")
plt.plot(X_test[~y_pred_idx, 1], X_test[~y_pred_idx, 2], 'r^', label="Negative")
plt.legend()
plt.show()


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Let's see if it works:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Let's take a peek at the dataset:
Can you write Python code for it?

plt.plot(X_moons[y_moons == 1, 0], X_moons[y_moons == 1, 1], 'go', label="Positive")
plt.plot(X_moons[y_moons == 0, 0], X_moons[y_moons == 0, 1], 'r^', label="Negative")
plt.legend()
plt.show()


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Linear Regression
Using the Normal Equation
Can you write Python code for it?

import numpy as np
from sklearn.datasets import fetch_california_housing

reset_graph()

housing = fetch_california_housing()
m, n = housing.data.shape
housing_data_plus_bias = np.c_[np.ones((m, 1)), housing.data]

X = tf.constant(housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
XT = tf.transpose(X)
theta = tf.matmul(tf.matmul(tf.matrix_inverse(tf.matmul(XT, X)), XT), y)

with tf.Session() as sess:
    theta_value = theta.eval()

theta_value


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Looks good. Now let's reshape y_train to make it a column vector (i.e. a 2D array with a single column):
Can you write Python code for it?

y_moons_column_vector = y_moons.reshape(-1, 1)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Managing graphs
Can you write Python code for it?

reset_graph()

x1 = tf.Variable(1)
x1.graph is tf.get_default_graph()

graph = tf.Graph()
with graph.as_default():
    x2 = tf.Variable(2)

x2.graph is graph

x2.graph is tf.get_default_graph()

w = tf.constant(3)
x = w + 2
y = x + 5
z = x * 3

with tf.Session() as sess:
    print(y.eval())  # 10
    print(z.eval())  # 15

with tf.Session() as sess:
    y_val, z_val = sess.run([y, z])
    print(y_val)  # 10
    print(z_val)  # 15


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Manually computing the gradients
Can you write Python code for it?

reset_graph()

n_epochs = 1000
learning_rate = 0.01

X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")
gradients = 2/m * tf.matmul(tf.transpose(X), error)
training_op = tf.assign(theta, theta - learning_rate * gradients)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        if epoch % 100 == 0:
            print("Epoch", epoch, "MSE =", mse.eval())
        sess.run(training_op)

    best_theta = theta.eval()

best_theta


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Mini-batch Gradient Descent
Can you write Python code for it?

n_epochs = 1000
learning_rate = 0.01

reset_graph()

X = tf.placeholder(tf.float32, shape=(None, n + 1), name="X")
y = tf.placeholder(tf.float32, shape=(None, 1), name="y")

theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

n_epochs = 10

batch_size = 100
n_batches = int(np.ceil(m / batch_size))

def fetch_batch(epoch, batch_index, batch_size):
    np.random.seed(epoch * n_batches + batch_index)  # not shown in the book
    indices = np.random.randint(m, size=batch_size)  # not shown
    X_batch = scaled_housing_data_plus_bias[indices] # not shown
    y_batch = housing.target.reshape(-1, 1)[indices] # not shown
    return X_batch, y_batch

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        for batch_index in range(n_batches):
            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})

    best_theta = theta.eval()

best_theta


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Modularity
An ugly flat code:
Can you write Python code for it?

reset_graph()

n_features = 3
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")

w1 = tf.Variable(tf.random_normal((n_features, 1)), name="weights1")
w2 = tf.Variable(tf.random_normal((n_features, 1)), name="weights2")
b1 = tf.Variable(0.0, name="bias1")
b2 = tf.Variable(0.0, name="bias2")

z1 = tf.add(tf.matmul(X, w1), b1, name="z1")
z2 = tf.add(tf.matmul(X, w2), b2, name="z2")

relu1 = tf.maximum(z1, 0., name="relu1")
relu2 = tf.maximum(z1, 0., name="relu2")  # Oops, cut&paste error! Did you spot it?

output = tf.add(relu1, relu2, name="output")


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Much better, using a function to build the ReLUs:
Can you write Python code for it?

reset_graph()

def relu(X):
    w_shape = (int(X.get_shape()[1]), 1)
    w = tf.Variable(tf.random_normal(w_shape), name="weights")
    b = tf.Variable(0.0, name="bias")
    z = tf.add(tf.matmul(X, w), b, name="z")
    return tf.maximum(z, 0., name="relu")

n_features = 3
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X) for i in range(5)]
output = tf.add_n(relus, name="output")

file_writer = tf.summary.FileWriter("logs/relu1", tf.get_default_graph())


reset_graph()

def relu(X):
    w_shape = (int(X.get_shape()[1]), 1)
    w = tf.Variable(tf.random_normal(w_shape), name="weights")
    b = tf.Variable(0.0, name="bias")
    z = tf.add(tf.matmul(X, w), b, name="z")
    return tf.maximum(z, 0., name="relu")

n_features = 3
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X) for i in range(5)]
output = tf.add_n(relus, name="output")

save_graph(run_id="relu1")

%tensorboard --logdir {root_logdir}


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Name scopes
Can you write Python code for it?

reset_graph()

n_epochs = 1000
learning_rate = 0.01

X = tf.placeholder(tf.float32, shape=(None, n + 1), name="X")
y = tf.placeholder(tf.float32, shape=(None, 1), name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")

with tf.name_scope("loss") as scope:
    error = y_pred - y
    mse = tf.reduce_mean(tf.square(error), name="mse")

optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

mse_summary = tf.summary.scalar('MSE', mse)

logdir = make_log_subdir()
file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())

n_epochs = 10
batch_size = 100
n_batches = int(np.ceil(m / batch_size))

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        for batch_index in range(n_batches):
            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)
            if batch_index % 10 == 0:
                summary_str = mse_summary.eval(feed_dict={X: X_batch, y: y_batch})
                step = epoch * n_batches + batch_index
                file_writer.add_summary(summary_str, step)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})

    best_theta = theta.eval()

file_writer.flush()
file_writer.close()
print("Best theta:")
print(best_theta)

print(error.op.name)

print(mse.op.name)

reset_graph()

a1 = tf.Variable(0, name="a")      # name == "a"
a2 = tf.Variable(0, name="a")      # name == "a_1"

with tf.name_scope("param"):       # name == "param"
    a3 = tf.Variable(0, name="a")  # name == "param/a"

with tf.name_scope("param"):       # name == "param_1"
    a4 = tf.Variable(0, name="a")  # name == "param_1/a"

for node in (a1, a2, a3, a4):
    print(node.op.name)


reset_graph()

now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
root_logdir = "tf_logs"
logdir = "{}/run-{}/".format(root_logdir, now)

n_epochs = 1000
learning_rate = 0.01

X = tf.placeholder(tf.float32, shape=(None, n + 1), name="X")
y = tf.placeholder(tf.float32, shape=(None, 1), name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")

with tf.name_scope("loss") as scope:
    error = y_pred - y
    mse = tf.reduce_mean(tf.square(error), name="mse")

optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

mse_summary = tf.summary.scalar('MSE', mse)
file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())

n_epochs = 10
batch_size = 100
n_batches = int(np.ceil(m / batch_size))

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        for batch_index in range(n_batches):
            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)
            if batch_index % 10 == 0:
                summary_str = mse_summary.eval(feed_dict={X: X_batch, y: y_batch})
                step = epoch * n_batches + batch_index
                file_writer.add_summary(summary_str, step)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})

    best_theta = theta.eval()

file_writer.flush()
file_writer.close()
print("Best theta:")
print(best_theta)

print(error.op.name)

print(mse.op.name)

reset_graph()

a1 = tf.Variable(0, name="a")      # name == "a"
a2 = tf.Variable(0, name="a")      # name == "a_1"

with tf.name_scope("param"):       # name == "param"
    a3 = tf.Variable(0, name="a")  # name == "param/a"

with tf.name_scope("param"):       # name == "param_1"
    a4 = tf.Variable(0, name="a")  # name == "param_1/a"

for node in (a1, a2, a3, a4):
    print(node.op.name)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Next, let's create the graph, using the logistic_regression() function. We will also create the FileWriter to save the summaries to the log directory for Tensorboard:
Can you write Python code for it?

n_inputs = 2 + 4
logdir = log_dir("logreg")

X = tf.placeholder(tf.float32, shape=(None, n_inputs + 1), name="X")
y = tf.placeholder(tf.float32, shape=(None, 1), name="y")

y_proba, loss, training_op, loss_summary, init, saver = logistic_regression(X, y)

file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Next, let's use the %tensorboard extension to start the TensorBoard server. We need to point it to the root log directory:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Note: we don't use the epoch number when generating batches, so we could just have a single for loop rather than 2 nested for loops, but it's convenient to think of training time in terms of number of epochs (i.e., roughly the number of times the algorithm went through the training set).
For each instance in the test set, y_proba_val contains the estimated probability that it belongs to the positive class, according to the model. For example, here are the first 5 estimated probabilities:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now let's define the logistic_regression() function to create the graph. We will leave out the definition of the inputs X and the targets y. We could include them here, but leaving them out will make it easier to use this function in a wide range of use cases (e.g. perhaps we will want to add some preprocessing steps for the inputs before we feed them to the Logistic Regression model).
Can you write Python code for it?

def logistic_regression(X, y, initializer=None, seed=42, learning_rate=0.01):
    n_inputs_including_bias = int(X.get_shape()[1])
    with tf.name_scope("logistic_regression"):
        with tf.name_scope("model"):
            if initializer is None:
                initializer = tf.random_uniform([n_inputs_including_bias, 1], -1.0, 1.0, seed=seed)
            theta = tf.Variable(initializer, name="theta")
            logits = tf.matmul(X, theta, name="logits")
            y_proba = tf.sigmoid(logits)
        with tf.name_scope("train"):
            loss = tf.losses.log_loss(y, y_proba, scope="loss")
            optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
            training_op = optimizer.minimize(loss)
            loss_summary = tf.summary.scalar('log_loss', loss)
        with tf.name_scope("init"):
            init = tf.global_variables_initializer()
        with tf.name_scope("save"):
            saver = tf.train.Saver()
    return y_proba, loss, training_op, loss_summary, init, saver


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now let's look at TensorBoard again. Note that this will reuse the existing TensorBoard server since we're reusing the same root log directory:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now let's look at TensorBoard. Try going to the SCALARS tab:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now let's save the default graph to our log subdirectory using tf.summary.FileWriter():
Can you write Python code for it?

file_writer = tf.summary.FileWriter(logdir, graph=tf.get_default_graph())


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now let's split the data into a training set and a test set:
Can you write Python code for it?

test_ratio = 0.2
test_size = int(m * test_ratio)
X_train = X_moons_with_bias[:-test_size]
X_test = X_moons_with_bias[-test_size:]
y_train = y_moons_column_vector[:-test_size]
y_test = y_moons_column_vector[-test_size:]


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now that's much, much better! Apparently the new features really helped a lot.
Let's open tensorboard, find the latest run and look at the learning curve:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now that's much, much better! Apparently the new features really helped a lot.
Try starting the tensorboard server, find the latest run and look at the learning curve (i.e., how the loss evaluated on the test set evolves as a function of the epoch number):
$ tensorboard --logdir=tf_logs
Now you can play around with the hyperparameters (e.g. the batch_size or the learning_rate) and run training again and again, comparing the learning curves. You can even automate this process by implementing grid search or randomized search. Below is a simple implementation of a randomized search on both the batch size and the learning rate. For the sake of simplicity, the checkpoint mechanism was removed.
Can you write Python code for it?

from scipy.stats import reciprocal

n_search_iterations = 10

for search_iteration in range(n_search_iterations):
    batch_size = np.random.randint(1, 100)
    learning_rate = reciprocal(0.0001, 0.1).rvs(random_state=search_iteration)

    n_inputs = 2 + 4
    logdir = log_dir("logreg")

    print("Iteration", search_iteration)
    print("  logdir:", logdir)
    print("  batch size:", batch_size)
    print("  learning_rate:", learning_rate)
    print("  training: ", end="")

    reset_graph()

    X = tf.placeholder(tf.float32, shape=(None, n_inputs + 1), name="X")
    y = tf.placeholder(tf.float32, shape=(None, 1), name="y")

    y_proba, loss, training_op, loss_summary, init, saver = logistic_regression(
        X, y, learning_rate=learning_rate)

    file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())

    n_epochs = 10001
    n_batches = int(np.ceil(m / batch_size))

    final_model_path = "./my_logreg_model_%d" % search_iteration

    with tf.Session() as sess:
        sess.run(init)

        for epoch in range(n_epochs):
            for batch_index in range(n_batches):
                X_batch, y_batch = random_batch(X_train_enhanced, y_train, batch_size)
                sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            loss_val, summary_str = sess.run([loss, loss_summary], feed_dict={X: X_test_enhanced, y: y_test})
            file_writer.add_summary(summary_str, epoch)
            if epoch % 500 == 0:
                print(".", end="")

        saver.save(sess, final_model_path)

        print()
        y_proba_val = y_proba.eval(feed_dict={X: X_test_enhanced, y: y_test})
        y_pred = (y_proba_val >= 0.5)

        print("  precision:", precision_score(y_test, y_pred))
        print("  recall:", recall_score(y_test, y_pred))


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Now you can play around with the hyperparameters (e.g. the batch_size or the learning_rate) and run training again and again, comparing the learning curves. You can even automate this process by implementing grid search or randomized search. Below is a simple implementation of a randomized search on both the batch size and the learning rate. For the sake of simplicity, the checkpoint mechanism was removed.
Can you write Python code for it?

from scipy.stats import reciprocal

n_search_iterations = 10

for search_iteration in range(n_search_iterations):
    batch_size = np.random.randint(1, 100)
    learning_rate = reciprocal(0.0001, 0.1).rvs(random_state=search_iteration)

    n_inputs = 2 + 4
    logdir = log_dir("logreg")

    print("Iteration", search_iteration)
    print("  logdir:", logdir)
    print("  batch size:", batch_size)
    print("  learning_rate:", learning_rate)
    print("  training: ", end="")

    reset_graph()

    X = tf.placeholder(tf.float32, shape=(None, n_inputs + 1), name="X")
    y = tf.placeholder(tf.float32, shape=(None, 1), name="y")

    y_proba, loss, training_op, loss_summary, init, saver = logistic_regression(
        X, y, learning_rate=learning_rate)

    file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())

    n_epochs = 10001
    n_batches = int(np.ceil(m / batch_size))

    final_model_path = "./my_logreg_model_%d" % search_iteration

    with tf.Session() as sess:
        sess.run(init)

        for epoch in range(n_epochs):
            for batch_index in range(n_batches):
                X_batch, y_batch = random_batch(X_train_enhanced, y_train, batch_size)
                sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            loss_val, summary_str = sess.run([loss, loss_summary], feed_dict={X: X_test_enhanced, y: y_test})
            file_writer.add_summary(summary_str, epoch)
            if epoch % 500 == 0:
                print(".", end="")

        saver.save(sess, final_model_path)

        print()
        y_proba_val = y_proba.eval(feed_dict={X: X_test_enhanced, y: y_test})
        y_pred = (y_proba_val >= 0.5)

        print("  precision:", precision_score(y_test, y_pred))
        print("  recall:", recall_score(y_test, y_pred))

file_writer.close()


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Numerical differentiation
Can you write Python code for it?

def gradients(func, vars_list, eps=0.0001):
    partial_derivatives = []
    base_func_eval = func.evaluate()
    for var in vars_list:
        original_value = var.value
        var.value = var.value + eps
        tweaked_func_eval = func.evaluate()
        var.value = original_value
        derivative = (tweaked_func_eval - base_func_eval) / eps
        partial_derivatives.append(derivative)
    return partial_derivatives

df_dx, df_dy = gradients(f, [x, y])
print("df/dx(3,4) =", df_dx)
print("df/dy(3,4) =", df_dy)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Ok, next let's reset the default graph:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Ok, now let's create a small function to generate training batches. In this implementation we will just pick random instances from the training set for each batch. This means that a single batch may contain the same instance multiple times, and also a single epoch may not cover all the training instances (in fact it will generally cover only about two thirds of the instances). However, in practice this is not an issue and it simplifies the code:
Can you write Python code for it?

def random_batch(X_train, y_train, batch_size):
    rnd_indices = np.random.randint(0, len(X_train), batch_size)
    X_batch = X_train[rnd_indices]
    y_batch = y_train[rnd_indices]
    return X_batch, y_batch


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Okay, now let's start TensorBoard! It runs as a web server in a separate process, so we first need to start it. One way to do that is to run the tensorboard command in a terminal window. Another is to use the %tensorboard Jupyter extension, which takes care of starting TensorBoard, and it allows us to view TensorBoard's user interface directly within Jupyter. Let's load this extension now:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Once again, we can make predictions by just classifying as positive all the instances whose estimated probability is greater or equal to 0.5:
Can you write Python code for it?

y_pred = (y_proba_val >= 0.5)

precision_score(y_test, y_pred)

recall_score(y_test, y_pred)

y_pred_idx = y_pred.reshape(-1) # a 1D array rather than a column vector
plt.plot(X_test[y_pred_idx, 1], X_test[y_pred_idx, 2], 'go', label="Positive")
plt.plot(X_test[~y_pred_idx, 1], X_test[~y_pred_idx, 2], 'r^', label="Negative")
plt.legend()
plt.show()


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Saving and restoring a model
Can you write Python code for it?

reset_graph()

n_epochs = 1000                                                                       # not shown in the book
learning_rate = 0.01                                                                  # not shown

X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name="X")            # not shown
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")            # not shown
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")                                      # not shown
error = y_pred - y                                                                    # not shown
mse = tf.reduce_mean(tf.square(error), name="mse")                                    # not shown
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)            # not shown
training_op = optimizer.minimize(mse)                                                 # not shown

init = tf.global_variables_initializer()
saver = tf.train.Saver()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        if epoch % 100 == 0:
            print("Epoch", epoch, "MSE =", mse.eval())                                # not shown
            save_path = saver.save(sess, "/tmp/my_model.ckpt")
        sess.run(training_op)

    best_theta = theta.eval()
    save_path = saver.save(sess, "/tmp/my_model_final.ckpt")

best_theta

with tf.Session() as sess:
    saver.restore(sess, "/tmp/my_model_final.ckpt")
    best_theta_restored = theta.eval() # not shown in the book

np.allclose(best_theta, best_theta_restored)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Sharing Variables
Sharing a threshold variable the classic way, by defining it outside of the relu() function then passing it as a parameter:
Can you write Python code for it?

reset_graph()

def relu(X, threshold):
    with tf.name_scope("relu"):
        w_shape = (int(X.get_shape()[1]), 1)                        # not shown in the book
        w = tf.Variable(tf.random_normal(w_shape), name="weights")  # not shown
        b = tf.Variable(0.0, name="bias")                           # not shown
        z = tf.add(tf.matmul(X, w), b, name="z")                    # not shown
        return tf.maximum(z, threshold, name="max")

threshold = tf.Variable(0.0, name="threshold")
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X, threshold) for i in range(5)]
output = tf.add_n(relus, name="output")

reset_graph()

def relu(X):
    with tf.name_scope("relu"):
        if not hasattr(relu, "threshold"):
            relu.threshold = tf.Variable(0.0, name="threshold")
        w_shape = int(X.get_shape()[1]), 1                          # not shown in the book
        w = tf.Variable(tf.random_normal(w_shape), name="weights")  # not shown
        b = tf.Variable(0.0, name="bias")                           # not shown
        z = tf.add(tf.matmul(X, w), b, name="z")                    # not shown
        return tf.maximum(z, relu.threshold, name="max")

X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X) for i in range(5)]
output = tf.add_n(relus, name="output")

reset_graph()

with tf.variable_scope("relu"):
    threshold = tf.get_variable("threshold", shape=(),
                                initializer=tf.constant_initializer(0.0))

with tf.variable_scope("relu", reuse=True):
    threshold = tf.get_variable("threshold")

with tf.variable_scope("relu") as scope:
    scope.reuse_variables()
    threshold = tf.get_variable("threshold")

reset_graph()

def relu(X):
    with tf.variable_scope("relu", reuse=True):
        threshold = tf.get_variable("threshold")
        w_shape = int(X.get_shape()[1]), 1                          # not shown
        w = tf.Variable(tf.random_normal(w_shape), name="weights")  # not shown
        b =...


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
The moons dataset has two input features, since each instance is a point on a plane (i.e., 2-Dimensional):
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
This is what the "enhanced" training set looks like:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
This means that you can import a pretrained model without having to have the corresponding Python code to build the graph. This is very handy when you keep tweaking and saving your model: you can load a previously saved model without having to search for the version of the code that built it.
Visualizing the graph
TensorBoard is a great tool to visualize TensorFlow graphs, training curves, and much more. Our TensorFlow code will write various files in a log directory, and the TensorBoard server will regularly read these files and produce nice interactive visualizations. It can plot graphs, learning curves (i.e., how the loss evaluated on the training set or test set evolves as a function of the epoch number), profiling data to identify performance bottlenecks, and more. In short, it helps keep track of everything. Here's the overall picture:
TensorFlow writes logs to ===> log directory ===> TensorBoard reads data and displays visualizations
If we want to visualize different graphs, or learning curves for different training runs, we don't want the log files to get all mixed up. So we will need one log subdirectory per graph, or per run. Let's use a root log directory that we will call tf_logs, and a sub-directory that we will call run- followed by the current timestamp (you can use any other name you prefer in your own code):
Can you write Python code for it?

from datetime import datetime

now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
root_logdir = "tf_logs"
logdir = "{}/run-{}/".format(root_logdir, now)

logdir


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
This means that you can import a pretrained model without having to have the corresponding Python code to build the graph. This is very handy when you keep tweaking and saving your model: you can load a previously saved model without having to search for the version of the code that built it.
Visualizing the graph
inside Jupyter
Can you write Python code for it?

from IPython.display import clear_output, Image, display, HTML

def strip_consts(graph_def, max_const_size=32):
    """Strip large constant values from graph_def."""
    strip_def = tf.GraphDef()
    for n0 in graph_def.node:
        n = strip_def.node.add()
        n.MergeFrom(n0)
        if n.op == 'Const':
            tensor = n.attr['value'].tensor
            size = len(tensor.tensor_content)
            if size > max_const_size:
                tensor.tensor_content = b"<stripped %d bytes>"%size
    return strip_def

def show_graph(graph_def, max_const_size=32):
    """Visualize TensorFlow graph."""
    if hasattr(graph_def, 'as_graph_def'):
        graph_def = graph_def.as_graph_def()
    strip_def = strip_consts(graph_def, max_const_size=max_const_size)
    code = """

        <link rel="import" href="https://tensorboard.appspot.com/tf-graph-basic.build.html" onload=load()>

          <tf-graph-basic id="{id}"></tf-graph-basic>

    """.format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))

    iframe = """

    """.format(code.replace('"', '"'))
    display(HTML(iframe))

show_graph(tf.get_default_graph())


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
To classify each instance, we can go for maximum likelihood: classify as positive any instance whose estimated probability is greater or equal to 0.5:
Can you write Python code for it?

y_pred = (y_proba_val >= 0.5)
y_pred[:5]


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Using Batch Gradient Descent
Gradient Descent requires scaling the feature vectors first. We could do this using TF, but let's just use Scikit-Learn for now.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaled_housing_data = scaler.fit_transform(housing.data)
scaled_housing_data_plus_bias = np.c_[np.ones((m, 1)), scaled_housing_data]

print(scaled_housing_data_plus_bias.mean(axis=0))
print(scaled_housing_data_plus_bias.mean(axis=1))
print(scaled_housing_data_plus_bias.mean())
print(scaled_housing_data_plus_bias.shape)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Using TensorBoard
Can you write Python code for it?

reset_graph()

from datetime import datetime

now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
root_logdir = "tf_logs"
logdir = "{}/run-{}/".format(root_logdir, now)

n_epochs = 1000
learning_rate = 0.01

X = tf.placeholder(tf.float32, shape=(None, n + 1), name="X")
y = tf.placeholder(tf.float32, shape=(None, 1), name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

mse_summary = tf.summary.scalar('MSE', mse)
file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())

n_epochs = 10
batch_size = 100
n_batches = int(np.ceil(m / batch_size))

with tf.Session() as sess:                                                        # not shown in the book
    sess.run(init)                                                                # not shown

    for epoch in range(n_epochs):                                                 # not shown
        for batch_index in range(n_batches):
            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)
            if batch_index % 10 == 0:
                summary_str = mse_summary.eval(feed_dict={X: X_batch, y: y_batch})
                step = epoch * n_batches + batch_index
                file_writer.add_summary(summary_str, step)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})

    best_theta = theta.eval()                                                     # not shown

file_writer.close()

best_theta


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Using a GradientDescentOptimizer
Can you write Python code for it?

reset_graph()

n_epochs = 1000
learning_rate = 0.01

X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")

optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        if epoch % 100 == 0:
            print("Epoch", epoch, "MSE =", mse.eval())
        sess.run(training_op)

    best_theta = theta.eval()

print("Best theta:")
print(best_theta)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Using a momentum optimizer
Can you write Python code for it?

reset_graph()

n_epochs = 1000
learning_rate = 0.01

X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")

optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,
                                       momentum=0.9)

training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        sess.run(training_op)

    best_theta = theta.eval()

print("Best theta:")
print(best_theta)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Using autodiff
Same as above except for the gradients = ... line:
Can you write Python code for it?

reset_graph()

n_epochs = 1000
learning_rate = 0.01

X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")

gradients = tf.gradients(mse, [theta])[0]

training_op = tf.assign(theta, theta - learning_rate * gradients)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        if epoch % 100 == 0:
            print("Epoch", epoch, "MSE =", mse.eval())
        sess.run(training_op)

    best_theta = theta.eval()

print("Best theta:")
print(best_theta)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
We must not forget to add an extra bias feature ($x_0 = 1$) to every instance. For this, we just need to add a column full of 1s on the left of the input matrix $\mathbf{X}$:
Can you write Python code for it?

X_moons_with_bias = np.c_[np.ones((m, 1)), X_moons]


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Well, that looks pretty bad, doesn't it? But let's not forget that the Logistic Regression model has a linear decision boundary, so this is actually close to the best we can do with this model (unless we add more features, as we will show in a second).
Now let's start over, but this time we will add all the bells and whistles, as listed in the exercise:
* Define the graph within a logistic_regression() function that can be reused easily.
* Save checkpoints using a Saver at regular intervals during training, and save the final model at the end of training.
* Restore the last checkpoint upon startup if training was interrupted.
* Define the graph using nice scopes so the graph looks good in TensorBoard.
* Add summaries to visualize the learning curves in TensorBoard.
* Try tweaking some hyperparameters such as the learning rate or the mini-batch size and look at the shape of the learning curve.
Before we start, we will add 4 more features to the inputs: ${x_1}^2$, ${x_2}^2$, ${x_1}^3$ and ${x_2}^3$. This was not part of the exercise, but it will demonstrate how adding features can improve the model. We will do this manually, but you could also add them using sklearn.preprocessing.PolynomialFeatures.
Can you write Python code for it?

X_train_enhanced = np.c_[X_train,
                         np.square(X_train[:, 1]),
                         np.square(X_train[:, 2]),
                         X_train[:, 1] ** 3,
                         X_train[:, 2] ** 3]
X_test_enhanced = np.c_[X_test,
                        np.square(X_test[:, 1]),
                        np.square(X_test[:, 2]),
                        X_test[:, 1] ** 3,
                        X_test[:, 2] ** 3]


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
Deploy model to Vertex AI
This will take several minutes.  Bash in Jupyter doesn't provide intermediate output.
You can check progress by visiting https://console.cloud.google.com/vertex-ai/endpoints
and https://console.cloud.google.com/vertex-ai/models
Can you write Python code for it?

%%bash
# note TF_VERSION and ENDPOINT_NAME set in 1st cell
# TF_VERSION=2-6
# ENDPOINT_NAME=flights

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
MODEL_NAME=${ENDPOINT_NAME}-${TIMESTAMP}
EXPORT_PATH=$(gsutil ls ${OUTDIR}/export | tail -1)
echo $EXPORT_PATH

if [[ $(gcloud ai endpoints list --region=$REGION \
        --format='value(DISPLAY_NAME)' --filter=display_name=${ENDPOINT_NAME}) ]]; then
    echo "Endpoint for $MODEL_NAME already exists"
else
    # create model
    echo "Creating Endpoint for $MODEL_NAME"
    gcloud ai endpoints create --region=${REGION} --display-name=${ENDPOINT_NAME}
fi

ENDPOINT_ID=$(gcloud ai endpoints list --region=$REGION \
              --format='value(ENDPOINT_ID)' --filter=display_name=${ENDPOINT_NAME})
echo "ENDPOINT_ID=$ENDPOINT_ID"

# delete any existing models with this name
for MODEL_ID in $(gcloud ai models list --region=$REGION --format='value(MODEL_ID)' --filter=display_name=${MODEL_NAME}); do
    echo "Deleting existing $MODEL_NAME ... $MODEL_ID "
    gcloud ai models delete --region=$REGION $MODEL_ID
done

# upload model
gcloud beta ai models upload --region=$REGION --display-name=$MODEL_NAME \
     --container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.${TF_VERSION}:latest \
     --artifact-uri=$EXPORT_PATH
MODEL_ID=$(gcloud ai models list --region=$REGION --format='value(MODEL_ID)' --filter=display_name=${MODEL_NAME})
echo "MODEL_ID=$MODEL_ID"

# deploy model to endpoint
gcloud ai endpoints deploy-model $ENDPOINT_ID \
  --region=$REGION \
  --model=$MODEL_ID \
  --display-name=$MODEL_NAME \
  --machine-type=n1-standard-2 \
  --min-replica-count=1 \
  --max-replica-count=1 \
  --traffic-split=0=100

%%writefile example_input.json
{"instances": [
  {"dep_hour": 2, "is_weekday": 1, "dep_delay": 40, "taxi_out": 17, "distance": 41, "carrier": "AS", "dep_airport_lat": 58.42527778, "dep_airport_lon": -135.7075, "arr_airport_lat": 58.35472222, "arr_airport_lon": -134.57472222, "origin": "GST", "dest": "JNU"},
  {"dep_hour": 22, "is_weekday": 0, "dep_delay": -7,...


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
Explanations
Add explainability
Can you write Python code for it?

%%bash
model_dir=$(gsutil ls ${OUTDIR}/export | tail -1)
echo $model_dir
saved_model_cli show --tag_set serve --signature_def serving_default --dir $model_dir

cols = ('dep_delay,taxi_out,distance,dep_hour,is_weekday,' +
        'dep_airport_lat,dep_airport_lon,' +
        'arr_airport_lat,arr_airport_lon,' +
        'carrier,origin,dest')
inputs = {x: {"inputTensorName": "{}".format(x)}
        for x in cols.split(',')}
expl = {
    "inputs": inputs,
    "outputs": {
    "pred": {
      "outputTensorName": "pred"
    }
  }
}
print(expl)
with open('explanation-metadata.json', 'w') as ofp:
    json.dump(expl, ofp, indent=2)

!cat explanation-metadata.json

%%bash
# note TF_VERSION set in 1st cell, but ENDPOINT_NAME is being changed
# TF_VERSION=2-6

ENDPOINT_NAME=flights_xai

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
MODEL_NAME=${ENDPOINT_NAME}-${TIMESTAMP}
EXPORT_PATH=$(gsutil ls ${OUTDIR}/export | tail -1)
echo $EXPORT_PATH

if [[ $(gcloud ai endpoints list --region=$REGION \
        --format='value(DISPLAY_NAME)' --filter=display_name=${ENDPOINT_NAME}) ]]; then
    echo "Endpoint for $MODEL_NAME already exists"
else
    # create model
    echo "Creating Endpoint for $MODEL_NAME"
    gcloud ai endpoints create --region=${REGION} --display-name=${ENDPOINT_NAME}
fi

ENDPOINT_ID=$(gcloud ai endpoints list --region=$REGION \
              --format='value(ENDPOINT_ID)' --filter=display_name=${ENDPOINT_NAME})
echo "ENDPOINT_ID=$ENDPOINT_ID"

# delete any existing models with this name
for MODEL_ID in $(gcloud ai models list --region=$REGION --format='value(MODEL_ID)' --filter=display_name=${MODEL_NAME}); do
    echo "Deleting existing $MODEL_NAME ... $MODEL_ID "
    gcloud ai models delete --region=$REGION $MODEL_ID
done

# upload model
gcloud beta ai models upload --region=$REGION --display-name=$MODEL_NAME \
     --container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.${TF_VERSION}:latest \
     --artifact-uri=$EXPORT_PATH \
     --explanation-method=sampled-shapley --explanation-path-count=10...


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
Export files that contain training, validation data
The training and validation dataset requires doing a join on several BigQuery tables. Because of this,
we'll need export the data we need to create new tables that are flattened (the BigQueryReader in
TensorFlow needs tables: see https://towardsdatascience.com/how-to-read-bigquery-data-from-tensorflow-2-0-efficiently-9234b69165c8). But if we are going to make a copy of the data, we might as well export the data to
Google Cloud Storage.
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE TABLE dsongcp.flights_train_data AS

SELECT
  IF(arr_delay < 15, 1.0, 0.0) AS ontime,
  dep_delay,
  taxi_out,
  distance,
  origin,
  dest,
  EXTRACT(hour FROM dep_time) AS dep_hour,
  IF (EXTRACT(dayofweek FROM dep_time) BETWEEN 2 AND 6, 1, 0) AS is_weekday,
  UNIQUE_CARRIER AS carrier,
  dep_airport_lat,
  dep_airport_lon,
  arr_airport_lat,
  arr_airport_lon
FROM dsongcp.flights_tzcorr f
JOIN dsongcp.trainday t
ON f.FL_DATE = t.FL_DATE
WHERE
  f.CANCELLED = False AND
  f.DIVERTED = False AND
  is_train_day = 'True'

%%bigquery
CREATE OR REPLACE TABLE dsongcp.flights_eval_data AS

SELECT
  IF(arr_delay < 15, 1.0, 0.0) AS ontime,
  dep_delay,
  taxi_out,
  distance,
  origin,
  dest,
  EXTRACT(hour FROM dep_time) AS dep_hour,
  IF (EXTRACT(dayofweek FROM dep_time) BETWEEN 2 AND 6, 1, 0) AS is_weekday,
  UNIQUE_CARRIER AS carrier,
  dep_airport_lat,
  dep_airport_lon,
  arr_airport_lat,
  arr_airport_lon
FROM dsongcp.flights_tzcorr f
JOIN dsongcp.trainday t
ON f.FL_DATE = t.FL_DATE
WHERE
  f.CANCELLED = False AND
  f.DIVERTED = False AND
  is_train_day = 'False'

%%bigquery
CREATE OR REPLACE TABLE dsongcp.flights_all_data AS

SELECT
  IF(arr_delay < 15, 1.0, 0.0) AS ontime,
  dep_delay,
  taxi_out,
  distance,
  origin,
  dest,
  EXTRACT(hour FROM dep_time) AS dep_hour,
  IF (EXTRACT(dayofweek FROM dep_time) BETWEEN 2 AND 6, 1, 0) AS is_weekday,
  UNIQUE_CARRIER AS carrier,
  dep_airport_lat,
  dep_airport_lon,
  arr_airport_lat,
  arr_airport_lon,
  IF (is_train_day = 'True',
      IF(ABS(MOD(FARM_FINGERPRINT(CAST(f.FL_DATE AS STRING)), 100)) < 60, 'TRAIN', 'VALIDATE'),
      'TEST') AS data_split
FROM dsongcp.flights_tzcorr f
JOIN dsongcp.trainday t
ON f.FL_DATE = t.FL_DATE
WHERE
  f.CANCELLED = False AND
  f.DIVERTED = False

%%bash
PROJECT=$(gcloud config get-value project)
for dataset in "train" "eval" "all"; do
  TABLE=dsongcp.flights_${dataset}_data
  CSV=gs://${BUCKET}/ch9/data/${dataset}.csv
  echo "Exporting ${TABLE} to ${CSV} and deleting table"
  bq...


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
Model Parameters
Can you write Python code for it?

NBUCKETS = 5 #
NEMBEDS = 3
TRAIN_BATCH_SIZE = 64
DNN_HIDDEN_UNITS = '64,32'


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
Setup in notebook
For development purposes, we'll train for just a few epochs.
That's why the NUM_EXAMPLES is so low.
Can you write Python code for it?

DEVELOP_MODE = True
NUM_EXAMPLES = 5000*1000 # doesn't need to be precise but get order of magnitude right.

# use wildcard in the name so that it works even if you have multiple files
training_data_uri = 'gs://{}/ch9/data/train*'.format(BUCKET)
validation_data_uri = 'gs://{}/ch9/data/eval*'.format(BUCKET)


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
Train the model and evaluate once in a while
Also checkpoint
Can you write Python code for it?

output_dir='gs://{}/ch9/trained_model'.format(BUCKET)
os.environ['OUTDIR'] = output_dir  # needed for deployment
print('Writing trained model to {}'.format(output_dir))

!gsutil -m rm -rf $OUTDIR

# Build a wide-and-deep model.
def wide_and_deep_classifier(inputs, linear_feature_columns, dnn_feature_columns, dnn_hidden_units):
    deep = tf.keras.layers.DenseFeatures(dnn_feature_columns, name='deep_inputs')(inputs)
    layers = [int(x) for x in dnn_hidden_units.split(',')]
    for layerno, numnodes in enumerate(layers):
        deep = tf.keras.layers.Dense(numnodes, activation='relu', name='dnn_{}'.format(layerno+1))(deep)
    wide = tf.keras.layers.DenseFeatures(linear_feature_columns, name='wide_inputs')(inputs)
    both = tf.keras.layers.concatenate([deep, wide], name='both')
    output = tf.keras.layers.Dense(1, activation='sigmoid', name='pred')(both)
    model = tf.keras.Model(inputs, output)
    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    return model

model = wide_and_deep_classifier(
    inputs,
    linear_feature_columns = sparse.values(),
    dnn_feature_columns = real.values(),
    dnn_hidden_units = DNN_HIDDEN_UNITS)
tf.keras.utils.plot_model(model, 'flights_model.png', show_shapes=False, rankdir='LR')


I am working on the file "09_vertexai/flights_model_tf2.ipynb".
The first task is:
You should see:
<img src="flights_model.png"/>
Can you write Python code for it?

# training and evaluation dataset
train_batch_size = TRAIN_BATCH_SIZE
if DEVELOP_MODE:
    eval_batch_size = 100
    steps_per_epoch = 3
    epochs = 2
    num_eval_examples = eval_batch_size*10
else:
    eval_batch_size = 100
    steps_per_epoch = NUM_EXAMPLES // train_batch_size
    epochs = 10
    num_eval_examples = eval_batch_size * 100
train_dataset = read_dataset(training_data_uri, train_batch_size)
eval_dataset = read_dataset(validation_data_uri, eval_batch_size, tf.estimator.ModeKeys.EVAL, num_eval_examples)

checkpoint_path = '{}/checkpoints/flights.cpt'.format(output_dir)
shutil.rmtree(checkpoint_path, ignore_errors=True)
cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1)

history = model.fit(train_dataset,
                    validation_data=eval_dataset,
                    epochs=epochs,
                    steps_per_epoch=steps_per_epoch,
                    callbacks=[cp_callback])

print(history.history.keys())

import matplotlib.pyplot as plt
nrows = 1
ncols = 2
fig = plt.figure(figsize=(10, 5))

for idx, key in enumerate(['loss', 'accuracy']):
    ax = fig.add_subplot(nrows, ncols, idx+1)
    plt.plot(history.history[key])
    plt.plot(history.history['val_{}'.format(key)])
    plt.title('model {}'.format(key))
    plt.ylabel(key)
    plt.xlabel('epoch')
    plt.legend(['train', 'validation'], loc='upper left');


I am working on the file "09a Sentiment Analysis with LSTM.ipynb".
The first task is:
Callbacks allow the model to report its progress during the course of training. Here we tell neon to save the model every epoch .
Can you write Python code for it?

from neon.callbacks import Callbacks

model_file = 'imdb_lstm.pkl'
callbacks = Callbacks(model, eval_set=valid_set, serialize=1, save_path=model_file)


I am working on the file "09a Sentiment Analysis with LSTM.ipynb".
The first task is:
Model Specification
For most of the layers, we randomly initialize the parameters either randomly uniform numbers or Xavier Glorot's initialization scheme.
Can you write Python code for it?

from neon.initializers import Uniform, GlorotUniform

init_glorot = GlorotUniform()
init_uniform = Uniform(-0.1/128, 0.1/128)


I am working on the file "09a Sentiment Analysis with LSTM.ipynb".
The first task is:
The network consists of sequential list of the following layers:

LookupTable is a word embedding that maps from a sparse one-hot representation to dense word vectors. The embedding is learned from the data.
LSTM is a recurrent layer with “long short-term memory” units. LSTM networks are good at learning temporal dependencies during training, and often perform better than standard RNN layers.
RecurrentSum is a recurrent output layer that collapses over the time dimension of the LSTM by summing outputs from individual steps.
Dropout performs regularization by silencing a random subset of the units during training.
Affine is a fully connected layer for the binary classification of the outputs.
Can you write Python code for it?

from neon.layers import LSTM, Affine, Dropout, LookupTable, RecurrentSum
from neon.transforms import Logistic, Tanh, Softmax
from neon.models import Model

layers = [
    LookupTable(vocab_size=20000, embedding_dim=128, init=init_uniform),
    LSTM(output_size=128, init=init_glorot, activation=Tanh(),
         gate_activation=Logistic(), reset_cells=True),
    RecurrentSum(),
    Dropout(keep=0.5),
    Affine(nout=2, init=init_glorot, bias=init_glorot, activation=Softmax())
]

# create model object
model = Model(layers=layers)


I am working on the file "09a Sentiment Analysis with LSTM.ipynb".
The first task is:
To train the model, we use neon's ArrayIterator object which will iterate over these numpy arrays, returning a minibatch of data with each call to pass to the model.
Can you write Python code for it?

from neon.data import ArrayIterator
import numpy as np
data['Y_train'] = np.array(data['Y_train'], dtype=np.int32)
data['Y_valid'] = np.array(data['Y_valid'], dtype=np.int32)

train_set = ArrayIterator(data['X_train'], data['Y_train'], nclass=2)
valid_set = ArrayIterator(data['X_valid'], data['Y_valid'], nclass=2)


I am working on the file "09a Sentiment Analysis with LSTM.ipynb".
The first task is:
Train model
To train the model, we call the fit() function and pass in the training set. Here we train for 2 epochs, meaning two complete passes through the dataset.
Can you write Python code for it?

model.fit(train_set, optimizer=optimizer, num_epochs=2,
          cost=cost, callbacks=callbacks)


I am working on the file "09a-DetectorModelling.ipynb".
The first task is:
Calculate the current through the detector for source and background irradiance using a simplified version of (Eq. 6.17):
$$
 i_{\cal R}=
A_{d} \widehat{\cal R}
\int_{0}^{\infty} E_{s\lambda}{\cal R}_\lambda\,
d\lambda,
$$
using the spectral responsivity calculated  above.  The following calculation is performed for both the radiant irradiance and photon irradiance, and should yield the same resulting current.
The detector area is 200 $\mu$m by 200 $\mu$m.
Can you write Python code for it?

# to calculate the detector currents
import scipy.constants as const
areaDet = (200e-6) ** 2   # detector area in m2

iSourceE = np.trapz(EsourceEL * areaDet * responsivity, wavelength)
iBkgE = np.trapz(EbkgEL * areaDet * responsivity, wavelength)
iSourceQ = np.trapz(EsourceQL * areaDet * quantumEffic * const.e,wavelength)
iBkgQ = np.trapz(EbkgQL * areaDet * quantumEffic * const.e, wavelength)
iTotalE = iSourceE + iBkgE
iTotalQ = iSourceQ + iBkgQ

print ("\nDetector current source        = {0:.3e} A and {1:.3e} A".\
    format(iSourceQ,iSourceE))
print ("Detector current background    = {0:.3e} A and {1:.3e} A".\
    format(iBkgQ,iBkgE))
print ("Detector current total         = {0:.3e} A and {1:.3e} A".\
    format(iTotalQ,iTotalE))


I am working on the file "09a-DetectorModelling.ipynb".
The first task is:
Calculate the current-voltage relationship using (Eq 5.128)
$$
I=
I_{\rm sat}
e^{qV/(kT\beta)}
-
I_{\rm sat}
 -I_{ph},
$$
The saturation current $I_{\rm sat}$ is calculated as explained  above.  The photocurrent is determined from (Eq. 5.4 and 5.5)
$$
{\cal R}{q\lambda}=\frac{i_d}{\Phi{q}}
=\left{\begin{array}{ll}
\eta q G
&{\rm for\ \ } \lambda \leq \lambda_c\
0.0
&{\rm for\ \ } \lambda >\lambda_c\
\end{array},
\right.
$$
and
$$
{\cal R}{e\lambda}
=\frac{i_d}{\Phi{e}}
= \frac{i_d}{\frac{hc}{\lambda}\Phi_{q}}
=\left{
\begin{array}{ll}
\frac{\eta q\lambda G}{hc}
&{\rm for\ \ } \lambda \leq \lambda_c\
0.0
&{\rm for\ \ } \lambda >\lambda_c\
\end{array}
\right.
$$
The value of $\beta$ is determined as explained in (Sec. 5.9.2).  The value for $\beta=1.7$ is used here because InSb operation is dominated by g-r current (for references see my book).
The current-voltage relationship can be calculated for different background flux values, such as 'dark' ($I_{ph}=0$), or for different irradiance values, such as those calculated above.
The following figure shows the current-voltage relationship for three cases: (a) the dark current (i.e., no background flux), (b) hemispherical background flux at 280 K, and (c) the test source flux.
Note that the y axis is logarithmic to show detail: negative values are shown positive because the logarithm of a negative number is not defined.
Note that the dark current I-V curve is close to the background current I-V curve, while the test source curve is shifted down by almost two orders of magnitude; this is caused by the photon current resulting from the test source flux.
Can you write Python code for it?

# to plot the IV curves
IVbeta = 1.7     # 1 = dominanr diffusion current, 2 = g-r dominant
Vbias = np.linspace(-200e-3, 100e-3, 200) # bias voltage range
ixvDark = rydetector.IXV(Vbias, IVbeta, tempDet, 0, Isat)
ixvBackgnd = rydetector.IXV(Vbias, IVbeta, tempDet, iBkgE, Isat)
ixvTotalE = rydetector.IXV(Vbias, IVbeta, tempDet, iTotalE, Isat)
IXVplot = ryplot.Plotter(1,2,1,figsize=(9,8))
IXVplot.semilogY(1,Vbias,np.abs(ixvDark) * 1e6, plotCol=['g'],label=['Dark'])
IXVplot.semilogY(1,Vbias,np.abs(ixvBackgnd) * 1e6, plotCol=['r'],label=['Background'])
IXVplot.semilogY(1,Vbias,np.abs(ixvTotalE) * 1e6,'IxV Curve',\
    'Voltage [V]','|Current| [uA]',plotCol=['b'],label=['Test source'])
IXVplot.plot(2,Vbias,ixvDark * 1e6, plotCol=['g'],label=['Dark'])
IXVplot.plot(2,Vbias,ixvBackgnd * 1e6, plotCol=['r'],label=['Background'])
IXVplot.plot(2,Vbias,ixvTotalE * 1e6,'IxV Curve',\
    'Voltage [V]','|Current| [uA]',plotCol=['b'],label=['Test source']);


I am working on the file "09a-DetectorModelling.ipynb".
The first task is:
Calculate the spectral photon and radiant irradiance on the detector for a hemispherical background blackbody source with unity emissivity and temperature of 280 K.  The calculation is done as above but for an hemispherical solid angle.
Can you write Python code for it?

# to prepare for detector background current calculation
tempBkg = 280.0              # background temperature in K
emisBkg = 1.0                # background emissivity
solidAngBkg = np.pi       # background is hemisphere

print ("Background temperature         = {0} K".format(tempBkg))
print ("Background emissivity          = {0} ".format(emisBkg))
print ("Background solid angle         = {0} sr".format(solidAngBkg))

#spectral irradiance for test setup, for both source and background
# in units of photon rate q/(s.m2)
EbkgQL = (emisBkg * ryplanck.planck(wavelength, tempBkg, 'ql') * \
    solidAngBkg ) / (np.pi )
#in radiant units W/m2
EbkgEL = (emisBkg * ryplanck.planck(wavelength, tempBkg, 'el') * \
    solidAngBkg) / (np.pi )


I am working on the file "09a-DetectorModelling.ipynb".
The first task is:
The spectral absorption coefficient is calculated by using (Eq 5.78) and (Eq 5.79)
$$
\alpha_{\nu} \propto \sqrt{h\nu-E_g} \approx \alpha_{\lambda_c} +  \alpha_0\sqrt{h\nu-E_g} \;\; {\rm for} \;\; h\nu>E_g.
$$
and
$$
\alpha_{\nu} = \alpha_{\lambda_c}\exp\left[\frac{h\nu-E_g}{kT}\right] \;\; {\rm for} \;\; h\nu<E_g.
$$
the data in (Table A.6).
The quantum efficiency is calculated by using (Eq 5.87 and 5.88)
$$
\frac{J}{q E_q}=\eta=(1-\rho)(1-e^{- \alpha d})
$$
and
$$
\rho=
\left(
\frac{n_2-n_1}{n_2+n_1}
\right)^2,
$$
with a refractive index of 3.42 for the detector and 1.0 for air, a depletion layer thickness of 5 $\mu$m, and the spectral absorption coefficient calculated above.
The spectral responsivity is calculated using the spectral quantum efficiency and (Eq. 5.5)
$$
{\cal R}{e\lambda}
=\frac{i_d}{\Phi{e}}
= \frac{i_d}{\frac{hc}{\lambda}\Phi_{q}}
=\left{
\begin{array}{ll}
\frac{\eta q\lambda G}{hc}
&{\rm for\ \ } \lambda \leq \lambda_c\
0.0
&{\rm for\ \ } \lambda >\lambda_c\
\end{array}
\right.
$$
The gain of the photovoltaic detector is 1.0.
Can you write Python code for it?

# to define detector parameters
# wavelength in micrometers, remember to scale down in functions.
wavelength = np.linspace(1, 5.5, 200)

a0 = 1.9e4 * 100   # absorption coefficient [m-1], Eq3.5 & 3.6 Dereniak
a0p = 800 * 100    # absorption coefficient [m-1] Eq3.5 & 3.6 Dereniak
d1 = 0                # start of the detector depletion layer in meter
d2 = 5e-6             # end of the detector depletion layer in meter
n1 = 1.0              # refraction index of the air
n2 = 3.42        # refraction index of the semiconductor being analyzed
theta1 = 0.0        # radiation incident angle in radians

absorption = rydetector.Absorption(wavelength / 1e6, Eg, tempDet, a0, a0p)
quantumEffic = rydetector.QuantumEfficiency(absorption, d1, d2, theta1, n1, n2)
responsivity = rydetector.Responsivity(wavelength / 1e6,quantumEffic)

# to plot absorption coefficient
absFig = ryplot.Plotter(1,figsize=(9,4))
absFig.semilogY(1,wavelength,absorption,'Absorption Coefficient',\
    r'Wavelength [$\mu$m]',r'Absorption Coefficient [m$^{-1}$]');

# to plot quantum efficiency
QE = ryplot.Plotter(1,figsize=(9,4))
QE.plot(1,wavelength,quantumEffic,'Spectral Quantum Efficiency',\
    r'Wavelength [$\mu$m]','Quantum Efficiency');

# to plot spectral response
Respons = ryplot.Plotter(1,figsize=(9,4))
Respons.plot(1,wavelength,responsivity,'Spectral Responsivity',\
    r'Wavelength [$\mu$m]','Responsivity [A/W]');


I am working on the file "09a-DetectorModelling.ipynb".
The first task is:
This notebook forms part of a series on computational optical radiometry
These notebooks are constantly revised and updated, please revisit from time to time.
The source for this notebook is available at
https://github.com/NelisW/ComputationalRadiometry/blob/master/09a-DetectorModelling.ipynb
The date of this document and module versions used in this document are given at the end of the file.
Feedback is appreciated: neliswillers at gmail dot com.
Overview
The pyradi rydetector  module models the primary functionality of optical detectors.  This notebook demonstrates the development of a simple single element InSb detector model suitable for high-level system modelling.  The performance of the detector is described in terms of its figures of merit. The theoretical background to this model is covered the text book.
Concept
The detector is modelled as a bulk material device, where a fraction of the incident photons are absorbed in the material and then converted to an electrical signal.
Analysis
Detector material properties
Calculate the bandgap $E_g$ of the InSb material at the detector operating temperature (80 K), using
(Eq 5.3) imlemented in rydetector.EgVarshni
$$
    E_g = E_{g}(0) -\left(\frac{A T^2}{T + B}\right).
$$
The Varshni model (Eq 5.3) is an approximated empirical fit to predict a semiconductor bandgap at a given temperature. The approximation employs three parameters: the reference bandgap at a
given temperature (usually 0 K), plus two parameters to predict the bandgap at other temperatures (relative to the reference bandgap). Several sets of different Varshni parameters were proposed (Table A.6) by Madelung, Piprek, and Dhanaraj, for the references see my book) using different reference bandgap values, and also based on different experimental data. The bandgap calculation considers three different sets of feasible Varshni coefficients.
The detector cutoff wavelength is calculated from $\lambda_c = 1.24/E_g$.
The data storage in the following cell is done in a Pandas dataframe, simply...
Can you write Python code for it?

# to calculate the semiconductor properties
tempDet = 80  # detector temperature in K
varshni = pd.DataFrame({ 'Madelung' : pd.Series([0.24, 6e-4, 500, tempDet], index=['Eg(0)','A','B','T']),
           'Piprek' : pd.Series([0.235, 3.2e-4, 170, tempDet], index=['Eg(0)','A','B','T']),
           'Dhanaraj' : pd.Series([0.235, 3.1e-4, 452, tempDet], index=['Eg(0)','A','B','T'])})
eg = pd.DataFrame(varshni.apply(lambda x: rydetector.EgVarshni(x['Eg(0)'], x['A'], x['B'], x['T']), axis=0)).T
lambdaC = 1.24 / eg
varshni = pd.concat([varshni, eg], axis=0).rename({0:'Eg'})
varshni = pd.concat([varshni, lambdaC], axis=0).rename({0:'$\lambda_c$'})
Eg = varshni['Madelung']['Eg']
print("Madelung's value for Eg {} is used in the rest of this analysis.".format(Eg))
print("lambda_c is {}.".format(1.24/Eg))
varshni


I am working on the file "09b Sentiment Analysis with LSTM (inference).ipynb".
The first task is:
Experimentation
To make it easy for you to experiment with the model inference, below we wrap all the text above into a single function that you can call.
Can you write Python code for it?

def sentiment(sent, model):
    input_device = be.zeros((sentence_length, 1), dtype=np.int32)
    input_numpy = np.zeros((sentence_length, 1), dtype=np.int32)
    tokens = clean_string(line).strip().split()

    sent = [len(vocab) + 1 if t not in vocab else vocab[t] for t in tokens]
    sent = [start] + [w + index_from for w in sent]
    sent = [oov if w >= vocab_size else w for w in sent]

    trunc = sent[-sentence_length:]  # take the last sentence_length words

    input_numpy[:] = 0  # fill with zeros
    input_numpy[-len(trunc):, 0] = trunc   # place the input into the numpy array
    input_device.set(input_numpy)  # copy the numpy array to device
    y_pred = model.fprop(input_device, inference=True)  # run the forward pass through the model

    return y_pred.get()[1]


I am working on the file "09b Sentiment Analysis with LSTM (inference).ipynb".
The first task is:
Inference
We first generate some buffers on both the host (CPU) and the device (GPU) to hold the input data that we would like to pass to the model for inference. Below the variable be is the backend that we creater with gen_backend earlier in the code. Our backend supports numpy-like functions for allocating buffers on the compute device.
Can you write Python code for it?

import numpy as np

input_device = be.zeros((sentence_length, 1), dtype=np.int32)  # `be` is the backend that we created earlier in the code.
input_numpy = np.zeros((sentence_length, 1), dtype=np.int32)


I am working on the file "09b Sentiment Analysis with LSTM (inference).ipynb".
The first task is:
Load Model
To load the model, we just pass in the saved model file. neon will automatically generate the layers specified in the model file and load the corresponding weights.
Can you write Python code for it?

from neon.models import Model

model = Model('imdb_lstm.pkl')

# we initialize the model, passing in the size of the input data.
model.initialize(dataset=(sentence_length, 1))


I am working on the file "09b Sentiment Analysis with LSTM (inference).ipynb".
The first task is:
Now we write our new movie review. We've included a sample here, but feel free to write your own and see how well the model responds.
POSITIVE:

"The pace is steady and constant, the characters full and engaging, the relationships and interactions natural showing that you do not need floods of tears to show emotion, screams to show fear, shouting to show dispute or violence to show anger. Naturally Joyce's short story lends the film a ready made structure as perfect as a polished diamond, but the small changes Huston makes such as the inclusion of the poem fit in neatly. It is truly a masterpiece of tact, subtlety and overwhelming beauty."

NEGATIVE:

"Beautiful attracts excellent idea, but ruined with a bad selection of the actors. The main character is a loser and his woman friend and his friend upset viewers. Apart from the first episode all the other become more boring and boring. First, it considers it illogical behavior. No one normal would not behave the way the main character behaves. It all represents a typical Halmark way to endear viewers to the reduced amount of intelligence. Does such a scenario, or the casting director and destroy this question is on Halmark producers. Cat is the main character is wonderful. The main character behaves according to his friend selfish."

NEUTRAL:

"The characters voices were very good. I was only really bothered by Kanga. The music, however, was twice as loud in parts than the dialog, and incongruous to the film. As for the story, it was a bit preachy and militant in tone. Overall, I was disappointed, but I would go again just to see the same excitement on my child's face. I liked Lumpy's laugh..."
Can you write Python code for it?

line = """Beautiful attracts excellent idea, but ruined with a bad selection of the actors. The main character is
          a loser and his woman friend and his friend upset viewers. Apart from the first episode all the other become
          more boring and boring. First, it considers it illogical behavior. No one normal would not behave the way the
          main character behaves. It all represents a typical Halmark way to endear viewers to the reduced amount of
          intelligence. Does such a scenario, or the casting director and destroy this question is on Halmark
          producers. Cat is the main character is wonderful. The main character behaves according to
          his friend selfish."""


I am working on the file "09b Sentiment Analysis with LSTM (inference).ipynb".
The first task is:
Now you can easily enter your own review and get the result. Here we included a more neutral review below:
Can you write Python code for it?

line = """The characters voices were very good. I was only really bothered by Kanga. The music, however, was twice
          as loud in parts than the dialog, and incongruous to the film. As for the story, it was a bit preachy and
          militant in tone. Overall, I was disappointed, but I would go again just to see the same excitement on my
          child's face. I liked Lumpy's laugh..."""

result = sentiment(line, model)
print("Sentiment: {}".format(result))


I am working on the file "09b Sentiment Analysis with LSTM (inference).ipynb".
The first task is:
We also define a few parameters, and the load the vocabulary. The vocab is a 1:1 mapping of words to numbers. The file imdb.vocab can be downloaded from https://s3-us-west-1.amazonaws.com/nervana-course/imdb.vocab and placed in the data directory.
Can you write Python code for it?

import pickle as pkl

sentence_length = 128
vocab_size = 20000

# we have some special codes
pad_char = 0  # padding character
start = 1  # marker for start of review
oov = 2  # when the word is out of the vocab
index_from = 3  # index of first word in vocab

# load the vocab
vocab, rev_vocab = pkl.load(open('data/imdb.vocab', 'rb'))


I am working on the file "0_Fundamentals.ipynb".
The first task is:
0.1 Computer Problems
Evaluate $P(x) = 1 + x + ... + x^{50}$ at x = 1.00001. <br/>
Find the error of the computation by comparing with the equivalent expression $Q(x) = (x^{51} - 1) / (x - 1)$.
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
0.4 Loss of significance
0.4 Computer Problems

Calculate the expressions that follow in double precision arithmetic $x = 10^{-1},\cdots,10^{-14}$. Then, using an alternative form of the expression that doesn’t suffer from subtracting nearly equal numbers, repeat the calculation and make a table of results. Report the number of correct digits in the original expression for each $x$. <br/>

$(a)\frac{1 - secx}{tan^{2}x}$
$(b)\frac{1 - (1 - x)^{3}}{x}$
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
Alternative form :
$\frac{1 - (1 - x)^{3} }{x} \Rightarrow \frac{1 - (1 - 3x + 3x^2 - x^3)}{x} \Rightarrow x^2 - 3x + 3$
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
Evaluate $P(x) = 1 - x + x^2 - x^3 + ... + x^{98} - x^{99}$ at x = 1.00001. <br/>
Find a simpler, equivalent expression, and use it to estimate the error of the nested multiplication.
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
Evaluate the quantity $ \sqrt{c^2 + d} - c $ to four correct signiﬁcant digits, where $c = 246886422468$ and $d = 13579$.

$ \sqrt{c^2 + d} - c \Rightarrow \frac{(\sqrt{c^2 + d} - c)(\sqrt{c^2 + d} + c)}{(\sqrt{c^2 + d} + c)} \Rightarrow \frac{(c^2 + d) - c^2}{\sqrt{c^2 + d} + c} \Rightarrow \frac{d}{\sqrt{c^2 + d} + c}$
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
Example
$P(x) = x^6 - 2x^5 + 3x^4 - 4x^3 + 5x^2 - 6x + 7$ at $x = 2$
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
Example : $P(x) = 2x^{4} + 3x^{3} - 3x^{2} + 5x - 1$
Evaluate $P(0.5)$
Can you write Python code for it?


I am working on the file "0_Fundamentals.ipynb".
The first task is:
Find the smallest value of $p$ for which the expression calculated in double precision arithmetic at $x = 10^{-p}$ has no correct significant digits. <br/>

$(a)\frac{tanx - x}{x^3}$
$(b)\frac{e^{x} + cos{x} - sin{x} - 2}{x^3}$
Can you write Python code for it?

def Px2(x, Px):
    while True:
        yield x, Px(x)
        x /= 10

Px2a_generator = Px2(0.1, lambda x : (np.tan(x) - x) / np.power(x, 3))

for i in range(10):
    print( 'P(x = {0:>e}) = {1:>.14f}'.format(*next(Px2a_generator)))
print( 'The smallest value of p is 8' )
print()

Px2b_generator = Px2(0.1, lambda x : (np.exp(x) + np.cos(x) - np.sin(x) - 2) / np.power(x, 3))

for i in range(10):
    print( 'P(x = {0:>e}) = {1:>.14f}'.format(*next(Px2b_generator)))
print( 'The smallest value of p is 6' )


I am working on the file "0_Introductie.ipynb".
The first task is:
Dat we de elemeneten van lijstje hier ding noemen is een keuze. We kunnen ze net zo goed wat anders noemen
Can you write Python code for it?

for iets in lijstje:
    print('iets =',iets)


I am working on the file "0_Introductie.ipynb".
The first task is:
Er is ook een andere manier om dit op te scrijven, we noemen dit list comprehension. Dit heeft als vordeel dat we het resultaat van de iteratie gelijk weer in een nieuwe lijst hebben
Can you write Python code for it?

nieuwe_lijst = [print(ding) for ding in lijstje]
print('nieuwe_lijst = ', nieuwe_lijst)

nieuwe_lijst = [ding+ding for ding in lijstje]
print('nieuwe_lijst = ', nieuwe_lijst)


I am working on the file "0_Introductie.ipynb".
The first task is:
Je kan de invoer veranderen, en per stukje opnieuw berekenen. Doe dit door op een blokje te klikken, iets aan te passen, en met ctr+enter opnieuw uit te voeren
Basis python
In deze workshop gaan we een paar een paar bases (python) begrippen gebruiken, due handig als je alvast weet hoe het werkt
List
Can you write Python code for it?

lijstje = ['nulste','eerste',2,'derde']
print('lijstje =',lijstje)

print('plek 1 =',lijstje[1])
print('plek 0 =',lijstje[0])
print('plek 1 tot en met 3 =',lijstje[1:3])


I am working on the file "0_Introductie.ipynb".
The first task is:
Je kan ook het print commando gebruiken om meer uitvoer te laten zien:
Can you write Python code for it?

# er komen nu twee regels te staan
print('abc')
print('a+b =',a+b)


I am working on the file "0_Introductie.ipynb".
The first task is:
Set
Heel handig voor deze workshop is een set. Een set is een verzameling unieke objecten. Die kan je snel vergelijken.
Can you write Python code for it?

set1 = set([1,2,3,4])
set2 = set([3,4,5,6])
print('alles in set1 behalve als het in set2 is =',set1-set2)
print('alles wat zowel in set1 en set2 is=',set1 & set2)
print('alles wat in set1 en/of set2 is=',set1 | set2)


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
2.- Simple transformations
 Exercise: Define a function 'complete_word' that adds ' fruit' to the input string. Use this function to process all elements in the RDD using map. Print all of the elements in the resulting RDD using collect().
The answer should be:

Testing the function:
apple fruit

These are all the elements in the RDD:
['apple fruit', 'orange fruit', 'banana fruit', 'grape fruit', 'watermelon fruit', 'apple fruit', 'orange fruit', 'apple fruit']

Can you write Python code for it?

def complete_word(word):
    return <COMPLETAR>

print "Testing the function:"
print complete_word('apple')

dataRDDprocessed = dataRDD.map(<COMPLETAR>)

print "\nThese are all the elements in the RDD:"
print dataRDDprocessed.<COMPLETAR>()


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
3.- Creating a pair RDD and counting
Every element of a pair RDD is a tuple (k,v) where k is the key and v is the value.
 Exercise: Transform the original RDD into a pair RDD, where the value is always 1.
The answer should be:

[('apple', 1), ('orange', 1), ('banana', 1), ('grape', 1), ('watermelon', 1), ('apple', 1), ('orange', 1), ('apple', 1)]

Grouped pairs as an interable:
[('orange', <pyspark.resultiterable.ResultIterable object at 0xb0e455cc>), ('watermelon', <pyspark.resultiterable.ResultIterable object at 0xb0e45b8c>), ('grape', <pyspark.resultiterable.ResultIterable object at 0xb0e45bec>), ('apple', <pyspark.resultiterable.ResultIterable object at 0xb0e4546c>), ('banana', <pyspark.resultiterable.ResultIterable object at 0xb1f5dd6c>)]

Grouped pairs as a list
[('orange', [1, 1]), ('watermelon', [1]), ('grape', [1]), ('apple', [1, 1, 1]), ('banana', [1])]

Grouped pairs + count
[('orange', 2), ('watermelon', 1), ('grape', 1), ('apple', 3), ('banana', 1)]

Can you write Python code for it?

pairRDD = dataRDD.map(lambda x: (x, 1))
print pairRDD.collect()

print "Result: (key, iterable):"
groupedRDD = pairRDD.groupByKey()
print groupedRDD.collect()
print " "

print "Result: (key, list of results):"
groupedRDDprocessed = groupedRDD.mapValues(list)
print groupedRDDprocessed.collect()
print " "

print "Result: (key, count):"
groupedRDDprocessed = groupedRDD.mapValues(len)
print groupedRDDprocessed.collect()
print " "


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
6.- Cleaning the text
You may see in the results that we observe some words in capital letters, that some other punctuation characters appear as well. We will incorporate in the code a cleaning function such that we eliminate unwanted characters.  We provide a simple cleaning function that lowers all the characters.
 Exercise: Use it in the code and verify that the word "I" is printed as "i".
Note: Since we are modifying the strings, the counts will differ with respect to the previous values.
The answer should be:

[(u'the', 27267), (u'and', 25340), (u'i', 19540), (u'to', 18656), (u'of', 17301)]

Can you write Python code for it?

def clean_text(string):
    string = string.lower()
    return string

counts = (textRDD
              .flatMap(<COMPLETAR>)
              .map(<COMPLETAR>)
              .map(<COMPLETAR>)
              .reduceByKey(<COMPLETAR>)
              .takeOrdered(<COMPLETAR>)
         )
print counts


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
7.- Removing stopwords
Many of the most frequent words obtained in the previous section are irrelevant to many tasks, they are know as stop-words. We will use here a stop list (list of meaningless words) to clean out those terms.
 Exercise: Observe the  line used for converting the strings to unicode. This task could be implemented using a "for" loop, but we are using what is called a "List Comprehension".
Can you write Python code for it?

import csv
with open('data/english_stopwords.txt', 'rb') as csvfile:

    reader = csv.reader(csvfile)
    stopwords = []

    for row in reader:
        stopwords.append(row[0].replace("'",'').replace('\t',''))

    stopwords = [unicode(s, "utf-8") for s in stopwords]

print stopwords


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Apply the corresponding operation:
- obtain the total number of elements in the RDD (count)
- print the first two elements in the RDD (take)
- print the first two alphabetically sorted elements in the RDD (takeOrdered)
The answer should be:

There are 8 elements in the RDD

These are the first two:
['apple', 'orange']

These are the first two, alphabetically ordered:
['apple', 'apple']

Can you write Python code for it?

N_data = dataRDD.<COMPLETAR>()
print "There are %d elements in the RDD\n" % N_data

print "These are the first two:"
print dataRDD.<COMPLETAR>(2)

print "\nThese are the first two, alphabetically ordered:"
print dataRDD.<COMPLETAR>(2)


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Combine map, reduceByKey and collect to obtain the counts per word:
The answer should be:

[('orange', 2), ('watermelon', 1), ('grape', 1), ('apple', 3), ('banana', 1)]

Can you write Python code for it?

counts = (dataRDD
              .<COMPLETAR>
              .<COMPLETAR>
              .<COMPLETAR>
         )
print counts


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Modify the code by introducing a flatMap operation and observe the result.
The answer should be:

[(u'fawn', 11), (u'bishops.', 2), (u'divinely', 1), (u'mustachio', 1), (u'four', 114), (u'reproach-', 1), (u'drollery.', 1), (u'conjuring', 1), (u'slew.', 1), (u'Calen', 1)]

Can you write Python code for it?

counts = (textRDD
              .flatMap(lambda x: x.split())
              .map(<COMPLETAR>)
              .<COMPLETAR>
              .take(10)
         )
print counts


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Modify the code to obtain 5 words that appear exactly 111 times in the text.
The answer should be:

[(u'think,', 111), (u'see,', 111), (u'gone.', 111), (u"King's", 111), (u'having', 111)]

Can you write Python code for it?

counts = (textRDD
              .flatMap(<COMPLETAR>)
              .map(<COMPLETAR>)
              .reduceByKey(<COMPLETAR>)
              .filter(<COMPLETAR>)
              .take(<COMPLETAR>)
         )
print counts


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Modify the code to obtain the 5 words that most appear in the text.
The answer should be:

[(u'the', 23197), (u'I', 19540), (u'and', 18263), (u'to', 15592), (u'of', 15507)]

Can you write Python code for it?

counts = (textRDD
              .<COMPLETAR>
              .<COMPLETAR>
              .<COMPLETAR>
              .takeOrdered(5,key = lambda x: <COMPLETAR>)
         )
print counts


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Now that we have completely cleaned the words, try to find the 20 most frequent cleaned strings.
The answer should be:

Processing the dataset to find the 20 most frequent strings:

[(u'the', 27361), (u'and', 26028), (u'i', 20681), (u'to', 19150), (u'of', 17463), (u'a', 14593), (u'you', 13615), (u'my', 12481), (u'in', 10956), (u'that', 10890), (u'is', 9134), (u'not', 8497), (u'with', 7771), (u'me', 7769), (u'it', 7678), (u'for', 7558), (u'be', 6857), (u'his', 6857), (u'your', 6655), (u'this', 6602)]

Can you write Python code for it?

print "Processing the dataset to find the 20 most frequent strings:\n"
countsRDDclean = (textRDD
          .<COMPLETAR>
          )
countsRDDclean.cache()
print countsRDDclean.takeOrdered(20,key = lambda x: -x[1])


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Repeat the counting using reduceByKey, a much more efficient approach, since it operates at every worker before sharing results.
The answer should be:

Result: (key, count):
[('orange', 2), ('watermelon', 1), ('grape', 1), ('apple', 3), ('banana', 1)]

Can you write Python code for it?

print "Result: (key, count):"
countRDD = pairRDD.reduceByKey(<COMPLETAR>)
print countRDD.collect()
print " "


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Repeat the scheme above to obtain the total number of characters in the RDD:
The answer should be:

48
48

Can you write Python code for it?

Nchars = sum(dataRDD.<COMPLETAR>)
print Nchars

Nchars = dataRDD.map(len).reduce(<COMPLETAR>)
print Nchars


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Use groupByKey to count the frequencies of every word ( caution!: groupByKey transformation  can be very inefficient, since it needs to exchange data among workers):
The answer should be:

Result: (key, count):
[('apple', 1), ('orange', 1)]
[('orange', 2), ('watermelon', 1), ('grape', 1), ('apple', 3), ('banana', 1)]

Can you write Python code for it?

print "Result: (key, count):"

countRDD = pairRDD.groupByKey().map(<COMPLETAR>)

print countRDD.collect()
print " "


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
Exercise: Use the code written in the previous sections to obtain the counts for every word in the text. Print the first 10 results. Observe the result, is this what we want? What is going wrong?
The answer should be:

[(u'', 9493), (u'    thou diest in thine unthankfulness, and thine ignorance makes', 1), (u"    Which I shall send you written, be assur'd", 1), (u'    I do beseech you, take it not amiss:', 1), (u'    their mastiffs are of unmatchable courage.', 1), (u'    With us in Venice, if it be denied,', 1), (u"  Hot. I'll have it so. A little charge will do it.", 1), (u'     By what yourself, too, late have spoke and done,', 1), (u"  FIRST LORD. He's but a mad lord, and nought but humours sways him.", 1), (u'    none will entertain it.', 1)]

Can you write Python code for it?

counts = (textRDD
              .map(lambda x: (x, 1))
              .<COMPLETAR>
              .take(10)
         )
print counts


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
We will now search for non-alphabetical characters in the dataset. We can use the Python method 'isalpha' to decide wether or not a string is composed of characters a-z.
 Exercise: Use that function to print the 20 words with non-alphabetic characters that most appear in the text and print the total number of strings with non-alphabetic characters.
The answer should be:

The database has 40957 words that need cleaning, for example:

[(u"i'll", 1737), (u'you,', 1478), (u"'tis", 1367), (u'sir,', 1235), (u'me,', 1219), (u"th'", 1146), (u'o,', 1008), (u'lord,', 977), (u'come,', 875), (u'me.', 823), (u'you.', 813), (u'why,', 805), (u'now,', 785), (u'it.', 784), (u'him.', 755), (u'lord.', 702), (u'him,', 698), (u'ay,', 661), (u'well,', 647), (u'and,', 647)]

Can you write Python code for it?

countsRDD = (textRDD
              .flatMap(<COMPLETAR>)
              .map(<COMPLETAR>)
              .filter(lambda x: not x.isalpha())
              .map(<COMPLETAR>)
              .reduceByKey(<COMPLETAR>)
              )
countsRDD.cache()

print "The database has %d words that need cleaning, for example:\n" % countsRDD.count()
print countsRDD.takeOrdered(20,key = lambda x: -x[1])


I am working on the file "0_Introduction/Intro_PySpark_1.ipynb".
The first task is:
We will use now a lambda function to do the same task
Can you write Python code for it?

dataRDDprocessed_lambda = dataRDD.map(lambda x: x + ' fruit')

print "Result with a lambda function:"
print dataRDDprocessed_lambda.<COMPLETAR>()


I am working on the file "0_Introduction/Intro_PySpark_2.ipynb".
The first task is:
Basic counts
We want to compute some counts on the log file, to answer some simple questions:
EXERCISE: Obtain the unique values of field 4, and how many times they appear in the dataset.
The answer should be:

[('HEAD', 2395), ('GET', 962318), ('POST', 68)]

Can you write Python code for it?

unique_field4 = (cleanRDD
                     .map(lambda x: <COMPLETAR>)
                     .reduceByKey(lambda x, y: x + y)
                     .collect()
                 )

print unique_field4


I am working on the file "0_Introduction/Intro_PySpark_2.ipynb".
The first task is:
Unfortunately, some preprocessing errors have occurred and some of the RDD elements do not have 9 fields. We need to fix that before we can continue.
EXERCISE: Obtain the size of every RDD element as a Python List and identify the unique values.
The answer should be:

We have extracted 977769 sizes, and the unique values are:
[9, 10, 11, 1]

Can you write Python code for it?

Element_sizes = (RDDlist
               .map(lambda x: <COMPLETAR>)
                .collect()
         )

Nsizes = len(Element_sizes)
Unique_values = list(set(Element_sizes))

print "We have extracted %d sizes, and the unique values are:" % Nsizes
print Unique_values


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
Counting tokens: we want to know the vocabulary size in every one of the datasets.
EXERCISE:  Count the number of tokens in every dataset. Also count the number of UNIQUE tokens in every dataset.
The answer should be:

amazonSmall has 14052 tokens.
amazon has 133267 tokens.
googleSmall has 5710 tokens.
google has 98588 tokens.

amazonSmall has 3700 unique tokens.
amazon has 11505 unique tokens.
googleSmall has 2305 unique tokens.
google has 11176 unique tokens.

Can you write Python code for it?

NtokensamazonSmall = amazonSmallRDDtokenized.<COMPLETAR>
print "amazonSmall has %d tokens." % NtokensamazonSmall

Ntokensamazon = amazonRDDtokenized.<COMPLETAR>
print "amazon has %d tokens." % Ntokensamazon

NtokensgoogleSmall = googleSmallRDDtokenized.<COMPLETAR>
print "googleSmall has %d tokens." % NtokensgoogleSmall

Ntokensgoogle = googleRDDtokenized.<COMPLETAR>
print "google has %d tokens." % Ntokensgoogle

print " "

NtokensuniqueamazonSmall = len(amazonSmallRDDtokenized.<COMPLETAR>
print "amazonSmall has %d unique tokens." % NtokensuniqueamazonSmall

Ntokensuniqueamazon = len(amazonRDDtokenized.<COMPLETAR>
print "amazon has %d unique tokens." % Ntokensuniqueamazon

NtokensuniquegoogleSmall = len(googleSmallRDDtokenized.<COMPLETAR>
print "googleSmall has %d unique tokens." % NtokensuniquegoogleSmall

Ntokensuniquegoogle = len(googleRDDtokenized.<COMPLETAR>
print "google has %d unique tokens." % Ntokensuniquegoogle


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
EXERCISE:  As a final step, we will print the contents in amazon and google corresponding to that element, to check if they are similar or not.
The answer should be:

The google ID is: http://www.google.com/base/feeds/snippets/18411875162562199123

The amazon ID is: b000j4k804

The google content is: topics entertainment 40248 instant immersion spanish audio book audio book "instant immersion spanish (audio book) (audio book)" "topics entertainment"

The amazon content is: instant immersion spanish (audio book) "instant immersion spanish (audio book) (audio book)" "topics entertainment"

Can you write Python code for it?

googleID = <COMPLETAR>
print "The google ID is: %s\n" % googleID
amazonID = <COMPLETAR>
print "The amazon ID is: %s\n" % amazonID

googleContent = (googleSmallRDD
                         .map(lambda x: x.split(';'))
                         .filter(lambda x: <COMPLETAR> == googleID)
                         .map(lambda x: x[1])
                         .first()
                )

amazonContent = (amazonSmallRDD
                         .map(lambda x: x.split(';'))
                         .filter(lambda x: <COMPLETAR> == amazonID)
                         .map(lambda x: x[1])
                         .first()
                )

print "The google content is:\n"
print googleContent
print "\nThe amazon content is:\n"
print amazonContent


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
EXERCISE:  Now, find the element with the largest similarity.
The answer should be:

This is the element in cosimRDD with the largest similarity:

[('http://www.google.com/base/feeds/snippets/18411875162562199123', 'b000j4k804', 0.9712858623572642)]

Can you write Python code for it?

mostSimilarPair = (cosimRDD
              .takeOrdered(1, lambda x: <COMPLETAR>)
              )

print "This is the element in cosimRDD with the largest similarity:\n"
print mostSimilarPair


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
EXERCISE:  Repeat the computations with the full datasets. Open in a new browser the page localhost:4040 to see the evolution of the tasks.
The answer should be:

The allPairsRDD has 4397038 elements.

This is one of the elements in allPairsRDD:

[(('http://www.google.com/base/feeds/snippets/11125907881740407428', ['learning', 'quickbooks', '2007', 'learning', 'quickbooks', '2007', 'intuit']), ('b000jz4hqo', ['clickart', '950', '000', 'premier', 'image', 'pack', 'dvd', 'rom', 'broderbund']))]

This is the first element in cosimRDD:

[('http://www.google.com/base/feeds/snippets/11125907881740407428', 'b000jz4hqo', 0.0)]

This is the element in cosimRDD with the largest similarity:

[('http://www.google.com/base/feeds/snippets/17521446718236049500', 'b000v9yxj4', 1.0)]

The google ID is: http://www.google.com/base/feeds/snippets/17521446718236049500

The amazon ID is: b000v9yxj4

The google content is: nero inc nero 8 ultra edition

The amazon content is: nero 8 ultra edition  "nero inc."

Can you write Python code for it?

allPairsRDD = (<COMPLETAR>
              .cache())

print "The allPairsRDD has %d elements.\n" % allPairsRDD.count()

print "This is one of the elements in allPairsRDD:\n"
print allPairsRDD.take(1)

cosimRDD = (<COMPLETAR>
              .cache())

print "\nThis is the first element in cosimRDD:\n"
print cosimRDD.take(1)

mostSimilarPair = (cosimRDD
              .<COMPLETAR>
              )

print "\nThis is the element in cosimRDD with the largest similarity:\n"
print mostSimilarPair

googleID = <COMPLETAR>
print "The google ID is: %s\n" % googleID
amazonID = <COMPLETAR>
print "The amazon ID is: %s\n" % amazonID

googleContent = (googleRDD
                         .<COMPLETAR>
                )

amazonContent = (amazonRDD
                         .<COMPLETAR>
                )

print "The google content is: %s\n" % googleContent
print "The amazon content is: %s\n" % amazonContent


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
EXERCISE: Transform "allPairsRDD" to obtain a new "cosimRDD" with the format (googleID, amazonID, cosim).
The answer should be:

This is the first element in cosimRDD:

[('http://www.google.com/base/feeds/snippets/11448761432933644608', 'b000jz4hqo', 0.0)]

Can you write Python code for it?

cosimRDD = (allPairsRDD
              .map(lambda x: (<COMPLETAR>, <COMPLETAR>, <COMPLETAR>))
              .cache())

print "This is the first element in cosimRDD:\n"
print cosimRDD.take(1)


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
Largest and smallest item in amazon dataset.
EXERCISE:  Build an 'amazonCountRDD' that stores (ID, [Tokens], Number of tokens). Print the largest content (the record with the largest number of tokens) and the smallest content (the one with the smaller number of tokens).
The answer should be:

amazon smallest item, with length 2, is:
[('b000hlt5g2', ['monopoly', 'encore'], 2)]

amazon largest item, with length 1520, is:
[('b0007lw22q', ['apple', 'ilife', '06', 'family', 'pack', 'mac', 'dvd', 'older', ....

Can you write Python code for it?

amazonCountRDD = amazonRDDtokenized.map(lambda x: (<COMPLETAR>, <COMPLETAR>, <COMPLETAR>)).cache()

smallestItem = amazonCountRDD.takeOrdered(1, lambda x: x[2])
largestItem = amazonCountRDD.takeOrdered(1, lambda x: -x[2])

print "amazon smallest item, with length %d, is:" % smallestItem[0][2]
print smallestItem
print " "

print "amazon largest item, with length %d, is:" % largestItem[0][2]
print largestItem
print " "