I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Tokenize Punctuation
We'll be splitting the script into a word array using spaces as delimiters.  However, punctuations like periods and exclamation marks make it hard for the neural network to distinguish between the word "bye" and "bye!".
Implement the function token_lookup to return a dict that will be used to tokenize symbols like "!" into "||Exclamation_Mark||".  Create a dictionary for the following symbols where the symbol is the key and value is the token:
- Period ( . )
- Comma ( , )
- Quotation Mark ( " )
- Semicolon ( ; )
- Exclamation mark ( ! )
- Question mark ( ? )
- Left Parentheses ( ( )
- Right Parentheses ( ) )
- Dash ( -- )
- Return ( \n )
This dictionary will be used to token the symbols and add the delimiter (space) around it.  This separates the symbols as it's own word, making it easier for the neural network to predict on the next word. Make sure you don't use a token that could be confused as a word. Instead of using the token "dash", try using something like "||dash||".
Can you write Python code for it?

def token_lookup():
    """
    Generate a dict to turn punctuation into a token.
    :return: Tokenize dictionary where the key is the punctuation and the value is the token
    """
    # Implement Function
    token_dict = {
        '.': '<<period>>',
        ',': '<<comma>>',
        '"': '<<quotation_mark>>',
        ';': '<<semicolon>>',
        '!': '<<exclamation_mark>>',
        '?': '<<question_mark>>',
        '(': '<<left_parentheses>>',
        ')': '<<right_parentheses>>',
        '--': '<<dash>>',
        '\n': '<<return>>',
    }

    return token_dict

tests.test_tokenize(token_lookup)


I am working on the file "03.recurrent_neural_network/tv_script_generation.ipynb".
The first task is:
Train
Train the neural network on the preprocessed data.
Can you write Python code for it?

batches = get_batches(int_text, batch_size, seq_length)

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(num_epochs):
        state = sess.run(initial_state, {input_text: batches[0][0]})

        for batch_i, (x, y) in enumerate(batches):
            feed = {
                input_text: x,
                targets: y,
                initial_state: state,
                lr: learning_rate}
            train_loss, state, _ = sess.run([cost, final_state, train_op], feed)

            # Show every <show_every_n_batches> batches
            if (epoch_i * len(batches) + batch_i) % show_every_n_batches == 0:
                print('Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(
                    epoch_i,
                    batch_i,
                    len(batches),
                    train_loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_dir)
    print('Model Trained and Saved')


I am working on the file "03/Untitled.ipynb".
The first task is:
21. カテゴリ名を含む行を抽出
記事中でカテゴリ名を宣言している行を抽出せよ．
Can you write Python code for it?


I am working on the file "03/keras_CNN_architectures.ipynb".
The first task is:
Compute test accuracy by hand
Can you write Python code for it?

classProb = model.predict(x=tstData)
print(classProb.shape)

correctProb = (classProb * tstLabels).sum(axis=1)
wrongProb = (classProb * (1-tstLabels)).max(axis=1)
print(correctProb.shape, wrongProb.shape)

accuracy = (correctProb > wrongProb).mean()
print('Accuracy: ',  accuracy)


I am working on the file "03/keras_CNN_architectures.ipynb".
The first task is:
Predict and evaluate
Can you write Python code for it?

classProb = model.predict(x=tstData[0:2])
print('Class probabilities:', classProb, '\n')
loss, acc = model.evaluate(x=tstData, y=tstLabels, batch_size=1024)
print()
print('loss', loss)
print('acc', acc)


I am working on the file "0328_VectorizedFeedForwardNetworks-1553747879251.ipynb".
The first task is:
Generate data
Can you write Python code for it?

data, labels = make_blobs(n_samples=1000, centers=4, n_features=2, random_state=0)
print(data.shape, labels.shape)

plt.scatter(data[:,0], data[:,1], c=labels, cmap=my_cmap)
plt.show()

labels_orig = labels
labels = np.mod(labels_orig, 2)

plt.scatter(data[:,0], data[:,1], c=labels, cmap=my_cmap)
plt.show()


I am working on the file "0328_VectorizedFeedForwardNetworks-1553747879251.ipynb".
The first task is:
Multi class classification
Can you write Python code for it?

X_train, X_val, Y_train, Y_val = train_test_split(data, labels_orig, stratify=labels_orig, random_state=0)
print(X_train.shape, X_val.shape, labels_orig.shape)

enc = OneHotEncoder()
# 0 -> (1, 0, 0, 0), 1 -> (0, 1, 0, 0), 2 -> (0, 0, 1, 0), 3 -> (0, 0, 0, 1)
y_OH_train = enc.fit_transform(np.expand_dims(Y_train,1)).toarray()
y_OH_val = enc.fit_transform(np.expand_dims(Y_val,1)).toarray()
print(y_OH_train.shape, y_OH_val.shape)

W1 = np.random.randn(2,2)
W2 = np.random.randn(2,4)
print(W1)
print(W2)


I am working on the file "03A_Variational_Autoencoder.ipynb".
The first task is:
[1] The primary purpose of TensorFlow is to construct a computation graph connecting Tensors and operations. Each of these nodes must be assigned a unique name; if the user does not specify one, a unique name is automatically generated, like 'Placeholder_2', with the number at the end incrementing each time you create a new node of that type. Attempting to create a node with a name already found in the graph raises an error.
So how can this be problematic? In the Coding Environments notebook (00B), it was mentioned that code from previously run cells persists. As such, if we're programming interactively and want to rebuild our graph after some updates, the new updated nodes we want to add collide with the names from our previous run, throwing an error. Why didn't we have to worry about this before? In the past, we haven't been naming our variables, so TensorFlow has been giving the nodes new unique names every time we update the graph and adding them to the collection of nodes from previous runs; the old nodes are never called, so they just sit there. However, TF-Slim does name the variables it generates, thus causing the problem. We can solve this by creating a new graph object before we define our computation graph, so every time we want to make modifications to the graph, we start anew.
If you're confused by that explanation, I wouldn't worry about it. It's not necessary for the program to run. It's there so we can re-run the cell defining the computation graph without restarting the entire kernel to clear memory of previous variables. In a traditionally written Python program (i.e. not IPython), you wouldn't need to do this.
For training, we'll stay simple and train for 20000 iterations, visualizing our results with 5 digits from the validation set after every 1000 minibatches. Notice that this model is completely unsupervised: we never include the digit labels at any point in the process. Within a few thousand iterations, the model should start producing reasonable...
Can you write Python code for it?

with tf.Session() as sess:
    # Initialize all variables
    sess.run(tf.global_variables_initializer())

    # Train VAE model
    for i in range(20000):
        # Get a training minibatch
        batch = mnist.train.next_batch(100)

        # Binarize the data
        x_binarized = (batch[0] > 0.5).astype(np.float32)

        # Train on minibatch
        sess.run(train_step_unlabeled, feed_dict={x: x_binarized}) # No labels

        # Visualize reconstructions every 1000 iterations
        if i % 1000 == 0:
            batch = mnist.validation.next_batch(5)
            x_binarized = (batch[0] > 0.5).astype(np.float32)
            reconstructions = sess.run(x_hat, feed_dict={x: x_binarized})
            print("Iteration {0}:".format(i))
            visualize_row(batch[0], reconstructions)


I am working on the file "03A_Variational_Autoencoder.ipynb".
The first task is:
Define the graph and train
All of the functions we've written thus far are just that: functions. We still need to call them to assemble our TensorFlow computation graph. At this point, this should be becoming familiar.
One of the small differences is the inclusion of tf.reset_default_graph(), added to remedy a small, unfortunate side effect of using Jupyter and TensorFlow in conjunction, but you don't have to worry about it too much to understand the model. A more detailed explanation if you're interested below [1].
Can you write Python code for it?

# Reset the graph
tf.reset_default_graph()

# Define input placeholder
x = tf.placeholder(tf.float32,[None, 784], name='x')

# Define VAE graph
with tf.variable_scope('encoder'):
    mu_logvar = encoder(x)
with tf.variable_scope('decoder'):
    x_logits, x_dist = decoder(mu_logvar)
    x_hat = x_dist.sample()

# Optimization
with tf.variable_scope('unlabeled') as scope:
    train_step_unlabeled = optimizer(x_logits, x, mu_logvar)


I am working on the file "03A_Variational_Autoencoder.ipynb".
The first task is:
Encoder
The encoder deterministically transforms the data $x$ from the data space to the latent space of $z$. Since we're dealing with a variational autoencoder, we attempt to model the distribution of the latent space given the input, represented by $q(z|x)$. This isn't immediately obvious in the code implementation, but we assume a standard Gaussian prior on this distribution, and our encoder returns the mean and variance (actually log-variance) of this distribution. We use log-variance because our model returns a real number, while variances must be positive.
MNIST is a very simple dataset, so let's also keep the model simple: an MLP with 2 fully connected layers. We name the output mu_logvar as we will be interpretting the first half of the final 128-dimensional vector as the mean $\mu$ and the second half as the log-variance log($\sigma^2$).
Can you write Python code for it?

def encoder(x):
    """Network q(z|x)"""
    with slim.arg_scope([slim.fully_connected],
                    activation_fn=tf.nn.relu,
                    weights_initializer=tf.truncated_normal_initializer(0.0, 0.1)):
        mu_logvar = slim.fully_connected(x, 128, scope='fc1')
        mu_logvar = slim.fully_connected(mu_logvar, 128, activation_fn=None, scope='fc2')

    return mu_logvar


I am working on the file "03A_Variational_Autoencoder.ipynb".
The first task is:
Loss
Prof. Jun Zhu talked in class about the theoretical motivation for the loss of the VAE model. Like all variational inference techniques, it tries to match the variational posterior distribution (here a neural network) with the true posterior. However, at the end of the derivation, we can think of our model as trading off two goals:

Reconstruction loss: Our generator produces parameters to a Bernoulli distribution that is supposed to represent $p(x | z)$; because we assume that $z$ is the latent representation of an actual data point $x$, we can measure how well we achieve this goal by measuring the likelihood of $x$ according to that Bernoulli distribution. Another way of thinking of this is that we can measure how similar our reconstructed image is to our original image. The measure of similarity we use is cross-entropy: we think of our model as classifying each pixel as black or white, and we measure how good the classifier is using the classic sigmoid cross-entropy loss.

KL Divergence: Because this model is variational, we also include a KL penalty to impose a Gaussian prior on the latent space. The exact derivation of this term can be found in the original Auto-Encoding Variational Bayes paper. Is a standard Gaussian prior a good assumption? What are the potential weaknesses of this approach?

We use the ADAM algorithm that we've used before for optimization.
Can you write Python code for it?

def optimizer(x_logits, x, mu_logvar):
    """Define loss functions (reconstruction, KL divergence) and optimizer"""
    with tf.variable_scope('optimizer') as scope:
        # Reconstruction loss
        reconstruction = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=x, logits=x_logits), reduction_indices=[1])

        # KL divergence
        mu, logvar = tf.split(mu_logvar, num_or_size_splits=2, axis=1)
        kl_d = -0.5 * tf.reduce_sum(1.0 + logvar - tf.square(mu) - tf.exp(logvar), reduction_indices=[1])

        # Total loss
        loss = tf.reduce_mean(reconstruction + kl_d)

        # ADAM optimizer
        train_step = tf.train.AdamOptimizer().minimize(loss)

    return train_step


I am working on the file "03A_Variational_Autoencoder.ipynb".
The first task is:
Visualization
It'll be nice to visualize the reconstructions that our model generates to see what it learns. This helper function plots the original inputs in one column and the reconstructions next to them in another column. I also may or may not have stolen it from Alex Lew, who included it in his GAN notebook (03B)...
Can you write Python code for it?

def visualize_row(image, reconstruction, img_width=28, cmap='gray'):
    """
    Takes in a tensor of images of given width, and displays them in a column
    in a plot, using `cmap` to map from numbers to colors.
    """
    fig, ax = plt.subplots(1, 2)
    image = np.reshape(image, [-1, img_width])
    reconstruction = np.reshape(reconstruction, [-1, img_width])
    plt.figure()
    ax[0].imshow(np.clip(image, 0, 1), cmap=cmap)
    ax[1].imshow(np.clip(reconstruction, 0, 1), cmap=cmap)
    plt.show()


I am working on the file "03B_Generative_Adversarial_Network.ipynb".
The first task is:
Generator
A GAN is made up of two smaller networks: a generator and a discriminator. The generator is responsible for sampling images from a distribution that we hope will get closer and closer, as we train, to the real data distribution.
Neural networks are deterministic, so in order to sample a new image from the generator, we first create some random noise z (in our case, z will be a 100-dimensional uniform random variable) and then feed that noise to the network. You can think of z as being a latent, low-dimensional representation of some image G(z), though in a vanilla GAN, it is usually difficult to interpret z's components in a meaningful way.
Our generator is a dead-simple multi-layer perceptron (feed-forward network), with 128 hidden units.
Can you write Python code for it?

def generator(z):
    """
    Given random noise `z`, use a simple MLP with 128 hidden units to generate a
    sample image (784 values between 0 and 1, enforced with the sigmoid function).
    """
    with tf.variable_scope("fc1"):
        fc1 = fully_connected_layer(z, 128)
    with tf.variable_scope("fc2"):
        return fully_connected_layer(fc1, 784, activation_function=tf.sigmoid)


I am working on the file "03B_Generative_Adversarial_Network.ipynb".
The first task is:
Loading the data
As in previous examples, we'll use MNIST, because it's a small and easy-to-use dataset that comes bundled with Tensorflow.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)


I am working on the file "03B_Generative_Adversarial_Network.ipynb".
The first task is:
We'll also provide a simple function for displaying a few 28-pixel images. This will help us understand the progress of our GAN as it trains; we'll use it to visualize the generated 'fake digit' images.
Can you write Python code for it?

def visualize_row(images, img_width=28, cmap='gray'):
    """
    Takes in a tensor of images of given width, and displays them in a column
    in a plot, using `cmap` to map from numbers to colors.
    """
    im = np.reshape(images, [-1, img_width])
    plt.figure()
    plt.imshow(im, cmap=cmap)
    plt.show()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
And these are the results of applying the filter-weights to the second image.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Applying each of these convolutional filters to the first input image gives the following output images, which are then used as input to the second convolutional layer.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Classification Accuracy
We need to calculate the classification accuracy so we can report progress to the user.
First we create a vector of booleans telling us whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Convolution Layer 1
Now plot the filter-weights for the first convolutional layer.
Note that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Helper-function for plotting the output of a convolutional layer
Can you write Python code for it?

def plot_conv_layer(layer, image):
    # Assume layer is a TensorFlow op that outputs a 4-dim tensor
    # which is the output of a convolutional layer,
    # e.g. layer_conv1 or layer_conv2.

    # Create a feed-dict containing just one image.
    # Note that we don't need to feed y_true because it is
    # not used in this calculation.
    feed_dict = {x: [image]}

    # Calculate and retrieve the output values of the layer
    # when inputting that image.
    values = session.run(layer, feed_dict=feed_dict)

    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Helper-function for showing the performance
Below is a function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Note that this function can use a lot of computer memory, which is why the test-set is split into smaller batches. If you have little RAM in your computer and it crashes, then you can try and lower the batch-size.
Can you write Python code for it?

# Split the test-set into smaller batches of this size.
test_batch_size = 256

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.test.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + test_batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.test.images[i:j, :]

        # Get the associated labels.
        labels = data.test.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Convenience variable for the true class-numbers of the test-set.
    cls_true = data.test.cls

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, correct_sum,...


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Helper-function to perform optimization iterations
There are 55,000 images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to do more optimization iterations.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Helper-function to plot confusion matrix
Can you write Python code for it?

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Initialize variables
The variables for the TensorFlow graph must be initialized before we start optimizing them.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Input Images
Helper-function for plotting an image.
Can you write Python code for it?

def plot_image(image):
    plt.imshow(image.reshape(img_shape),
               interpolation='nearest',
               cmap='binary')

    plt.show()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Layers Implementation
We now implement the same Convolutional Neural Network using the Layers API that is included in TensorFlow version 1.1. This requires more code than PrettyTensor, although a lot of the following are just comments.
We use the net-variable to refer to the last layer while building the Neural Network. This makes it easy to add or remove layers in the code if you want to experiment. First we set the net-variable to the reshaped input image.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Next we want to add fully-connected layers to the Neural Network, but these require 2-rank tensors as input, so we must first flatten the tensors.
The tf.layers API was first located in tf.contrib.layers before it was moved into TensorFlow Core. But even though it has taken the TensorFlow developers a year to move these fairly simple functions, they have somehow forgotten to move the even simpler flatten() function. So we still need to use the one in tf.contrib.layers.
Can you write Python code for it?

net = tf.contrib.layers.flatten(net)

# This should eventually be replaced by:
# net = tf.layers.flatten(net)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
One of the advantages of constructing neural networks in this fashion, is that we can now easily pull out a reference to a layer. This was more complicated in PrettyTensor.
Further below we want to plot the output of the first convolutional layer, so we create another variable for holding a reference to that layer.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Performance after 1 optimization iteration
The classification accuracy does not improve much from just 1 optimization iteration, because the learning-rate for the optimizer is set very low.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Performance after 10,000 optimization iterations
After 10,000 optimization iterations, the model has a classification accuracy on the test-set of about 99%.
Can you write Python code for it?

%%time
optimize(num_iterations=9000) # We performed 1000 iterations above.

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Performance after 100 optimization iterations
After 100 optimization iterations, the model has significantly improved its classification accuracy.
Can you write Python code for it?

%%time
optimize(num_iterations=99) # We already performed 1 iteration above.

print_test_accuracy(show_example_errors=True)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Performance after 1000 optimization iterations
After 1000 optimization iterations, the model has greatly increased its accuracy on the test-set to more than 90%.
Can you write Python code for it?

%%time
optimize(num_iterations=900) # We performed 100 iterations above.

print_test_accuracy(show_example_errors=True)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the variables for the neural network have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Plot an image from the test-set which will be used as an example below.
Can you write Python code for it?

image1 = data.test.images[0]
plot_image(image1)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Plot another example image from the test-set.
Can you write Python code for it?

image2 = data.test.images[13]
plot_image(image2)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
PrettyTensor Implementation
This section shows the implementation of a Convolutional Neural Network using PrettyTensor taken from Tutorial #03 so it can be compared to the implementation using the Layers API below. This code has been enclosed in an if False: block so it does not run here.
The basic idea is to wrap the input tensor x_image in a PrettyTensor object which has helper-functions for adding new computational layers so as to create an entire Convolutional Neural Network. This is a fairly simple and elegant syntax.
Can you write Python code for it?

if False:
    x_pretty = pt.wrap(x_image)

    with pt.defaults_scope(activation_fn=tf.nn.relu):
        y_pred, loss = x_pretty.\
            conv2d(kernel=5, depth=16, name='layer_conv1').\
            max_pool(kernel=2, stride=2).\
            conv2d(kernel=5, depth=36, name='layer_conv2').\
            max_pool(kernel=2, stride=2).\
            flatten().\
            fully_connected(size=128, name='layer_fc1').\
            softmax_classifier(num_classes=num_classes, labels=y_true)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used for inputting data to the graph.
Variables that are going to be optimized so as to make the convolutional network perform better.
The mathematical formulas for the convolutional neural network.
A so-called cost-measure or loss-function that can be used to guide the optimization of the variables.
An optimization method which updates the variables.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional array. The...
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "03B_Layers_API.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The following images are the results of applying the convolutional filters to the second image.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The input image is then input to the first convolutional layer, which has 16 filters each of size 5x5 pixels. The activation-function is the Rectified Linear Unit (ReLU) described in more detail in Tutorial #02.
Can you write Python code for it?

net = tf.layers.conv2d(inputs=net, name='layer_conv1', padding='same',
                       filters=16, kernel_size=5, activation=tf.nn.relu)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
The output of the second convolutional layer is also max-pooled for down-sampling the images.
Can you write Python code for it?

net = tf.layers.max_pooling2d(inputs=net, pool_size=2, strides=2)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
This function performs a number of optimization iterations so as to gradually improve the variables of the neural network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations


I am working on the file "03B_Layers_API.ipynb".
The first task is:
This tells us how likely the neural network thinks the input image is of each possible class. The one that has the highest value is considered the most likely so its index is taken to be the class-number.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
This was developed using Python 3.6 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Using this helper-function we can retrieve the variables. These are TensorFlow objects. In order to get the contents of the variables, you must do something like: contents = session.run(weights_conv1) as demonstrated further below.
Can you write Python code for it?

weights_conv1 = get_weights_variable(layer_name='layer_conv1')
weights_conv2 = get_weights_variable(layer_name='layer_conv2')


I am working on the file "03B_Layers_API.ipynb".
The first task is:
Visualization of Weights and Layers
Helper-function for plotting convolutional weights
Can you write Python code for it?

def plot_conv_weights(weights, input_channel=0):
    # Assume weights are TensorFlow ops for 4-dim variables
    # e.g. weights_conv1 or weights_conv2.

    # Retrieve the values of the weight-variables from TensorFlow.
    # A feed-dict is not necessary because nothing is calculated.
    w = session.run(weights)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Number of filters used in the conv. layer.
    num_filters = w.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = w[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We also want to plot the output of this convolutional layer, so we keep a reference for later use.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We can now add fully-connected layers to the neural network. These are called dense layers in the Layers API.
Can you write Python code for it?

net = tf.layers.dense(inputs=net, name='layer_fc1',
                      units=128, activation=tf.nn.relu)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the image classifications.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We have now created the exact same Convolutional Neural Network in a few lines of code that required many complex lines of code in the direct TensorFlow implementation.
The Layers API is perhaps not as elegant as PrettyTensor, but it has some other advantages, e.g. that we can more easily refer to intermediate layers, and it is also easier to construct neural networks with branches and multiple outputs using the Layers API.
Loss-Function to be Optimized
To make the model better at classifying the input images, we must somehow change the variables of the Convolutional Neural Network.
The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the variables of the model.
TensorFlow has a function for calculating the cross-entropy, which uses the values of the logits-layer because it also calculates the softmax internally, so as to to improve numerical stability.
Can you write Python code for it?

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=logits)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We need the neural network to classify the input images into 10 different classes. So the final fully-connected layer has num_classes=10 output neurons.
Can you write Python code for it?

net = tf.layers.dense(inputs=net, name='layer_fc_out',
                      units=num_classes, activation=None)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We now add the second convolutional layer which has 36 filters each with 5x5 pixels, and a ReLU activation function again.
Can you write Python code for it?

net = tf.layers.conv2d(inputs=net, name='layer_conv2', padding='same',
                       filters=36, kernel_size=5, activation=tf.nn.relu)


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We now do the max-pooling on the output of the convolutional layer. This was also described in more detail in Tutorial #02.
Can you write Python code for it?


I am working on the file "03B_Layers_API.ipynb".
The first task is:
We use the softmax function to 'squash' the outputs so they are between zero and one, and so they sum to one.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Convolutional Weights
Now that we have the layers we can easily get their weights.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
# This is used for plotting the images.
img_shape = (img_size, img_size)

# Tuple with height, width and depth used to reshape arrays.
# This is used for reshaping in Keras.
img_shape_full = (img_size, img_size, 1)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Delete the model from memory so we are sure it is no longer used.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Evaluation
Now that the model has been trained we can test its performance on the test-set. This also uses numpy-arrays as input.
Can you write Python code for it?

result = model.evaluate(x=data.x_test,
                        y=data.y_test)


result = model.evaluate(x=data.test.images,
                        y=data.test.labels)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Evaluation
Once the model has been trained we can evaluate its performance on the test-set. This is the same syntax as for the Sequential Model.
Can you write Python code for it?

result = model2.evaluate(x=data.x_test,
                         y=data.y_test)


result = model2.evaluate(x=data.test.images,
                         y=data.test.labels)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Examples of Mis-Classified Images
We can plot some examples of mis-classified images from the test-set.
First we get the predicted classes for all the images in the test-set:
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
For a classification-problem such as MNIST which has 10 possible classes, we need to use the loss-function called categorical_crossentropy. The performance metric we are interested in is the classification accuracy.
Can you write Python code for it?

model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Functional Model
The Keras API can also be used to construct more complicated networks using the Functional Model. This may look a little confusing at first, because each call to the Keras API will create and return an instance that is itself callable. It is not clear whether it is a function or an object - but we can call it as if it is a function. This allows us to build computational graphs that are more complex than the Sequential Model allows.
Can you write Python code for it?

# Create an input layer which is similar to a feed_dict in TensorFlow.
# Note that the input-shape must be a tuple containing the image-size.
inputs = Input(shape=(img_size_flat,))

# Variable used for building the Neural Network.
net = inputs

# The input is an image as a flattened array with 784 elements.
# But the convolutional layers expect images with shape (28, 28, 1)
net = Reshape(img_shape_full)(net)

# First convolutional layer with ReLU-activation and max-pooling.
net = Conv2D(kernel_size=5, strides=1, filters=16, padding='same',
             activation='relu', name='layer_conv1')(net)
net = MaxPooling2D(pool_size=2, strides=2)(net)

# Second convolutional layer with ReLU-activation and max-pooling.
net = Conv2D(kernel_size=5, strides=1, filters=36, padding='same',
             activation='relu', name='layer_conv2')(net)
net = MaxPooling2D(pool_size=2, strides=2)(net)

# Flatten the output of the conv-layer from 4-dim to 2-dim.
net = Flatten()(net)

# First fully-connected / dense layer with ReLU-activation.
net = Dense(128, activation='relu')(net)

# Last fully-connected / dense layer with softmax-activation
# so it can be used for classification.
net = Dense(num_classes, activation='softmax')(net)

# Output of the Neural Network.
outputs = net


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Get the class-numbers as integers.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Get the predicted classes as One-Hot encoded arrays.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Get the predicted classes as integers.
Can you write Python code for it?

cls_pred = np.argmax(y_pred, axis=1)

plot_images(images=images,
            cls_true=cls_true,
            cls_pred=cls_pred)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Helper-function for plotting the output of a convolutional layer
Can you write Python code for it?

def plot_conv_output(values):
    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred):
    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Boolean array whether the predicted class is incorrect.
    incorrect = (cls_pred != data.test.cls)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


def plot_example_errors(cls_pred):
    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Boolean array whether the predicted class is incorrect.
    incorrect = (cls_pred != data.y_test_cls)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.x_test[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.y_test_cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Input Image
Helper-function for plotting a single image.
Can you write Python code for it?

def plot_image(image):
    plt.imshow(image.reshape(img_shape),
               interpolation='nearest',
               cmap='binary')

    plt.show()


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Loading the model is then just a single function-call, as it should be.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Model Compilation
The Neural Network has now been defined and must be finalized by adding a loss-function, optimizer and performance metrics. This is called model "compilation" in Keras.
We can either define the optimizer using a string, or if we want more control of its parameters then we need to instantiate an object. For example, we can set the learning-rate.
Can you write Python code for it?

from tensorflow.python.keras.optimizers import Adam

optimizer = Adam(lr=1e-3)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Model Compilation
We have now defined the architecture of the model with its input and output. We now have to create a Keras model and compile it with a loss-function and optimizer, so it is ready for training.
Can you write Python code for it?

from tensorflow.python.keras.models import Model


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Or we can just print the classification accuracy.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Output of Convolutional Layer
In order to show the output of a convolutional layer, we can create another Functional Model using the same input as the original model, but the output is now taken from the convolutional layer that we are interested in.
Can you write Python code for it?

output_conv2 = Model(inputs=layer_input.input,
                     outputs=layer_conv2.output)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Output of Convolutional Layer - Method 1
There are different ways of getting the output of a layer in a Keras model. This method uses a so-called K-function which turns a part of the Keras model into a function.
Can you write Python code for it?

from tensorflow.python.keras import backend as K

output_conv1 = K.function(inputs=[layer_input.input],
                          outputs=[layer_conv1.output])


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Plot an image from the test-set which will be used as an example below.
Can you write Python code for it?

image1 = data.test.images[0]
plot_image(image1)


image1 = data.x_test[0]
plot_image(image1)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Plot some of the mis-classified images.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Plot the images with their true and predicted class-numbers.
Can you write Python code for it?

plot_images(images=images,
            cls_pred=cls_pred,
            cls_true=cls_true)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Plot the weights using the helper-function from above.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Prediction
We can also predict the classification for new images. We will just use some images from the test-set but you could load your own images into numpy arrays and use those instead.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
PrettyTensor API
This is how the Convolutional Neural Network was implemented in Tutorial #03 using the PrettyTensor API. It is shown here for easy comparison to the Keras implementation below.
Can you write Python code for it?

if False:
    x_pretty = pt.wrap(x_image)

    with pt.defaults_scope(activation_fn=tf.nn.relu):
        y_pred, loss = x_pretty.\
            conv2d(kernel=5, depth=16, name='layer_conv1').\
            max_pool(kernel=2, stride=2).\
            conv2d(kernel=5, depth=36, name='layer_conv2').\
            max_pool(kernel=2, stride=2).\
            flatten().\
            fully_connected(size=128, name='layer_fc1').\
            softmax_classifier(num_classes=num_classes, labels=y_true)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Save & Load Model
NOTE: You need to install h5py for this to work!
Tutorial #04 was about saving and restoring the weights of a model using native TensorFlow code. It was an absolutely horrible API! Fortunately, Keras makes this very easy.
This is the file-path where we want to save the Keras model.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Saving a Keras model with the trained weights is then just a single function call, as it should be.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Sequential Model
The Keras API has two modes of constructing Neural Networks. The simplest is the Sequential Model which only allows for the layers to be added in sequence.
Can you write Python code for it?

# Start construction of the Keras Sequential model.
model = Sequential()

# Add an input layer which is similar to a feed_dict in TensorFlow.
# Note that the input-shape must be a tuple containing the image-size.
model.add(InputLayer(input_shape=(img_size_flat,)))

# The input is a flattened array with 784 elements,
# but the convolutional layers expect images with shape (28, 28, 1)
model.add(Reshape(img_shape_full))

# First convolutional layer with ReLU-activation and max-pooling.
model.add(Conv2D(kernel_size=5, strides=1, filters=16, padding='same',
                 activation='relu', name='layer_conv1'))
model.add(MaxPooling2D(pool_size=2, strides=2))

# Second convolutional layer with ReLU-activation and max-pooling.
model.add(Conv2D(kernel_size=5, strides=1, filters=36, padding='same',
                 activation='relu', name='layer_conv2'))
model.add(MaxPooling2D(pool_size=2, strides=2))

# Flatten the 4-rank output of the convolutional layers
# to 2-rank that can be input to a fully-connected / dense layer.
model.add(Flatten())

# First fully-connected / dense layer with ReLU-activation.
model.add(Dense(128, activation='relu'))

# Last fully-connected / dense layer with softmax-activation
# for use in classification.
model.add(Dense(num_classes, activation='softmax'))


I am working on the file "03C_Keras_API.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "03C_Keras_API.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70.000 images and class-numbers for the images. The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(data.num_train))
print("- Validation-set:\t{}".format(data.num_val))
print("- Test-set:\t\t{}".format(data.num_test))


I am working on the file "03C_Keras_API.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
The first convolutional layer has index 2.
Can you write Python code for it?

layer_conv1 = model3.layers[2]
layer_conv1


I am working on the file "03C_Keras_API.ipynb".
The first task is:
The result is a list of values, containing the loss-value and all the metrics we defined when we compiled the model. Note that 'accuracy' is now called 'acc' which is a small inconsistency.
Can you write Python code for it?

for name, value in zip(model.metrics_names, result):
    print(name, value)


for name, value in zip(model2.metrics_names, result):
    print(name, value)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
The second convolutional layer has index 4.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Then we convert the predicted class-numbers from One-Hot encoded arrays to integers.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
These are the true class-number for those images. This is only used when plotting the images.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
This creates a new model-object where we can call the typical Keras functions. To get the output of the convoloutional layer we call the predict() function with the input image.
Can you write Python code for it?

layer_output2 = output_conv2.predict(np.array([image1]))
layer_output2.shape


I am working on the file "03C_Keras_API.ipynb".
The first task is:
This gives us a 4-rank tensor.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Training
Now that the model has been fully defined with loss-function and optimizer, we can train it. This function takes numpy-arrays and performs the given number of training epochs using the given batch-size. An epoch is one full use of the entire training-set. So for 10 epochs we would iterate randomly over the entire training-set 10 times.
Can you write Python code for it?

model.fit(x=data.train.images,
          y=data.train.labels,
          epochs=1, batch_size=128)


model.fit(x=data.x_train,
          y=data.y_train,
          epochs=1, batch_size=128)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Training
The model has now been defined and compiled so it can be trained using the same fit() function as used in the Sequential Model above. This also takes numpy-arrays as input.
Can you write Python code for it?

model2.fit(x=data.x_train,
           y=data.y_train,
           epochs=1, batch_size=128)


model2.fit(x=data.train.images,
           y=data.train.labels,
           epochs=1, batch_size=128)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
Visualization of Layer Weights and Outputs
Helper-function for plotting convolutional weights
Can you write Python code for it?

def plot_conv_weights(weights, input_channel=0):
    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(weights)
    w_max = np.max(weights)

    # Number of filters used in the conv. layer.
    num_filters = weights.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = weights[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We can also get the weights for the second convolutional layer and plot them.
Can you write Python code for it?

weights_conv2 = layer_conv2.get_weights()[0]

plot_conv_weights(weights=weights_conv2, input_channel=0)


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We can also print the classification accuracy as a percentage:
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We can print all the performance metrics for the test-set.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We can then plot the images for all 36 channels.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We can then plot the output of all 16 channels of the convolutional layer.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We can then use the model again e.g. to make predictions. We get the first 9 images from the test-set and their true class-numbers.
Can you write Python code for it?

images = data.x_test[0:9]

cls_true = data.y_test_cls[0:9]


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We count the indices to get the layers we want.
The input-layer has index 0.
Can you write Python code for it?


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We need to import several things from Keras.
Can you write Python code for it?

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import InputLayer, Input
from tensorflow.keras.layers import Reshape, MaxPooling2D
from tensorflow.keras.layers import Conv2D, Dense, Flatten


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We need to import several things from Keras. Note the long import-statements. This might be a bug. Hopefully it will be possible to write shorter and more elegant lines in the future.
Can you write Python code for it?

# from tf.keras.models import Sequential  # This does not work!
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import InputLayer, Input
from tensorflow.python.keras.layers import Reshape, MaxPooling2D
from tensorflow.python.keras.layers import Conv2D, Dense, Flatten


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We need to import this Keras function for loading the model.
Can you write Python code for it?

from tensorflow.python.keras.models import load_model


I am working on the file "03C_Keras_API.ipynb".
The first task is:
We then use the restored model to predict the class-numbers for those images.
Can you write Python code for it?


I am working on the file "03_00_Intro_to_Census_API.ipynb".
The first task is:
Focusing on sf1 +2010 census
How to map out the geographical hierachy and pull out total population figures?

Nation
Regions
Divisions
State
County
Census Tract
Block Group
Census Block

Questions

What identifiers are used for these various geographic entities?
Can we get an enumeration of each of these entities?
How to figure out which census tract, block group, census block one is in?

Total Population of California
2010 Census Summary File 1
P0010001 is found in 2010 SF1 API Variables [XML] = "total population"
Can you write Python code for it?

from settings import CENSUS_KEY
import census

c=census.Census(settings.CENSUS_KEY)
c.sf1.get(('NAME', 'P0010001'), {'for': 'state:%s' % states.CA.fips})

"population of California: {0}".format(
        int(c.sf1.get(('NAME', 'P0010001'), {'for': 'state:%s' % states.CA.fips})[0]['P0010001']))


I am working on the file "03_00_Intro_to_Census_API.ipynb".
The first task is:
Let's try to get at the counties of California and their populations
Can you write Python code for it?

ca_counties = c.sf1.get(('NAME', 'P0010001'), geo={'for': 'county:*', 'in': 'state:%s' % states.CA.fips})

# create a DataFrame, convert the 'P0010001' column
# show by descending population
df = DataFrame(ca_counties)
df['P0010001'] = df['P0010001'].astype('int')
df.sort_index(by='P0010001', ascending=False)

#http://stackoverflow.com/a/13130357/7782
count,division = np.histogram(df['P0010001'])
df['P0010001'].hist(bins=division)


I am working on the file "03_00_Intro_to_Census_API.ipynb".
The first task is:
Questions to ponder: How many states are in the list? Is DC included the states list?  How to access the territories?
Formulating URL requests by hand
It's immensely useful to be able to access the census API directly but creating a URL with the proper parameters -- as well as using the census package.
Can you write Python code for it?

import requests

# get the total population of all states
url = "http://api.census.gov/data/2010/sf1?key={key}&get=P0010001,NAME&for=state:*".format(key=settings.CENSUS_KEY)

# note the structure of the response
r = requests.get(url)
print(r.content.decode('utf-8'))


I am working on the file "03_00_Intro_to_Census_API.ipynb".
The first task is:
Some Context
The US Census is complex....so it's good, even essential, to have a framing question to guide your explorations so that you don't get distracted or lost.
I got into thinking of the census in 2002 when I saw a woman I knew in the following SF Chronicle article:
Claremont-Elmwood / Homogeneity in Berkeley? Well, yeah - SFGate
I thought at that point it should be easy for regular people to do census calculations....
In the summer of 2013, I wrote the following note to Greg Wilson about diversity calculations:
notes for Greg Wilson about an example Data Science Workflow
There's a whole cottage industry in musing on "diversity" in the USA:

The Most Diverse Cities In The US - Business Insider -- using 4 categories:  Vallejo.

Most And Least Diverse Cities: Brown University Study Evaluates Diversity In The U.S.

The Top 10 Most Diverse Cities in America -- LA?

and let's not forget the Racial Dot Map and some background.
Can you write Python code for it?

#  import useful classes of pandas
import numpy as np
import pandas as pd
from pandas import Series, DataFrame, Index


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Check properties of counties_df
Can you write Python code for it?

# number of counties
assert len(counties_df) == 3143 #3143 county/county-equivs in US

# check that the total population by adding all counties == population by adding all states

assert np.sum(counties_df['P0010001']) == np.sum(states_df.P0010001)

# check we have same columns between counties_df and df
set(counties_df.columns) == set(df.columns)


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Counties
Looking at http://api.census.gov/data/2010/sf1/geo.html, we see
state-county: http://api.census.gov/data/2010/sf1?get=P0010001&for=county:*

if we want to grab all counties in one go, or you can grab counties state-by-state:
http://api.census.gov/data/2010/sf1?get=P0010001&for=county:*&in=state:06

for all counties in the state with FIPS code 06 (which is what state?)
Can you write Python code for it?

# Here's a way to use translate
# http://api.census.gov/data/2010/sf1?get=P0010001&for=county:*
# into a call using the census.Census object

r = c.sf1.get('NAME,P0010001', geo={'for':'county:*'})

# ask yourself what len(r) means and what it should be
len(r)

# let's try out one of the `census` object convenience methods
# instead of using `c.sf1.get`

r = c.sf1.state_county('NAME,P0010001',census.ALL,census.ALL)
r

# convert the json from the API into a DataFrame
# coerce to integer the P0010001 column

df = DataFrame(r)
df['P0010001'] = df['P0010001'].astype('int')

# display the first records
df.head()

# calculate the total population
# what happens when you google the number you get?

np.sum(df['P0010001'])

# often you can use dot notation to access a DataFrame column
df.P0010001.head()

# let's filter out PR -- what's the total population now
sum(df[np.in1d(df.state, states_fips)].P0010001)

# fall back to non-Pandas solution if you need to
np.sum([int(county['P0010001']) for county in r if county['state'] in states_fips])

# construct counties_df with only 50 states + DC

#counties_df = df[np.in1d(df.state, states_fips)]
counties_df = df.loc[np.in1d(df.state, states_fips)].copy()
len(counties_df)

set(counties_df.columns) == set(df.columns)


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Counties in California
Let's look at home: California state and Alameda County
Can you write Python code for it?

# boolean indexing to pull up California
states_df[states_df.NAME == 'California']

# use .ix -- most general indexing
# http://pandas.pydata.org/pandas-docs/dev/indexing.html#different-choices-for-indexing-loc-iloc-and-ix
states_df.ix['06']

# California counties

counties_df[counties_df.state=='06']

counties_df[counties_df.NAME == 'Alameda County']

counties_df[counties_df.NAME == 'Alameda County']['P0010001']


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Different ways to read off the population of Alameda County -- still looking for the best way
Can you write Python code for it?

list(counties_df[counties_df.NAME == 'Alameda County']['P0010001'].to_dict().values())[0]

list(counties_df[counties_df.NAME == 'Alameda County']['P0010001'].iteritems())[0][1]

int(counties_df[counties_df.NAME == 'Alameda County']['P0010001'].values)


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Goal
For background, see Mapping Census Data, including the
scan of the 10-question form.  Keep in mind what people were asked and the range of data available in the census.
Using the census API to get an understanding of some of the geographic entities in the 2010 census.  We'll specifically be using the variable P0010001, the total population.
What you will do in this notebook:

Sum the population of the states (or state-like entity like DC) to get the total population of the nation
Add up the counties for each state and validate the sums
Add up the census tracts for each county and validate the sums

We will make use of pandas in this notebook.
I often have the following diagram in mind to help understand the relationship among entities.  Also use the list of example URLs  -- it'll come in handy.

Working out the geographical hierarchy for Cafe Milano
It's helpful to have a concrete instance of a place to work with, especially when dealing with rather intangible entities like census tracts, block groups, and blocks.  You can use the American FactFinder site to look up for any given US address the corresponding census geographies.
Let's use Cafe Milano in Berkeley as an example. You can verify the following results by typing in the address into http://factfinder2.census.gov/faces/nav/jsf/pages/searchresults.xhtml?refresh=t.
https://www.evernote.com/shard/s1/sh/dc0bfb96-4965-4fbf-bc28-c9d4d0080782/2bd8c92a045d62521723347d62fa2b9d
2522 Bancroft Way, BERKELEY, CA, 94704

State: California
County: Alameda County
County Subdivision: Berkeley CCD, Alameda County, California
Census Tract: Census Tract 4228, Alameda County, California
Block Group: Block Group 1, Census Tract 4228, Alameda County, California
Block: Block 1001, Block Group 1,...
Can you write Python code for it?

# YouTube video I made on how to use the American Factfinder site to look up addresses
from IPython.display import YouTubeVideo
YouTubeVideo('HeXcliUx96Y')

#  standard numpy, pandas, matplotlib imports

import numpy as np
import matplotlib.pyplot as plt
from pandas import DataFrame, Series, Index
import pandas as pd

# check that CENSUS_KEY is defined
import census
import us

import requests

import settings
assert settings.CENSUS_KEY is not None


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
If states_df is calculated properly, the following asserts will pass silently.
Can you write Python code for it?

# check that we have three columns
assert set(states_df.columns) == set((u'NAME', u'P0010001', u'state'))

# check that the total 2010 census population is correct
assert np.sum(states_df.P0010001) == 308745538

# check that the number of states+DC is 51
assert len(states_df) == 51


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
If you know the FIPS code for Alameda County, just read off the population using .ix
Can you write Python code for it?

# this is like accessing a cell in a spreadsheet -- row, col

ALAMEDA_COUNTY_FIPS = '06001'

counties_df.ix[ALAMEDA_COUNTY_FIPS,'P0010001']


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Note:  we can use c.sf1 to access 2010 census (SF1: Census Summary File 1 (2010, 2000, 1990) available in API -- 2010 is the default)
see documentation: sunlightlabs/census
Summing up populations by state
Let's make a DataFrame named states_df with columns NAME, P0010001 (for population), and state (to hold the FIPS code).  Make sure to exclude Puerto Rico.
Can you write Python code for it?

# call the API and instantiate `df`
df = DataFrame(c.sf1.get('NAME,P0010001', geo={'for':'state:*'}))
# convert the population to integer
df['P0010001'] = df['P0010001'].astype(np.int)
df.head()

states_df = df[df['NAME'] != 'Puerto Rico']

'a' in ['a', 'b']


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Reading off all the tracts in Alameda County
Can you write Python code for it?

counties_df.ix[ALAMEDA_COUNTY_FIPS,'county']

# http://api.census.gov/data/2010/sf1/geo.html
# state-county-tract

geo = {'for': 'tract:*',
       'in': 'state:%s county:%s' % (us.states.CA.fips,
                                     counties_df.ix[ALAMEDA_COUNTY_FIPS,'county'])}

r = c.sf1.get('NAME,P0010001', geo=geo)

#use state_county_tract to make a DataFrame

alameda_county_tracts_df = DataFrame(r)
alameda_county_tracts_df['P0010001'] = alameda_county_tracts_df['P0010001'].astype('int')
alameda_county_tracts_df['FIPS'] = alameda_county_tracts_df.apply(lambda s: s['state']+s['county']+s['tract'], axis=1)
alameda_county_tracts_df.head()

alameda_county_tracts_df.apply(lambda s: s['state']+s['county']+s['tract'], axis=1)

alameda_county_tracts_df.P0010001.sum()

# Cafe Milano is in tract 4228
MILANO_TRACT_ID = '422800'
alameda_county_tracts_df[alameda_county_tracts_df.tract==MILANO_TRACT_ID]


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
The census documentation has example URLs but needs your API key to work.  In this notebook, we'll use the IPython notebook HTML display mechanism to help out.
Can you write Python code for it?


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Using FIPS code as the Index
From Mapping Census Data:

Each state (SUMLEV = 040) has a 2-digit FIPS ID; Delaware's is 10.
Each county (SUMLEV = 050) within a state has a 3-digit FIPS ID, appended to the 2-digit state ID. New Castle County, Delaware, has FIPS ID 10003.
Each Census Tract (SUMLEV = 140) within a county has a 6-digit ID, appended to the county code. The Tract in New Castle County DE that contains most of the the UD campus has FIPS ID 10003014502.
Each Block Group (SUMLEV = 150) within a Tract has a single digit ID appended to the Tract ID. The center of campus in the northwest corner of the tract is Block Group100030145022.
Each Block (SUMLEV = 750) within a Block Group is identified by three more digits appended to the Block Group ID. Pearson Hall is located in Block 100030145022009.
Can you write Python code for it?

# take a look at the current structure of counties_df

counties_df.head()

states_df.head()

# reindex states_df by state FIPS
# http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.set_index.html

states_df.set_index(keys='state', inplace=True)
states_df.head()

states_df.columns

# display the result of using set_index
counties_df.head()

# df.loc[np.in1d(df.state, states_fips), 'FIPS'] = counties_df.apply(lambda s:s['state']+s['county'], axis=1)

counties_df['FIPS'] = counties_df.apply(lambda s:s['state']+s['county'], axis=1)

df[np.in1d(df.state, states_fips)].head()

counties_df.head()

def double(x):
    return 2*x

counties_df.P0010001.apply(double)

# http://manishamde.github.io/blog/2013/03/07/pandas-and-python-top-10/#create

counties_df['FIPS'] = counties_df.apply(lambda s:s['state'] + s['county'], axis=1)
counties_df.set_index('FIPS', inplace=True)

counties_df.head()

counties_df.groupby('state').sum().head()

states_df.P0010001.head()

# now we're ready to compare for each state, if you add all the counties, do you get the same
# population?
# not that you can do .agg('sum') instead of .sum()
# look at http://pandas.pydata.org/pandas-docs/dev/groupby.html to learn more about agg

np.all(states_df.P0010001 == counties_df.groupby('state').agg('sum').P0010001)


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
Using Generators to yield all the tracts in the country
http://www.jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/
Can you write Python code for it?

import time
import us

from itertools import islice

def census_tracts(variable=('NAME','P0010001'), sleep_time=1.0):

    for state in us.states.STATES:
        print (state)
        for tract in c.sf1.get(variable,
                    geo={'for':"tract:*",
                        'in':'state:{state_fips}'.format(state_fips=state.fips)
                        }):
            yield tract
        # don't hit the API more than once a second
        time.sleep(sleep_time)

# limit the number of tracts we crawl for until we're reading to get all of them
tracts_df = DataFrame(list(islice(census_tracts(), 100)))
tracts_df['P0010001'] = tracts_df['P0010001'].astype('int')

tracts_df.head()


I am working on the file "03_01_Geographical_Hierarchies.ipynb".
The first task is:
You can filter Puerto Rico (PR) in a number of ways -- use the way you're most comfortable with.
Optional fun: filter PR in the following way

calculate a np.array holding the the fips of the states
then use numpy.in1d, which is a analogous to the in operator to test membership in a list
Can you write Python code for it?

states_fips = np.array([state.fips for state in us.states.STATES])
states_df = df[np.in1d(df.state,states_fips)]


I am working on the file "03_02_Displaying_Census_URLs.ipynb".
The first task is:
Scraping the examples
Can you write Python code for it?

import requests
from lxml.html import parse, fromstring

url = "http://api.census.gov/data/2010/sf1/geo.html"
r = requests.get(url).content
doc = fromstring(r)

rows = doc.xpath("//table/tr")

# first row is the header

headers = [col.text for col in rows[0].findall('th')]
headers

# next rows are the census URL examples

row = rows[1]
cols = row.findall('td')

# col[s0]:  Summmary Level

print (cols[0].text)

# cols[1]:  Description

print (cols[1].text)

from itertools import islice
from lxml.html import parse

# let's actually now decorate the urls

def decorated_parse_examples(examples, api_key=settings.CENSUS_KEY):
    for row in examples:
        new_row = row.copy()
        # need to change URLs

        example_urls_col = new_row[headers[2]]
        #urls_with_key  = [add_census_key(url) for url in example_urls_col]

        new_row[headers[2]] = "
".join(
            ["""{url}""".format(
                url=url,
                url_with_key=add_census_key(url)
                ) for url in example_urls_col
            ])

        yield new_row

def parse_urls_col(col):
    # http://stackoverflow.com/a/15074386/7782
    return [child for child in col.itertext()]

def parse_census_examples():

    url = "http://api.census.gov/data/2010/sf1/geo.html"
    doc = parse(url)

    rows = doc.xpath("//table/tr")

    # first row is the header

    headers = [col.text for col in rows[0].findall('th')]

    for row in rows[1:]:
        cols = row.findall('td')
        yield ({headers[0]:cols[0].text,
                headers[1]:cols[1].text,
                headers[2]:parse_urls_col(cols[2])})

#parsed_examples = list(islice(parse_census_examples(),None))
parsed_examples = parse_census_examples()

# let's redisplay the table with

from IPython.display import HTML
from jinja2 import Template

URLS_TEMPLATE= """

   {% for header in headers %}
     {{header}}
   {% endfor %}

   {% for row in rows %}

   ...


I am working on the file "03_AnalysisCorpora.ipynb".
The first task is:
Compare the fraction of emotional sentences per text
For the different corpora.
An emotional sentence is a sentence for which at least one HEEM label is predicted.
Can you write Python code for it?

complete.loc[:, 'frac_emotional'] = complete.apply(lambda row: (row['#emotional']+0.0)/row['#lines'], axis=1)
combined.loc[:, 'frac_emotional'] = combined.apply(lambda row: (row['#emotional']+0.0)/row['#lines'], axis=1)
annotation.loc[:, 'frac_emotional'] = annotation.apply(lambda row: (row['#emotional']+0.0)/row['#lines'], axis=1)
corpus_big.loc[:, 'frac_emotional'] = corpus_big.apply(lambda row: (row['#emotional']+0.0)/row['#lines'], axis=1)
ceneton.loc[:, 'frac_emotional'] = ceneton.apply(lambda row: (row['#emotional']+0.0)/row['#lines'], axis=1)
edbo.loc[:, 'frac_emotional'] = edbo.apply(lambda row: (row['#emotional']+0.0)/row['#lines'], axis=1)

data = [complete['frac_emotional'], combined['frac_emotional'], annotation['frac_emotional'], corpus_big['frac_emotional'], ceneton['frac_emotional'], edbo['frac_emotional']]
plt.boxplot(data)
plt.xticks([1,2,3,4,5,6],['Complete', 'Combined', 'Annotation', 'Corpus big','Ceneton','EDBO'])
plt.title('Fraction of emotional sentences in the different datasets');

from scipy import stats
import statsmodels.api as sm

f_val, p_val = stats.f_oneway(annotation['frac_emotional'], corpus_big['frac_emotional'], ceneton['frac_emotional'], edbo['frac_emotional'])
print "P value ANOVA: {:10.10f}\n".format(p_val)

annotation.loc[:, 'corpus'] = annotation.apply(lambda row: 'annotation', axis=1)
corpus_big.loc[:, 'corpus'] = corpus_big.apply(lambda row: 'corpus_big', axis=1)
ceneton.loc[:, 'corpus'] = ceneton.apply(lambda row: 'ceneton', axis=1)
edbo.loc[:, 'corpus'] = edbo.apply(lambda row: 'edbo', axis=1)

df = pd.concat([annotation, corpus_big, ceneton, edbo])

result = sm.stats.multicomp.pairwise_tukeyhsd(df.frac_emotional, df.corpus)
print(result.summary())


I am working on the file "03_AnalysisCorpora.ipynb".
The first task is:
Compare the number of lines per text
For the different corpora.
Can you write Python code for it?

data = [complete['#lines'], combined['#lines'], annotation['#lines'], corpus_big['#lines'], ceneton['#lines'], edbo['#lines']]
plt.boxplot(data)
plt.xticks([1,2,3,4,5,6],['Complete', 'Combined', 'Annotation', 'Corpus big','Ceneton','EDBO'])
plt.title('The number of lines per text in different datasets');

f_val, p_val = stats.f_oneway(annotation['#lines'], corpus_big['#lines'], ceneton['#lines'], edbo['#lines'])
print "P value ANOVA: {:10.10f}\n".format(p_val)

result = sm.stats.multicomp.pairwise_tukeyhsd(df.get('#lines'), df.corpus)
print(result.summary())


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Inorder Traversal

https://leetcode.com/problems/binary-tree-inorder-traversal/#/description
Given a binary tree, return the inorder traversal of its nodes' values.
For example:
Given binary tree [1,null,2,3],
   1
    \
     2
    /
   3
return [1,3,2].
Note: Recursive solution is trivial, could you do it iteratively?
Can you write Python code for it?

# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def inorderTraversal(self, root):
        """
        :type root: TreeNode
        :rtype: List[int]
        """
        ans = []
        if root is None:
            return ans

        # iterative
        stack = []
        while True:
            while root:
                stack.append(root)
                root = root.left
            if not stack:
                return ans
            node = stack.pop()
            ans.append(node.val)
            root = node.right

        # recursive
        return self.inorderTraversal(root.left) + [root.val] + self.inorderTraversal(root.right)


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Longest Consecutive Sequence
http://www.lintcode.com/en/problem/binary-tree-longest-consecutive-sequence/
Can you write Python code for it?

# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution:
    # @param {TreeNode} root the root of binary tree
    # @return {int} the length of the longest consecutive sequence path
    def longestConsecutive(self, root):
        # Write your code here
        start, longest = self.helper(root)
        return longest

    def helper(self, root):
        if root is None:
            return 0, 0

        left_start, left_longest = self.helper(root.left)
        right_start, right_longest = self.helper(root.right)

        if root.left and root.val+1 == root.left.val:
            if left_start:
                left_start += 1
            else:
                left_start = 2
            left_longest = max(left_start, left_longest)
        else:
            left_start = 0

        if root.right and root.val+1 == root.right.val:
            if right_start:
                right_start += 1
            else:
                right_start = 2
            right_longest = max(right_start, right_longest)
        else:
            right_start = 0

        return max(left_start, right_start), max(left_longest, right_longest, 1)

# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution:
    # @param {TreeNode} root the root of binary tree
    # @return {int} the length of the longest consecutive sequence path
    longest = 0
    def longestConsecutive(self, root):
        # Write your code here
        self.helper(root)
        return self.longest

    def helper(self, root):
        if root is None:
            return 0

        left = self.helper(root.left)
        right = self.helper(root.right)
        sublongest = 1

        if root.left and root.val + 1 == root.left.val:
  ...


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Longest Consecutive Sequence III

It's follow up problem for Binary Tree Longest Consecutive Sequence II
Given a k-ary tree, find the length of the longest consecutive sequence path.
The path could be start and end at any node in the tree
Can you write Python code for it?

# Definition for a multi tree node.
# class MultiTreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         children = [] # children is a list of MultiTreeNode

class Solution:
    # @param {MultiTreeNode} root the root of k-ary tree
    # @return {int} the length of the longest consecutive sequence path
    def longestConsecutive3(self, root):
        # Write your code here
        longest, _, _ = self.helper(root)
        return longest

    def helper(self, root):
        if root is None:
            return 0, 0, 0

        len, down, up = 0, 0, 0
        for child in root.children:
            clen, cdown, cup = self.helper(child)
            if child.val + 1 == root.val:
                down = max(down, cdown + 1)
            if child.val - 1 == root.val:
                up = max(up, cup + 1)
            len = max(len, clen)

        len = max(len, down + up + 1)

        return len, down, up


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Path Sum
Given a binary tree, find all paths that sum of the nodes in the path equals to a given number target.
A valid path is from root node to any of the leaf nodes.
https://www.lintcode.com/en/problem/binary-tree-path-sum/
https://leetcode.com/problems/path-sum-ii/#/description
Can you write Python code for it?

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        self.val = val
        self.left, self.right = None, None
"""
class Solution:
    # @param {TreeNode} root the root of binary tree
    # @param {int} target an integer
    # @return {int[][]} all valid paths
    def binaryTreePathSum(self, root, target):
        # Write your code here

        if root is None:
            return []

        if root.left is None and root.right is None and root.val == target:
            return [[root.val]]

        left = self.binaryTreePathSum(root.left, target - root.val)
        right = self.binaryTreePathSum(root.right, target - root.val)

        paths = []
        for p in left:
            paths += [[root.val] + p]
        for p in right:
            paths += [[root.val] + p]
        return paths


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Path Sum II

Your are given a binary tree in which each node contains a value. Design an algorithm to get all paths which sum to a given value. The path does not need to start or end at the root or a leaf, but it must go in a straight line down.
http://www.lintcode.com/en/problem/binary-tree-path-sum-ii/
Can you write Python code for it?

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        self.val = val
        self.left, self.right = None, None
"""
class Solution:
    # @param {TreeNode} root the root of binary tree
    # @param {int} target an integer
    # @return {int[][]} all valid paths
    def binaryTreePathSum2(self, root, target):
        # Write your code here
        path, result = [], []
        self.dfs(root, path, result, 0, target)
        return result

    def dfs(self, root, path, result, l, target):
        if root is None:
            return
        path.append(root.val)
        temp = target
        for i in xrange(l, -1, -1):
            temp -= path[i]
            if temp == 0:
                result.append(path[i:])

        self.dfs(root.left, path, result, l + 1, target)
        self.dfs(root.right, path, result, l + 1, target)

        path.pop()