def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.y_test_cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.x_test[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.y_test_cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Helper-functions for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Initialize variables
The variables for the TensorFlow graph must be initialized before we start optimizing them.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from mnist import MNIST
data = MNIST(data_dir="data/MNIST/")


from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Loss-Function to be Optimized
To make the model better at classifying the input images, we must somehow change the variables of the neural network.
The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the variables of the model.
TensorFlow has a function for calculating the cross-entropy, which uses the values of the logits-layer because it also calculates the softmax internally, so as to to improve numerical stability.
Can you write Python code for it?

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=logits)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Neural Network
We now implement the Convolutional Neural Network using the Layers API. We use the net-variable to refer to the last layer while building the neural network. This makes it easy to add or remove layers in the code if you want to experiment. First we set the net-variable to the reshaped input image.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable x. The shape of this placeholder variable is [None, num_classes] which means it may hold an arbitrary number of labels and each label is a vector of length num_classes which is 10 in this case.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Note how the predicted class indeed becomes 2 already within the first few iterations so the optimization is working as intended. Also note how the loss-measure is increasing rapidly until it apparently converges. This is because the loss-measure is actually just the value of the feature or neuron that we are trying to maximize. Because this is the logits-layer prior to the softmax, these values can potentially be infinitely high, but they are limited because we limit the image-values between 0 and 1.
Now plot the image that was found. This is the image that the neural network believes looks most like the digit 2.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Note how these are very simple shapes such as lines and angles. Some of these images may be completely white, which suggests that those features of the neural network are perhaps unused, so the number of features could be reduced in this layer.
Second Convolutional Layer
This shows the images that maximize the features or neurons in the second convolutional layer, so these are the input images it likes to see. Note how these are more complex lines and patterns compared to the first convolutional layer.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Optimization Method
Now that we have a cost measure that must be minimized, we can then create an optimizer. In this case it is the Adam optimizer with a learning-rate of 1e-4.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Optimize the Neural Network
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Performance after 10,000 optimization iterations
After 10,000 optimization iterations, the model has a classification accuracy on the test-set of about 99%.
Can you write Python code for it?

%%time
optimize(num_iterations=10000)

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the variables for the neural network have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


# Get the first images from the test-set.
images = data.x_test[0:9]

# Get the true classes for those images.
cls_true = data.y_test_cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-rank tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
The input image is then input to the first convolutional layer, which has 16 filters each of size 5x5 pixels. The activation-function is the Rectified Linear Unit (ReLU) described in more detail in Tutorial #02.
Can you write Python code for it?

net = tf.layers.conv2d(inputs=net, name='layer_conv1', padding='same',
                       filters=16, kernel_size=5, activation=tf.nn.relu)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
The output then needs to be flattened so it can be used in fully-connected (aka. dense) layers.
Can you write Python code for it?

net = tf.layers.flatten(net)

# This should eventually be replaced by:
# net = tf.layers.flatten(net)


net = tf.contrib.layers.flatten(net)

# This should eventually be replaced by:
# net = tf.layers.flatten(net)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
The outputs of the final fully-connected layer are sometimes called logits, so we have a convenience variable with that name which we will also use further below.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
Then we make a second convolutional layer, also with max-pooling.
Can you write Python code for it?

net = tf.layers.conv2d(inputs=net, name='layer_conv2', padding='same',
                       filters=36, kernel_size=5, activation=tf.nn.relu)

net = tf.layers.max_pooling2d(inputs=net, pool_size=2, strides=2)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
These images may vary each time you run the optimization. Some of the images can be seen to somewhat resemble the hand-written digits. But the other images are often impossible to recognize and it is hard to understand why the neural network thinks these are the optimal input images for those digits.
The reason is perhaps that the neural network tries to recognize all digits simultaneously, and it has found that certain pixels often determine whether the image shows one digit or another. So the neural network has learned to differentiate those pixels that it has found to be important, but not the underlying curves and shapes of the digits, in the same way that a human recognizes the digits.
Another possibility is that the data-set contains mis-classified digits which may confuse the neural network during training. We have previously seen how some of the digits in the data-set are very hard to read even for humans, and this may cause the neural network to become distorted and trying to recognize strange artifacts in the images.
Yet another possibility is that the optimization process has stagnated in a local optimum. One way to test this, would be to run the optimization 50 times for the digits that are unclear, and see if some of the resulting images become more clear.
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
This function performs a number of optimization iterations so as to gradually improve the variables of the neural network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch, _ = data.random_batch(batch_size=train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations


# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
This tells us how likely the neural network thinks the input image is of each possible class. The one that has the highest value is considered the most likely so its index is taken to be the class-number.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
This was developed using Python 3.6 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
We can now add fully-connected (or dense) layers to the neural network.
Can you write Python code for it?

net = tf.layers.dense(inputs=net, name='layer_fc1',
                      units=128, activation=tf.nn.relu)


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the image classifications.
Can you write Python code for it?


I am working on the file "13B_Visual_Analysis_MNIST.ipynb".
The first task is:
We need the neural network to classify the input images into 10 different classes. So the final fully-connected layer has num_classes=10 output neurons.
Can you write Python code for it?

net = tf.layers.dense(inputs=net, name='layer_fc_out',
                      units=num_classes, activation=None)


I am working on the file "13_Statistics.ipynb".
The first task is:
Averages and variances
Q5. Compute the median of flattened x.
Can you write Python code for it?

x = np.arange(1, 10).reshape((3, 3))
print("x=\n", x)


I am working on the file "13_Statistics.ipynb".
The first task is:
Correlating
Q8. Compute the covariance matrix of x and y.
Can you write Python code for it?


I am working on the file "13_Statistics.ipynb".
The first task is:
Histograms
Q12. Compute the histogram of x against the bins.
Can you write Python code for it?

x = np.array([0.5, 0.7, 1.0, 1.2, 1.3, 2.1])
bins = np.array([0, 1, 2, 3])
print("ans=\n", ...)

import matplotlib.pyplot as plt
%matplotlib inline
plt.hist(x, bins=bins)
plt.show()


I am working on the file "13_Statistics.ipynb".
The first task is:
Order statistics
Q1. Return the minimum value of x along the second axis.
Can you write Python code for it?

x = np.arange(4).reshape((2, 2))
print("x=\n", x)


I am working on the file "13_Statistics.ipynb".
The first task is:
Q13. Compute the 2d histogram of x and y.
Can you write Python code for it?


I am working on the file "13_Statistics.ipynb".
The first task is:
Q3. Calcuate the difference between the maximum and the minimum of x along the second axis.
Can you write Python code for it?

x = np.arange(10).reshape((2, 5))
print("x=\n", x)


I am working on the file "13_Statistics.ipynb".
The first task is:
Q4. Compute the 75th percentile of x along the second axis.
Can you write Python code for it?

x = np.arange(1, 11).reshape((2, 5))
print("x=\n", x)


I am working on the file "13_Statistics.ipynb".
The first task is:
Q6. Compute the weighted average of x.
Can you write Python code for it?

x = np.arange(5)
weights = np.arange(1, 6)


I am working on the file "13_Statistics.ipynb".
The first task is:
Q7. Compute the mean, standard deviation, and variance of x along the second axis.
Can you write Python code for it?


I am working on the file "13_Statistics.ipynb".
The first task is:
Q9. In the above covariance matrix, what does the -1 mean?
Q10. Compute Pearson product-moment correlation coefficients of x and y.
Can you write Python code for it?


I am working on the file "13_Superposed_deformation.ipynb".
The first task is:
$$\mathbf{F} = \underset{n \mathrm{\, times}}{\mathbf{F_i}\mathbf{F_i}\cdots\mathbf{F_i}} = \mathbf{F_i}^n$$
Can you write Python code for it?


I am working on the file "13_Superposed_deformation.ipynb".
The first task is:
Now we calculate axial ratios of accumulated pure shear
Can you write Python code for it?

PSar = [axialratio(matrix_power(PSi, n)) for n in nrange]
SSar = [axialratio(matrix_power(SSi, n)) for n in nrange]

plot(nrange, PSar, label='pure shear')
plot(nrange, SSar, label='simple shear')
plt.legend(loc=2)


I am working on the file "13_Superposed_deformation.ipynb".
The first task is:
Now we can visualize accumulation of deformation for pure shear and simple shear increments. Firstly we will defined range of increments to be plotted.
Can you write Python code for it?


I am working on the file "13_Superposed_deformation.ipynb".
The first task is:
Small deformations
Using above described concept, we can imagine that any deformation is composed from small deformation increments superposed on each other. So lets demonstrate how small increments of pure shear and simple shear accumulate with increasing amounts of steps. For this purpose we define incremental simple shear with $\gamma=0.01$ and pure shear increment with $S_x=1.01$ and we will use numpy.linalg.matrix_power function to calculate total deformation.
Can you write Python code for it?


I am working on the file "13_Superposed_deformation.ipynb".
The first task is:
To visualize change of angle between long axis of strain ellipse and shear zone boundary during simple shear, we can use similar approach:
Can you write Python code for it?

nrange = arange(1, 1000)
ang = [orientation(matrix_power(SSi, n)) for n in nrange]

plot(nrange, ang)

from IPython.core.display import HTML
def css_styling():
    styles = open("./css/sg2.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "13_Superposed_deformation.ipynb".
The first task is:
for simple shear accumulated simple shear growths linearly with amounts of steps.
To see how intensity of deformation i.e. axial ratio of strain ellipse evolves with accumulation of pure shear or simple shear, we have to understand how to calculate axial ration of strain ellipse from deformation gradient.
In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix $\mathbf{F} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^$. For deformation gradient, where $\mathbf{F}$ is an m × m real square matrix with positive determinant, $\mathbf{U}$, $\mathbf{V}^$ and $\boldsymbol{\Sigma}$ are real m × m matrices as well, $\boldsymbol{\Sigma}$ can be regarded as a scaling matrix, and $\mathbf{U}$, $\mathbf{V}^$ can be viewed as rotation matrices. Thus the expression $\mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^$ can be intuitively interpreted as a composition (or sequence) of three transformations: a rotation, a scaling, and another rotation. For instance, the figure above explains how a shear matrix can be described as such a sequence.
<img width="85%" src="images/svd_explained.png">
Knowing that and using numpy.linalg.svd function, we can define functions to calculate axial ratio or orientation of strain ellipse
Can you write Python code for it?

# function to calculate axial ratio of strain ellipse
def axialratio(F):
    u,s,v = svd(F)
    return s[0]/s[1]
# function to calculate orientation of long axis of the strain ellipse (counterclockwise from horizontal)
def orientation(F):
    u,s,v = svd(F)
    return rad2deg(arctan2(u[1,0], u[0,0]))

F = array([[1, 1], [0, 1]])

axialratio(F)

orientation(F)


I am working on the file "13_camera_application.ipynb".
The first task is:
How about some text on the image.
Can you write Python code for it?

camera.annotate_foreground = Color(1.0,1.0,0.5)

camera.resolution = (120, 120)

camera.annotate_text = ""
camera.annotate_text_size = 36

camera.start_preview()
sleep(2)
camera.capture('./img/cubeU.jpg')
camera.stop_preview()

camera.start_preview()
sleep(2)
camera.capture('./img/cubeR.jpg')
camera.stop_preview()

camera.start_preview()
sleep(2)
camera.capture('./img/cubeF.jpg')
camera.stop_preview()

camera.start_preview()
sleep(2)
camera.capture('./img/cubeD.jpg')
camera.stop_preview()

camera.start_preview()
sleep(2)
camera.capture('./img/cubeL.jpg')
camera.stop_preview()

camera.start_preview()
sleep(2)
camera.capture('./img/cubeB.jpg')
camera.stop_preview()


I am working on the file "13_camera_application.ipynb".
The first task is:
Now, how about red vs green vs blue (a 3d cross plot)?
Can you write Python code for it?

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(square_frame.r, square_frame.g, square_frame.b, c='gray', marker='o')

ax.set_xlabel('red')
ax.set_ylabel('green')
ax.set_zlabel('blue')

X = square_frame[['r', 'g', 'b']]

km = KMeans(n_clusters=6, random_state=123)
km.fit(X)

km.cluster_centers_

centers_list = []
center_rgb = square_frame[(square_frame['x']==1) & (square_frame['y']==1)]
print center_rgb

square_mapper = {}
center_map = km.predict(center_rgb[['r', 'g', 'b']])
for i in range(6):
    square_mapper[center_map[i]] = face_order[i]

center_map

square_mapper

square_predict = km.predict(square_frame[['r', 'g', 'b']])

squares = []
for square in square_predict:
    squares.append(square_mapper[square])

square_frame['predict'] = squares

square_color_map = {'U': 'black', 'R': 'red', 'F': 'green', 'D': 'yellow', 'L': 'orange', 'B': 'blue'}

square_colors = [square_color_map[sq] for sq in squares]

square_frame['colors'] = square_colors

square_frame[:15]

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(square_frame.r, square_frame.g, square_frame.b, c=square_frame.colors, marker='o')

ax.set_xlabel('red')
ax.set_ylabel('green')
ax.set_zlabel('blue')

import kociemba

kociemba.solve('URRDUUDRDRLBFRBBDUFBBBFLBRLLDUDDBDUFFLRRLUFFDUFLLBURFL')


I am working on the file "13_camera_application.ipynb".
The first task is:
Once images are captured, let's try to get a 'color' from each square
Can you write Python code for it?

face_order = 'URFDLB'

face_images = {}
face_images_out = []
squares = {}
masks = {}  # for QA

for face in face_order:
    # read and cache images in dict
    face_images[face] = (imread('./img/cube%s.jpg' % face))

    img = face_images[face]
    mask = np.empty(img.shape[:2], dtype=np.bool)
    mask[::]=False
    row_coords = [(25, 30), (50, 55), (75, 80)]
    col_coords = [(38, 43), (65, 70), (95, 100)]
    squares[face] = np.zeros((3, 3, 3))
    row = 0
    # extract average RGB values from approximate square centers
    for i in row_coords:
        col = 0
        for j in col_coords:
            mask[i[0]:i[1], j[0]:j[1]] = True
            squares[face][row, col] = (np.average(img[i[0]:i[1], j[0]:j[1], 0]),
                                 np.average(img[i[0]:i[1], j[0]:j[1], 1]),
                                 np.average(img[i[0]:i[1], j[0]:j[1], 2]))
            col+=1
        row+=1
    # to show last image alignment
    masks[face] = mark_boundaries(img, mask)

fig_mask = plt.figure()
ax = fig_mask.add_subplot(1,1,1)
ax.imshow(masks['D'])
plt.axis("off")

# this basically puts all the average color values for each square into a big table, so we can do some math
square_list = []
for face in face_order:
    for x in range(3):
        for y in range(3):
            r, g, b = squares[face][x,y]
            square_list.append([face, x, y, r, g, b])

columns = ['face', 'x', 'y', 'r', 'g', 'b']
square_frame = pd.DataFrame(square_list, columns=columns)

square_frame.head(10)


I am working on the file "13_camera_application.ipynb".
The first task is:
Our task is to assign each square a 'color.'  To do this, we try to group them (since the computer doesn't really know what 'color' is.
Looking at the colors on a plot, say red vs blue, we get this:
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(square_frame.r, square_frame.b, c='gray', marker='o')

ax.set_xlabel('red')
ax.set_ylabel('blue')


I am working on the file "13_camera_application.ipynb".
The first task is:
TADA ... wait, nothing happened.
Can you write Python code for it?

camera.hflip = True

camera.vflip = True

camera.brightness = 50 # the default is 50, but you can set it to whatever.


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
Options Modeling
We model a certain number of derivative instruments with the following major assumptions.
Can you write Python code for it?

me_option = market_environment('option', me.pricing_date)
# choose from a set of maturity dates (month ends)
maturities = pd.date_range(start=me.pricing_date,
                           end=val_env.get_constant('final_date'),
                           freq='M').to_pydatetime()
me_option.add_constant('maturity', np.random.choice(maturities))
me_option.add_constant('currency', 'EUR')
me_option.add_environment(val_env)


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
Risk Analysis
Full distribution of portfolio present values illustrated via histogram.
Can you write Python code for it?

%time pvs = port.get_present_values()

plt.figure(figsize=(10, 6))
plt.hist(pvs, bins=30);
plt.xlabel('portfolio present values')
plt.ylabel('frequency')


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
Sample paths for three underlyings.
Can you write Python code for it?

paths_0 = list(port.underlying_objects.values())[0]
paths_0.generate_paths()
paths_1 = list(port.underlying_objects.values())[1]
paths_1.generate_paths()
paths_2 = list(port.underlying_objects.values())[2]
paths_2.generate_paths()

pa = 5
plt.figure(figsize=(10, 6))
plt.plot(port.time_grid, paths_0.instrument_values[:, :pa], 'b');
print('Paths for %s (blue)' % paths_0.name)
plt.plot(port.time_grid, paths_1.instrument_values[:, :pa], 'r.-');
print ('Paths for %s (red)' % paths_1.name)
plt.plot(port.time_grid, paths_2.instrument_values[:, :pa], 'g-.', lw=2.5);
print('Paths for %s (green)' % paths_2.name)
plt.ylabel('risk factor level')
plt.gcf().autofmt_xdate()


paths_0 = port.underlying_objects.values()[0]
paths_0.generate_paths()
paths_1 = port.underlying_objects.values()[1]
paths_1.generate_paths()
paths_2 = port.underlying_objects.values()[2]
paths_2.generate_paths()

pa = 5
plt.figure(figsize=(10, 6))
plt.plot(port.time_grid, paths_0.instrument_values[:, :pa], 'b');
print 'Paths for %s (blue)' % paths_0.name
plt.plot(port.time_grid, paths_1.instrument_values[:, :pa], 'r.-');
print 'Paths for %s (red)' % paths_1.name
plt.plot(port.time_grid, paths_2.instrument_values[:, :pa], 'g-.', lw=2.5);
print 'Paths for %s (green)' % paths_2.name
plt.ylabel('risk factor level')
plt.gcf().autofmt_xdate()


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
Some statistics via pandas.
Can you write Python code for it?


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
The call of the get_values method to value all instruments.
Can you write Python code for it?

%time res = port.get_statistics(fixed_seed=True)

res.set_index('position', inplace=False)


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
The vega risk report.
Can you write Python code for it?

%%time
vegas, benchmark = port.get_port_risk(Greek='Vega', fixed_seed=True, step=0.2,
                                      risk_factors=list(risk_factors.keys())[:3])
risk_report(vegas)


%%time
vegas, benchmark = port.get_port_risk(Greek='Vega', fixed_seed=True, step=0.2,
                                      risk_factors=risk_factors.keys()[:3])
risk_report(vegas)


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
Using 2,500 paths and monthly discretization for the example.
Can you write Python code for it?

# valuation environment
val_env = market_environment('val_env', dt.datetime(2015, 1, 1))
val_env.add_constant('paths', 1000)
val_env.add_constant('frequency', 'M')
val_env.add_curve('discount_curve', ssr)
val_env.add_constant('starting_date', dt.datetime(2015, 1, 1))
val_env.add_constant('final_date', dt.datetime(2016, 12, 31))

# add valuation environment to market environments
me.add_environment(val_env)

no = 50  # 50 different risk factors in total

risk_factors = {}
for rf in range(no):
    # random model choice
    sm = np.random.choice(['gbm', 'jd', 'sv'])
    key = '%3d_%s' % (rf + 1, sm)
    risk_factors[key] = market_environment(key, me.pricing_date)
    risk_factors[key].add_environment(me)
    # random initial_value
    risk_factors[key].add_constant('initial_value',
                                    np.random.random() * 40. + 20.)
    # radnom volatility
    risk_factors[key].add_constant('volatility',
                                    np.random.random() * 0.6 + 0.05)
    # the simulation model to choose
    risk_factors[key].add_constant('model', sm)

correlations = []
keys = sorted(risk_factors.keys())
for key in keys[1:]:
    correlations.append([keys[0], key, np.random.choice([-0.1, 0.0, 0.1])])
correlations[:3]


# valuation environment
val_env = market_environment('val_env', dt.datetime(2015, 1, 1))
val_env.add_constant('paths', 2500)
val_env.add_constant('frequency', 'M')
val_env.add_curve('discount_curve', ssr)
val_env.add_constant('starting_date', dt.datetime(2015, 1, 1))
val_env.add_constant('final_date', dt.datetime(2016, 12, 31))

# add valuation environment to market environments
me.add_environment(val_env)

no = 50  # 50 different risk factors in total

risk_factors = {}
for rf in range(no):
    # random model choice
    sm = np.random.choice(['gbm', 'jd', 'sv'])
    key = '%3d_%s' % (rf + 1, sm)
    risk_factors[key] = market_environment(key, me.pricing_date)
    risk_factors[key].add_environment(me)
    # random initial_value
    risk_factors[key].add_constant('initial_value',
                                    np.random.random() * 40. + 20.)
    # radnom volatility
    risk_factors[key].add_constant('volatility',
                                    np.random.random() * 0.6 + 0.05)
    # the simulation model to choose
    risk_factors[key].add_constant('model', sm)

correlations = []
keys = sorted(risk_factors.keys())
for key in keys[1:]:
    correlations.append([keys[0], key, np.random.choice([-0.1, 0.0, 0.1])])
correlations[:3]


I am working on the file "13_dx_quite_complex_portfolios.ipynb".
The first task is:
Visualization of Results
Selected results visualized.
Can you write Python code for it?

res[['pos_value', 'pos_delta', 'pos_vega']].hist(bins=30, figsize=(9, 6))
plt.ylabel('frequency')


I am working on the file "13_image_processing.ipynb".
The first task is:
So, let's compare this to another image:
Can you write Python code for it?

test_image2 = imread("img/frame-1.jpg", as_grey=True)

p2 = figure(plot_width=480, plot_height=320, x_range=(0, 10), y_range=(0, 10))
p2.image(image=[test_image2[::-1]], x=[0], y=[0], dw=[10], dh=[10])

show(p2)

test_difference = test_image - test_image2

p3 = figure(plot_width=480, plot_height=320, x_range=(0, 10), y_range=(0, 10))
p3.image(image=[test_difference[::-1]], x=[0], y=[0], dw=[10], dh=[10])

show(p3)


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
2.2 Decoder
The decoder is probably the most involved part of this model. The following steps are needed to create it:
1- Process decoder inputs
2- Set up the decoder components
    - Embedding
    - Decoder cell
    - Dense output layer
    - Training decoder
    - Inference decoder

Process Decoder Input
In the training process, the target sequences will be used in two different places:

Using them to calculate the loss
Feeding them to the decoder during training to make the model more robust.

Now we need to address the second point. Let's assume our targets look like this in their letter/word form (we're doing this for readibility. At this point in the code, these sequences would be in int form):

We need to do a simple transformation on the tensor before feeding it to the decoder:
1- We will feed an item of the sequence to the decoder at each time step. Think about the last timestep -- where the decoder outputs the final word in its output. The input to that step is the item before last from the target sequence. The decoder has no use for the last item in the target sequence in this scenario. So we'll need to remove the last item.
We do that using tensorflow's tf.strided_slice() method. We hand it the tensor, and the index of where to start and where to end the cutting.

2- The first item in each sequence we feed to the decoder has to be GO symbol. So We'll add that to the beginning.

Now the tensor is ready to be fed to the decoder. It looks like this (if we convert from ints to letters/symbols):

Can you write Python code for it?

# Process the input we'll feed to the decoder
def process_decoder_input(target_data, vocab_to_int, batch_size):
    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''
    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])
    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)

    return dec_input


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
Get Batches
There's little processing involved when we retreive the batches. This is a simple example assuming batch_size = 2
Source sequences (it's actually in int form, we're showing the characters for clarity):

Target sequences (also in int, but showing letters for clarity):

Can you write Python code for it?

def pad_sentence_batch(sentence_batch, pad_int):
    """Pad sentences with <PAD> so that each sentence of a batch has the same length"""
    max_sentence = max([len(sentence) for sentence in sentence_batch])
    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]

def get_batches(targets, sources, batch_size, source_pad_int, target_pad_int):
    """Batch targets, sources, and the lengths of their sentences together"""
    for batch_i in range(0, len(sources)//batch_size):
        start_i = batch_i * batch_size
        sources_batch = sources[start_i:start_i + batch_size]
        targets_batch = targets[start_i:start_i + batch_size]
        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))
        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))

        # Need the lengths for the _lengths parameters
        pad_targets_lengths = []
        for target in pad_targets_batch:
            pad_targets_lengths.append(len(target))

        pad_source_lengths = []
        for source in pad_sources_batch:
            pad_source_lengths.append(len(source))

        yield pad_targets_batch, pad_sources_batch, pad_targets_lengths, pad_source_lengths


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
Input
Can you write Python code for it?

def get_model_inputs():
    input_data = tf.placeholder(tf.int32, [None, None], name='input')
    targets = tf.placeholder(tf.int32, [None, None], name='targets')
    lr = tf.placeholder(tf.float32, name='learning_rate')

    target_sequence_length = tf.placeholder(tf.int32, (None,), name='target_sequence_length')
    max_target_sequence_length = tf.reduce_max(target_sequence_length, name='max_target_len')
    source_sequence_length = tf.placeholder(tf.int32, (None,), name='source_sequence_length')

    return input_data, targets, lr, target_sequence_length, max_target_sequence_length, source_sequence_length


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
Model outputs training_decoder_output and inference_decoder_output both contain a 'rnn_output' logits tensor that looks like this:
<img src="images/logits.png"/>
The logits we get from the training tensor we'll pass to tf.contrib.seq2seq.sequence_loss() to calculate the loss and ultimately the gradient.
Can you write Python code for it?

# Build the graph
train_graph = tf.Graph()
# Set the graph to default to ensure that it is ready for training
with train_graph.as_default():

    # Load the model inputs
    input_data, targets, lr, target_sequence_length, max_target_sequence_length, source_sequence_length = get_model_inputs()

    # Create the training and inference logits
    training_decoder_output, inference_decoder_output = seq2seq_model(input_data,
                                                                      targets,
                                                                      lr,
                                                                      target_sequence_length,
                                                                      max_target_sequence_length,
                                                                      source_sequence_length,
                                                                      len(source_letter_to_int),
                                                                      len(target_letter_to_int),
                                                                      encoding_embedding_size,
                                                                      decoding_embedding_size,
                                                                      rnn_size,
                                                                      num_layers)

    # Create tensors for the training logits and inference logits
    training_logits = tf.identity(training_decoder_output.rnn_output, 'logits')
    inference_logits = tf.identity(inference_decoder_output.sample_id, 'predictions')

    # Create the weights for sequence_loss
    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length,
                             dtype=tf.float32, name='masks')

    with tf.name_scope("optimization"):

        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(training_logits, targets, masks)

        # Optimizer
      ...


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
Prediction
Can you write Python code for it?

def source_to_seq(text):
    '''Prepare the text for the model'''
    sequence_length = 7
    return [source_letter_to_int.get(word, source_letter_to_int['<UNK>']) for word in text]+ [source_letter_to_int['<PAD>']]*(sequence_length-len(text))

input_sentence = 'prejudice'
text = source_to_seq(input_sentence)

checkpoint = "./best_model.ckpt"

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(checkpoint + '.meta')
    loader.restore(sess, checkpoint)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('predictions:0')
    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')
    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')

    #Multiply by batch_size to match the model's input parameters
    answer_logits = sess.run(logits, {input_data: [text]*batch_size,
                                      target_sequence_length: [len(text)]*batch_size,
                                      source_sequence_length: [len(text)]*batch_size})[0]

pad = source_letter_to_int["<PAD>"]

print('Original Text:', input_sentence)

print('\nSource')
print('  Word Ids:    {}'.format([i for i in text]))
print('  Input Words: {}'.format(" ".join([source_int_to_letter[i] for i in text])))

print('\nTarget')
print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))
print('  Response Words: {}'.format(" ".join([target_int_to_letter[i] for i in answer_logits if i != pad])))


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
Preprocess
To do anything useful with it, we'll need to turn the each string into a list of characters:
<img src="images/source_and_target_arrays.png"/>
Then convert the characters to their int values as declared in our vocabulary:
Can you write Python code for it?

def extract_character_vocab(data):
    special_words = ['<PAD>', '<UNK>', '<GO>',  '<EOS>']

    set_words = set([character for line in data.split('\n') for character in line])
    int_to_vocab = {word_i: word for word_i, word in enumerate(special_words + list(set_words))}
    vocab_to_int = {word: word_i for word_i, word in int_to_vocab.items()}

    return int_to_vocab, vocab_to_int

# Build int2letter and letter2int dicts
source_int_to_letter, source_letter_to_int = extract_character_vocab(source_sentences)
target_int_to_letter, target_letter_to_int = extract_character_vocab(target_sentences)

# Convert characters to ids
source_letter_ids = [[source_letter_to_int.get(letter, source_letter_to_int['<UNK>']) for letter in line] for line in source_sentences.split('\n')]
target_letter_ids = [[target_letter_to_int.get(letter, target_letter_to_int['<UNK>']) for letter in line] + [target_letter_to_int['<EOS>']] for line in target_sentences.split('\n')]

print("Example source sequence")
print(source_letter_ids[:3])
print("\n")
print("Example target sequence")
print(target_letter_ids[:3])


I am working on the file "13_seq2seq/sequence_to_sequence_implementation.ipynb".
The first task is:
Sequence to Sequence Model
We can now start defining the functions that will build the seq2seq model. We are building it from the bottom up with the following components:
2.1 Encoder
2.2 Decoder
2.3 Seq2seq model connecting the encoder and decoder
2.4 Build the training graph hooking up the model with the
    optimizer

2.1 Encoder
The first bit of the model we'll build is the encoder. Here, we'll embed the input data, construct our encoder, then pass the embedded data to the encoder.

Embed the input data using tf.contrib.layers.embed_sequence

Pass the embedded input into a stack of RNNs.  Save the RNN state and ignore the output.

Can you write Python code for it?

def encoding_layer(input_data, rnn_size, num_layers,
                   source_sequence_length, source_vocab_size,
                   encoding_embedding_size):

    # Encoder embedding
    enc_embed_input = tf.contrib.layers.embed_sequence(input_data,
                                                       source_vocab_size,
                                                       encoding_embedding_size)
    # RNN cell
    def make_cell(rnn_size):
        return tf.contrib.rnn.LSTMCell(rnn_size,
                                       initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))

    enc_cell = tf.contrib.rnn.MultiRNNCell([make_cell(rnn_size) for _ in range(num_layers)])

    enc_output, enc_state = tf.nn.dynamic_rnn(enc_cell, enc_embed_input,
                                              sequence_length=source_sequence_length, dtype=tf.float32)
    return enc_output, enc_state


I am working on the file "14--pagerank-partial-solns2.ipynb".
The first task is:
Exercise. Check your result by first inserting the final computed PageRank vector back into the database, and then using a SQL query to see the ranked URLs. In your query output, also include both the in-degrees and out-degrees of each vertex.
Can you write Python code for it?

# Write some code here to create a table in the database
# called PageRank. It should have one column to hold the
# page (vertex) ID, and one for the rank value.

# Some helper code to compute a view containing the indegrees.

query = '''
  CREATE VIEW IF NOT EXISTS Indegrees AS
    SELECT Target AS Id, COUNT(*) AS Degree
      FROM Edges
      GROUP BY Target
'''
c = conn.cursor ()
c.execute (query)

# Complete this query:
query = '''
  ...
'''
df_ranks = pd.read_sql_query (query, conn)
display (df_ranks)
sum (df_ranks['Rank'])


I am working on the file "14--pagerank-partial-solns2.ipynb".
The first task is:
Exercise. Implement a function to multiply a sparse matrix by a dense vector, assuming a dense vector defined as follows.
Can you write Python code for it?

def dense_vector (n, init_val=0.0):
    """
    Returns a dense vector of length `n`, with all entries set to
    `init_val`.
    """
    return [init_val] * n

def spmv (n, A, x):
    """Returns a dense vector y of length n, where y = A*x."""
    y = dense_vector (n)
    for (i, A_i) in A.items ():
        s = 0
        for (j, a_ij) in A_i.items ():
            s += a_ij * x[j]
        y[i] = s
    return y


I am working on the file "14--pagerank-partial-solns2.ipynb".
The first task is:
Exercise. Query the database to extract a report of which URLs point to which URLs. Also include the source vertex out-degree and order the rows in descending order by it.
Can you write Python code for it?

query = '''
  SELECT S.Url, T.Url, Out.Degree
    FROM Edges AS E,
         (SELECT Id, Url FROM Vertices) AS S,
         (SELECT Id, Url FROM Vertices) AS T,
         (SELECT Id, Degree FROM Outdegrees) AS Out
    WHERE (E.Source=S.Id) AND (E.Target=T.Id) AND (E.Source=Out.Id)
    ORDER BY -Out.Degree
'''
df_G = pd.read_sql_query (query, conn)

from IPython.display import display
display (df_G.head ())
print ("...")
display (df_G.tail ())


I am working on the file "14--pagerank-partial-solns2.ipynb".
The first task is:
Exercise. Write a snippet of code to verify that the vertex IDs are dense in some interval $[1, n]$. That is, there is a minimum value of $1$, some maximum value $n$, and no missing values between $1$ and $n$.
Can you write Python code for it?

query = '''
  SELECT MIN(Id) AS MinId,
         MAX(Id) AS MaxId,
         COUNT(DISTINCT Id) AS NumDistinctIds
    FROM Vertices
'''
df = pd.read_sql_query (query, conn)
print df
assert df.MinId[0] == 1
assert df.MaxId[0] == df.NumDistinctIds[0]
print ("\n==> Verified: Vertex ids cover [1, %d] densely." \
       % df.NumDistinctIds[0])


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
Binarizer
Binarizer 는 threshold 값을 기준으로 결과를 0, 1로 구분한다. 디폴트 threshold 는 0 이다.

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Binarizer.html
Can you write Python code for it?

from sklearn.preprocessing import Binarizer
X = [[ 1., -1.,  2.],
     [ 2.,  0.,  0.],
     [ 0.,  1., -1.]]
binarizer = Binarizer().fit(X)
binarizer.transform(X)

binarizer = Binarizer(threshold=1.1)
binarizer.transform(X)


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
DictVectorizer
DictVectorizer는 사전 형태로 되어 있는 feature 정보를 matrix 형태로 변환하기 위한 것으로  feature_extraction 서브 패키지에서 제공한다.
사전 정보는  텍스트 정보에서 corpus 상의 각 단어의 사용 빈도를 나타내는 경우가 많다.

http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html
Can you write Python code for it?

from sklearn.feature_extraction import DictVectorizer
v = DictVectorizer(sparse=False)
D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 1}]
X = v.fit_transform(D)
X

v.feature_names_

v.inverse_transform(X)

v.transform({'foo': 4, 'unseen_feature': 3})

instances = [ {'city': 'New York'}, {'city': 'San Francisco'}, {'city': 'Chapel Hill'} ]
v.fit_transform(instances)

v.feature_names_


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
FunctionTransformer
FunctionTransformer 입력값 $x$를 다항식이 아닌 사용자가 원하는 함수를 사용하여 변환한다.
$$ x \;\; \rightarrow \;\; [ f_1(x),  f_2(x),  f_3(x),  \cdots ] $$

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html
Can you write Python code for it?

from sklearn.preprocessing import FunctionTransformer
def all_but_first_column(X):
    return X[:, 1:]

X = np.arange(12).reshape(4, 3)
X

FunctionTransformer(all_but_first_column).fit_transform(X)


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
Imputer
Imputer는 누락된 정보를 채우는 변환이다. 다음과 같은 인수를 제공한다.

missing_values : 누락 정보
strategy : 채우는 방법. 디폴트는 "mean"
"mean": 평균
"median": 중앙값

"most_frequent": 최빈값

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html
Can you write Python code for it?

from sklearn.preprocessing import Imputer
imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit_transform([[1, 2], [np.nan, 3], [7, 6]])


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
Label Binarizer
Label Binarizer는 one-hot-encoder와 유사하지만 실제로 사용된 클래스만 인코딩에 사용하며 0/1 대신 사용할 값을 지정할 수 있다.

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html
Can you write Python code for it?

from sklearn.preprocessing import LabelBinarizer
lb = LabelBinarizer()
lb.fit([1, 2, 6, 1, 2])

lb.classes_

lb.transform([1, 6])

lb.fit_transform(['yes', 'no', 'no', 'yes'])

lb = LabelBinarizer(neg_label=-1)
lb.fit_transform(['yes', 'no', 'no', 'yes'])


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
Scikit-Learn의 전처리 기능
Scikit-Learn 패키지의 전처리 기능은 크게 다음과 같은 4가지이다. 이 기능들은 preprocessing 서브 패키지와 feature_extraction 서브 패키지에 구현되어 있다.

스케일링(Scaling): 자료의 크기 조정
scale, StandardScaler
normalize, Normalizer
minmax_scale, MinMaxScaler
maxabs_scale, MaxAbsScaler

robust_scale, RobustScaler

인코딩(Encoding): 카테고리 값의 정수 표현

binarize, Binarizer
label_binarize, LabelBinarizer
LabelEncoder
OneHotEncoder

DictVectorizer

Imputation: 결손 데이터(missing data) 처리

Imputer

Transform: 데이터 변환

PolynomialFeatures
FunctionTransformer

스케일링
스케일링은 자료 집합에 적용되는 전처리 과정으로 모든 자료에 다음과 같은 선형 변환을 적용하여 전체 자료의 분포를 평균 0, 분산 1이 되도록 만드는 과정이다.
$$ x' = c_0 + c_x x $$
스케일링은 자료의 overflow나 underflow를 방지하고 독립 변수의 공분산 행렬의 condition number 를 감소시켜 최적화 과정에서의 안정성 및 수렴 속도를 향상시킨다.
Scikit-Learn에서는 스케일링을 위한 함수와 클래스를 각각 제공한다. 스케일링 함수는 다음과 같다.

scale(X): 표준정상분포 Standard Normal Gaussian 기본 스케일
robust_scale(X): median, interquartile range 사용. outlier 영향 최소화
minmax_scale(X): 최대/최소값 사용
maxabs_scale(X): 최대절대값 사용
Can you write Python code for it?

from sklearn.preprocessing import scale, robust_scale, minmax_scale, maxabs_scale

x = (np.arange(10, dtype=np.float) - 3).reshape(-1, 1)
df = pd.DataFrame(np.hstack([x, scale(x), robust_scale(x), minmax_scale(x), maxabs_scale(x)]),
                  columns=["x", "scale(x)", "robust_scale(x)", "minmax_scale(x)", "maxabs_scale(x)"])
df

from sklearn.datasets import load_iris
iris = load_iris()

data1 = iris.data
data2 = scale(iris.data)

print("old mean:", np.mean(data1, axis=0))
print("old std: ", np.std(data1, axis=0))
print("new mean:", np.mean(data2, axis=0))
print("new std: ", np.std(data2, axis=0))

sns.jointplot(data1[:,0], data1[:,1])
plt.show()

sns.jointplot(data2[:,0], data2[:,1])
plt.show()


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
인코딩(Encoding)
인코딩은 카테고리 값이나 텍스트 정보를 처리가 쉬운 정수로 변환하는 과정이다.

크기가 없는 1-of-K 라벨을 숫자로 인식되지 않도록 0/1 vector로 변환
실수 타겟을 유한한 정수개의 카테고리로 변환
복수 라벨을 가지는 경우를 포함하여 카테고리로 변환

One-Hot-Encoder
One-Hot-Encoder는 one-of-K 인코딩이라고도 불리며 K개의 값을 가지는 스칼라 값을 0 또는1 값을 가지는 K-차원 벡터로 변환한다.
만약 입력이 스칼라가 아니라 벡터이면 stack된 결과가 출력된다. 이 때는 feature_indices_ 를 사용하여 분리해야 한다.
또 입력이 벡터인 경우에 특정한 열(column)만 카테고리 값이면 categorical_features 인수를 사용하여 카테고리 열을 지정한다. 이 때 인코딩 결과는 순서가 바뀔 수 있으므로 주의한다.
fit 메서드를 호출하면 다음과 같은 속성이 지정된다.

n_values_ : 최대 클래스 수
feature_indices_ : 입력이 벡터인 경우 각 원소를 나타내는 slice 정보
active_features_ : 실제로 사용된 클래스들

One-Hot-Encoding 결과는 메모리 절약을 위한 sparse matrix 형식으로 출력된다. 일반적인 배열로 바꾸려면 toarray 메서드를 사용한다.

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html
Can you write Python code for it?

from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()

X = np.array([[0], [1], [2]])
X

ohe.fit(X)
ohe.n_values_, ohe.feature_indices_, ohe.active_features_

ohe.transform(X).toarray()

X = np.array([[0, 0, 4], [1, 1, 0], [0, 2, 1], [1, 0, 2]])
X

ohe.fit(X)
ohe.n_values_, ohe.feature_indices_, ohe.active_features_

ohe.transform(X).toarray()

ohe = OneHotEncoder(categorical_features=[False, True, False])
ohe.fit_transform(X).toarray()


I am working on the file "14. 데이터 전처리/01. Scikit-Learn의 전처리 기능.ipynb".
The first task is:
정규화 (Normalization)
정규화(Normarlization)는 스케일링과 달리 개별 데이터의 크기를 모두 같게 만들기 위한 변환이다. 따라서 개별 데이터에 대해 서로 다른 변환 계수가 적용된다.
정규화는 다차원 독립 변수 벡터가 있을 때 각 벡터 원소들의 상대적 크기만 중요한 경우에 사용된다.
Can you write Python code for it?

from sklearn.preprocessing import normalize

x = np.vstack([np.arange(5, dtype=float) - 20, np.arange(5, dtype=float) - 2]).T
y1 = scale(x)
y2 = normalize(x)

print("original x:\n", x)
print("scale:\n", y1)
print("norms (scale)\n", np.linalg.norm(y1, axis=1))
print("normlize:\n", y2)
print("norms (normalize)\n", np.linalg.norm(y2, axis=1))

from sklearn.datasets import load_iris
iris = load_iris()

data1 = iris.data[:,:2]
data3 = normalize(data1)

sns.jointplot(data1[:,0], data1[:,1])
plt.show()

sns.jointplot(data3[:,0], data3[:,1])
plt.show()


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Doing our analysis
Use the sklearn package and a plain boring CountVectorizer to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.
Be sure to include English-language stopwords
Can you write Python code for it?

count_vectorizer = CountVectorizer(stop_words = 'english')

count_vectorizer.get_feature_names()

X = count_vectorizer.fit_transform(speeches_df['content'])

pd.DataFrame(X.toarray(), columns = count_vectorizer.get_feature_names())


countvectorizer = CountVectorizer(stop_words='english')

tokens = countvectorizer.fit_transform(speeches_df['content'])

countvectorizer.get_feature_names()


count_vectorizer = CountVectorizer(stop_words='english')

X=count_vectorizer.fit_transform(speeches_df['content'])

len(count_vectorizer.get_feature_names())

tokens_df=pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
tokens_df


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Can you write Python code for it?


# Simple counting vectorizer
vectorizer = CountVectorizer(tokenizer=stemming_tokenizer, stop_words='english', max_features = 10000)
X = vectorizer.fit_transform(speeches_df['content'])

from sklearn.cluster import KMeans

number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

# Term frequency vectorizer
vectorizer = TfidfVectorizer(use_idf=False, tokenizer=stemming_tokenizer, stop_words='english', max_features = 10000, norm = 'l1')
X = vectorizer.fit_transform(speeches_df['content'])

from sklearn.cluster import KMeans

number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

# Term frequency inverse document frequency vectorizer
vectorizer = TfidfVectorizer(tokenizer=stemming_tokenizer, stop_words='english', max_features = 10000)
X = vectorizer.fit_transform(speeches_df['content'])

from sklearn.cluster import KMeans

number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))


#SIMPLE COUNTING VECTORIZER
count_vectorizer = CountVectorizer(stop_words='english',max_features=100)
X = count_vectorizer.fit_transform(speeches_df['content'])
from sklearn.cluster import KMeans
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

#SIMPLE TERM FREQUENCY VECTORIZER
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=stemming_tokenizer, stop_words='english')
X = vectorizer.fit_transform(speeches_df['content'])
from sklearn.cluster import KMeans
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

# SIMPLE TFIDF VECTORIZER
vectorizer = TfidfVectorizer(tokenizer=stemming_tokenizer,use_idf=False,stop_words='english')
X = vectorizer.fit_transform(speeches_df['content'])
from sklearn.cluster import KMeans
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Can you write Python code for it?

df['chairman'].value_counts().head(5) #Chairman is NOT mentioned in 250 speeches.

len(df[df['chairman']==0])

len(df[(df['chairman']==0) & (df['mr']==0)])


only_100_tokens.describe()
#There are 702 speeches in this Dataframe

only_100_tokens['chairman_less'] = only_100_tokens['chairman'] == 0
only_100_tokens['chairman_less'].describe()

print('There are',702-452,'Speeches withou the word chairman in it')

only_100_tokens['mr_less'] = only_100_tokens['mr'] == 0
only_100_tokens['mr_less'].describe()

print('There are',702-623,'Speeches withou the word chairman in it')


# how many speeches are there total
len(speeches_df['content'])

# how many speeches don't mention "chairman"
len(speeches_df[speeches_df['content'].str.contains('chairman') == False])

# how many speeches don't mention "chairman" OR "mr."
len(speeches_df[(speeches_df['content'].str.contains('chairman') == False) & (speeches_df['content'].str.contains('mr.') == False)])


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
Now what if I'm using a TfidfVectorizer?
What's the content of the speeches? Here's a way to get them:
Can you write Python code for it?

# index 0 is the first speech, which was the first one imported.
paths[0]

# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!cat {paths[0]}


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
Can you write Python code for it?

paths = glob.glob('hp/hp/*')
paths[:5]

hp_fics = []
for path in paths:
    with open(path) as hp_file:
        hp_fic = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': hp_file.read()
        }
    hp_fics.append(hp_fic)
hp_fics_df = pd.DataFrame(hp_fics)
hp_fics_df.head()

tfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = tfidf_vectorizer.fit_transform(hp_fics_df['content'])
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

# i would say people are either writing about harry and hermione or lily and james
# ive never read harry potter, but i think that means either the harry generation or their parents generation?
# seems legit


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Homework 14 (or so): TF-IDF text analysis and clustering
Hooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the TF and IDF parts make a little sense. Kind of. Somewhat.
No, just kidding, we're professionals now.
Investigating the Congressional Record
The Congressional Record is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?
Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from this page here.
Can you write Python code for it?

# If you'd like to download it through the command line...
!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz

# And then extract it through the command line...
!tar -zxf convote_v1.1.tar.gz


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
If I'm searching for China and trade, what are the top 3 speeches to read according to the CountVectoriser?
Can you write Python code for it?

# i sorted by china here, on a lark. lets see if it holds true if i sort by trade
tokens_df[(tokens_df['china']>0) & (tokens_df['trade']>0)].sort_values(by='china', ascending=False)[['china', 'trade']].head(3)

tokens_df[(tokens_df['china']>0) & (tokens_df['trade']>0)].sort_values(by='trade', ascending=False)[['china', 'trade']].head(3)

# kind of! at any rate, speech 294 seems to be the most china and trade related. lets look at it!
speeches_df['content'][294]

# thats another super long speech, but it does seem to mostly be about trade and china.


ctdf=df[(df['china']!=0) & (df['trade']!=0)]
nctdf=pd.DataFrame([ctdf['china'], ctdf['trade'], ctdf['china'] + ctdf['trade']], index=["china", "trade", "china+trade"]).T

nctdf.sort_values(by='china+trade',ascending=False).head(3)


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Can you write Python code for it?


speech_num = 0
for speech in speeches_df['content'].head(5):
    speech_num += 1
    print(speech_num)
    print(speech)
    print('')


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Now let's push all of that into a dataframe with nicely named columns.
Can you write Python code for it?

only_100_tokens = pd.DataFrame(tokens_new.toarray(), columns=countvectorizer_new.get_feature_names())
only_100_tokens


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Can you write Python code for it?

tokens_complete['america'] = tokens_complete['america'].sort_values(ascending=False) >= 1
tokens_complete['america'].describe()

tokens_complete[tokens_complete['america'] == True].count().head(1)
#wow! Someone said america 75 times in an speech!

tokens_complete['elections'] = tokens_complete['elections'].sort_values(ascending=False) >= 1
tokens_complete['elections'].describe()
# the word elections has been present in nearly all speeches!


ecdf=pd.DataFrame([newdf['elect'], newdf['chao'], newdf['elect'] + newdf['chao']], index=["elections", "chaos", "elections+chaos"]).T

ecdf.sort_values(by='elections+chaos',ascending=False).head(5)


election_chaos_df=pd.DataFrame([tfidf_tokens_df['election'], tfidf_tokens_df['chaos'], tfidf_tokens_df['election'] + tfidf_tokens_df['chaos']], index=["election", "chaos", "election + chaos"]).T
election_chaos_df[election_chaos_df.any(axis=1)].sort_values(by='chaos', ascending=False).head(10)

# i did the sort that way because i guess they dont talk about chaos much. lets look at that speech
!cat {paths[257]}

# thats pretty weak. i tried to come up with something spicier but i cant tell what years these are from.
# how about this?
clinton_welfare_df=pd.DataFrame([tfidf_tokens_df['clinton'], tfidf_tokens_df['welfare'], tfidf_tokens_df['clinton'] + tfidf_tokens_df['welfare']], index=["clinton", "welfare", "clinton + welfare"]).T
clinton_welfare_df[clinton_welfare_df.any(axis=1)].sort_values(by='clinton + welfare', ascending=False).head(10)

!cat {paths[346]}

# still pretty dull. oh well.


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Can you write Python code for it?

from sklearn.cluster import KMeans

number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(x)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Now what if I'm using a TfidfVectorizer?
Can you write Python code for it?

l2_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, max_features = 100)
X = l2_vectorizer.fit_transform(speeches_df['content'])
l2_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())

(l2_df['china'] + l2_df['trade']).sort_values(ascending = False).head(3)


l2_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = l2_vectorizer.fit_transform(speeches_df['content'])
tfidf_tokens_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
china_trade_df=pd.DataFrame([tfidf_tokens_df['china'], tfidf_tokens_df['trade'], tfidf_tokens_df['china'] + tfidf_tokens_df['trade']], index=["china", "trade", "china + trade"]).T
china_trade_df[china_trade_df.any(axis=1)].sort_values(by='china + trade', ascending=False).head(3)

# wow, that comes up with a totally different list of speeches. lets look at speech 636

speeches_df['content'][636]

# that one is very short by comparison and this time, its really only about china and trade


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Now what if I'm using a TfidfVectorizer?
What's the content of the speeches? Here's a way to get them:
Can you write Python code for it?


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Can you write Python code for it?

count_vectorizer = CountVectorizer(stop_words='english', max_features=100)
x = count_vectorizer.fit_transform(list_of_speeches)
print(count_vectorizer.get_feature_names())


count_vectorizer = CountVectorizer(stop_words='english', max_features=100)

X=count_vectorizer.fit_transform(speeches_df['content'])
len(count_vectorizer.get_feature_names())


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Now let's push all of that into a dataframe with nicely named columns.
Can you write Python code for it?

from nltk.stem.porter import PorterStemmer
import re

porter_stemmer = PorterStemmer()

def stemming_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    words = [porter_stemmer.stem(word) for word in words]
    return words

count_vectorizer = CountVectorizer(stop_words = 'english', tokenizer = stemming_tokenizer, max_features = 100)
X = count_vectorizer.fit_transform(speeches_df['content'])
pd.DataFrame(X.toarray(), columns = count_vectorizer.get_feature_names())


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
So great, we have 702 of them. Now let's import them.
Can you write Python code for it?

speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
speeches_df.head()


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
Can you write Python code for it?

df['thank'].max()

df[df['thank']==9] #Speech No 9


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
What is the index of the speech which is the most thankful, a.k.a. includes the word 'thank' the most times?
Can you write Python code for it?

# so speech index 375
tokens_df[tokens_df['thank']==tokens_df['thank'].max()].index

# lets look at the speech
speeches_df['content'][375]

# wow that was long. but its the most thankful one, so whatever.


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
What's the content of the speeches? Here's a way to get them:
Can you write Python code for it?

# index 0 is the first speech, which was the first one imported.
paths[0]

# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!type "convote_v1.1\data_stage_one\development_set\052_400011_0327014_DON.txt"
#!type "{paths[0].replace("/","\\")}"

!type "{paths[0].replace("/","\\")}"


# index 0 is the first speech, which was the first one imported.
paths[0]

# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!cat {paths[0]}


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Which one do you think works the best?
Can you write Python code for it?

# well, it seems to be between results including wild horses and those including frivolous lawsuits.
# i prefer wild horses but the tfidf is probably the best representation of the actual document


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Which one do you think works the best?
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
Can you write Python code for it?

paths = glob.glob('hp/10196712.txt*')
paths[:5]

len(paths)


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Which one do you think works the best?
IDF, works the best
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
Can you write Python code for it?

paths = glob.glob('hp/*')

paths[0]

len(paths)

speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
hpfanfic_df = pd.DataFrame(speeches)
hpfanfic_df.head()

hpfanfic_df['content'].head(5)

vectorizer = TfidfVectorizer(use_idf=True, max_features=10000, stop_words='english')
X = vectorizer.fit_transform(hpfanfic_df['content'])

print(vectorizer.get_feature_names()[:10])

df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())

df.head(5)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

hpfanfic_df['category'] = km.labels_

hpfanfic_df.head()


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
Which one do you think works the best?
One of the second two, but it's hard to tell which.
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
Can you write Python code for it?

paths = glob.glob('hp/*')
paths[:5]

speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
hp_df = pd.DataFrame(speeches)
hp_df.head()

# # The two clusters for this are
# # Top terms per cluster:
# # Cluster 0: wa hi harri hermion t
# # Cluster 1: wa hi t s lili
# # Which is unintelligible thing to me

# vectorizer = TfidfVectorizer(tokenizer=stemming_tokenizer, stop_words='english', max_features = 10000)
# X = vectorizer.fit_transform(hp_df['content'])

# from sklearn.cluster import KMeans

# number_of_clusters = 2
# km = KMeans(n_clusters=number_of_clusters)
# km.fit(X)

# print("Top terms per cluster:")
# order_centroids = km.cluster_centers_.argsort()[:, ::-1]
# terms = vectorizer.get_feature_names()
# for i in range(number_of_clusters):
#     top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
#     print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

vectorizer = TfidfVectorizer(stop_words='english', max_features = 10000)
X = vectorizer.fit_transform(hp_df['content'])

from sklearn.cluster import KMeans

number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))


I am working on the file "14/14 - TF-IDF Homework.ipynb".
The first task is:
You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
Can you write Python code for it?

# glob finds files matching a certain filename pattern
import glob

# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]

len(paths)


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
CountVectorizer(): Convert a collection of text documents to a matrix of token counts
TfidfVectorizer(use_idf=False): Convert a collection of raw documents to a matrix of TF-IDF features. Equivalent to CountVectorizer followed by TfidfTransformer.
TfidfVectorizer(use_idf=True) (default): Enable inverse-document-frequency reweighting.
Can you write Python code for it?

countingVectorizer = CountVectorizer(tokenizer=simple_tokenizer, stop_words='english')
TF_Vectorizer = TfidfVectorizer(use_idf=False, tokenizer=simple_tokenizer, stop_words='english')
TF_IDF_Vectorizer = TfidfVectorizer(use_idf=True, tokenizer=simple_tokenizer, stop_words='english')
Vectorizer_list = [countingVectorizer, TF_Vectorizer, TF_IDF_Vectorizer]
Vectorizer_names = ['', 'simple counting vectorizer', 'term frequency vectorizer', 'term frequency IDF vectorizer']

count = 1
for vectorizer in Vectorizer_list:
    print("\n[" + str(count) + "]", Vectorizer_names[count])

    X = vectorizer.fit_transform(speeches_df['content'])
    number_of_clusters = 8
    km = KMeans(n_clusters=number_of_clusters)
    km.fit(X)

    order_centroids = km.cluster_centers_.argsort()[:, ::-1]
    terms = vectorizer.get_feature_names()
    for i in range(number_of_clusters):
        top_five_words = [terms[ind] for ind in order_centroids[i, :10]]
        print("Cluster {}: {}".format(i, ' '.join(top_five_words)))

    count += 1


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Can you write Python code for it?

no_chairman = X_df[X_df['chairman'] == 0]['chairman'].count()
no_chairman_no_mr = X_df[(X_df['chairman'] == 0) & (X_df['mr'] == 0)]['chairman'].count()
print("In a total of", len(X_df), "speeches,", no_chairman, "don't mention “chairman” and", no_chairman_no_mr, "mention neither “mr” nor “chairman”.")


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Can you write Python code for it?

for item in speeches_df['content'][:5]:
    print(item[:140], "\n")


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Can you write Python code for it?

numbers = list(range(0, 10))
numbers = list(map(str, numbers))
words_list = [i for i in list(TF_pd.columns) if i[0] not in numbers]
print(*words_list[5:100], sep='|') # to get some ideas

chaos = TF_pd.sort_values(by=['awfully', 'bacterial'], ascending=[0, 0])[['awfully', 'bacterial']].head(3)
print("The three top speeches have the indexes ", *list(chaos.index))
chaos

gun_bomb = TF_pd.sort_values(by=['gun', 'bomb'], ascending=[0, 0])[['gun', 'bomb']].head(3)
print("The three top speeches have the indexes ", *list(gun_bomb.index))
gun_bomb


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
So great, we have 702 of them. Now let's import them.
Can you write Python code for it?


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
What is the index of the speech thank is the most thankful, a.k.a. includes the word 'thank' the most times?
Can you write Python code for it?

print("The index of this speech is", X_df['thank'].idxmax())


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
What's the content of the speeches? Here's a way to get them:
Can you write Python code for it?


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
Which one do you think works the best?
The TF-IDF is definitely the most efficient one, at least in this case!
Harry Potter time
I have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.
I want you to read them in, vectorize them and cluster them. Use this process to find out the two types of Harry Potter fanfiction. What is your hypothesis?
curl -LO
Can you write Python code for it?

!curl -LO https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip

#!unzip hp.zip
paths_potter = glob.glob('hp/*')
paths_potter[:3]

potter_texts = []
for path in paths_potter:
    with open(path) as speech_file:
        text = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    potter_texts.append(text)
potter_df = pd.DataFrame(potter_texts)
potter_df.head(2)

#1
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=simple_tokenizer, stop_words='english')
X = vectorizer.fit_transform(potter_df['content'])

#2
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

#3
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

#4
results = pd.DataFrame()
results['text'] = potter_df['content']
results['category'] = km.labels_
results.head(10)


I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
You can explore the files if you'd like, but we're going to get the ones from convote_v1.1/data_stage_one/development_set/. It's a bunch of text files.
Can you write Python code for it?


I am working on the file "14_02_multilayer-networks.ipynb".
The first task is:
Challenge 14.2
Use SGD to train the single neuron in the previous notebook using a linearly separable set of 100 points, divided by the line $-\frac{5}{2}x+\frac{3}{2}y+3=0$
Can you write Python code for it?

### We provide a set of randomly generated training points
num_points = 100
w1 = -2.5
w2 = 1.5
w0 = 3.
np.random.seed(637163) # we make sure we always generate the same sequence
x_data = np.random.rand(num_points)*10.
y_data = np.random.rand(num_points)*10.
z_data = np.zeros(num_points)
for i in range(len(z_data)):
    if (y_data[i] > (-w0-w1*x_data[i])/w2):
        z_data[i] = 1.

pyplot.scatter(x_data,y_data,c=z_data,marker='o',linewidth=1.5,edgecolors='black')
pyplot.plot(x_data,(-w1*x_data-w0)/w2)
pyplot.gray()
pyplot.xlim(0,10)
pyplot.ylim(0,10);


I am working on the file "14_02_multilayer-networks.ipynb".
The first task is:
Finally, we'll use stochastic gradient descent to learn from the MNIST training_data over 30 epochs, with a mini-batch size of 10, and a learning rate of $\eta$=3.0:
Can you write Python code for it?

net.SGD(training_data, 30, 10, 3.0, test_data=test_data)


I am working on the file "14_02_multilayer-networks.ipynb".
The first task is:
Note also that the biases and weights are stored as lists of Numpy matrices. So, for example net.weights[1] is a Numpy matrix storing the weights connecting the second and third layers of neurons. (It's not the first and second layers, since Python's list indexing starts at 0.) Since net.weights[1] is rather verbose, let's just denote that matrix $w$
. It's a matrix such that $w_{jk}$
 is the weight for the connection between the $k^{th}$
 neuron in the second layer, and the $j^{th}$
 neuron in the third layer. This ordering of the $j$
 and $k$
 indices may seem strange. The big advantage of using this ordering is that it means that the vector of activations of the third layer of neurons is:
$$a'=\mathrm {sigmoid}(wa+b)$$
There's quite a bit going on in this equation, so let's unpack it piece by piece. $a$
 is the vector of activations of the second layer of neurons. To obtain $a'$
 we multiply $a$
 by the weight matrix $w$
, and add the vector $b$
 of biases. We then apply the function sigmoid
elementwise to every entry in the vector $wa+b$.
Of course, the main thing we want our Network objects to do is to learn. To that end we'll give them an SGD method which implements stochastic gradient descent.
<!--
The training_data is a list of tuples `(x, y)` representing the training inputs and corresponding desired outputs. The variables `epochs` and `mini_batch_size` are what you'd expect - the number of epochs to train for, and the size of the mini-batches to use when sampling. `eta` is the learning rate, $\eta$. If the optional argument `test_data` is supplied, then the program will evaluate the network after each epoch of training, and print out partial progress. This is useful for tracking progress, but slows things down substantially.

The code works as follows. In each epoch, it starts by randomly shuffling the training data, and then partitions it into mini-batches of the appropriate size. This is an easy way of sampling randomly from the training data. Then for each `mini_batch` we apply a single step of...
Can you write Python code for it?

"""
network.py
~~~~~~~~~~

A module to implement the stochastic gradient descent learning
algorithm for a feedforward neural network.  Gradients are calculated
using backpropagation.  Note that I have focused on making the code
simple, easily readable, and easily modifiable.  It is not optimized,
and omits many desirable features.
"""

#### Libraries
# Standard library
import random

# Third-party libraries
import numpy as np

class Network(object):

    def __init__(self, sizes):
        """The list ``sizes`` contains the number of neurons in the
        respective layers of the network.  For example, if the list
        was [2, 3, 1] then it would be a three-layer network, with the
        first layer containing 2 neurons, the second layer 3 neurons,
        and the third layer 1 neuron.  The biases and weights for the
        network are initialized randomly, using a Gaussian
        distribution with mean 0, and variance 1.  Note that the first
        layer is assumed to be an input layer, and by convention we
        won't set any biases for those neurons, since biases are only
        ever used in computing the outputs from later layers."""
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
        self.weights = [np.random.randn(y, x)
                        for x, y in zip(sizes[:-1], sizes[1:])]

    def feedforward(self, a):
        """Return the output of the network if ``a`` is input."""
        for b, w in zip(self.biases, self.weights):
            a = sigmoid(np.dot(w, a)+b)
        return a

    def SGD(self, training_data, epochs, mini_batch_size, eta,
            test_data=None):
        """Train the neural network using mini-batch stochastic
        gradient descent.  The ``training_data`` is a list of tuples
        ``(x, y)`` representing the training inputs and the desired
        outputs.  The other non-optional parameters are
        self-explanatory.  If ``test_data`` is provided then the
        network...


I am working on the file "14_02_multilayer-networks.ipynb".
The first task is:
We first load the MNIST data:
Can you write Python code for it?

training_data, validation_data, test_data = load_data_wrapper()


I am working on the file "14_02_multilayer-networks.ipynb".
The first task is:
We now write an SGD code for this problem. The training_data is a list of tuples (x, y) representing the training inputs and corresponding desired outputs. The variables epochs and mini_batch_size are what you'd expect - the number of epochs to train for, and the size of the mini-batches to use when sampling. eta is the learning rate, $\eta$. If the optional argument test_data is supplied, then the program will evaluate the network after each epoch of training, and print out partial progress. This is useful for tracking progress, but slows things down substantially.
The code works as follows. In each epoch, it starts by randomly shuffling the training data, and then partitions it into mini-batches of the appropriate size. This is an easy way of sampling randomly from the training data. Then for each mini_batch we apply a single step of gradient descent. This is done by the code self.update_mini_batch(mini_batch, eta), which updates the coefficients according to a single iteration of gradient descent, using just the training data in mini_batch.
Can you write Python code for it?

epochs = 1000
mini_batch_size = 10
eta = 0.01/mini_batch_size

a = 3.
b = 3.
def update_mini_batch(mini_batch, eta):
    global a, b
    a0 = a
    b0 = b
    for x, y, in mini_batch:
        e = eta*(a0+b0*x-y)
        a -= e
        b -= x*e

training_data = list(zip(x_data,y_data))
for j in range(epochs):
    np.random.shuffle(training_data)
    mini_batches = [training_data[k:k+mini_batch_size]
                    for k in range(0, len(training_data), mini_batch_size)]
    for mini_batch in mini_batches:
        update_mini_batch(mini_batch, eta)
    print ("Epoch {0}: {1} {2}".format(j,a,b))


I am working on the file "14_02_multilayer-networks.ipynb".
The first task is:
You will need the following auxiliary functions:
Can you write Python code for it?

def sigmoid(z):
    """The sigmoid function."""
    return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    """Derivative of the sigmoid function."""
    return sigmoid(z)*(1-sigmoid(z))


I am working on the file "14_Analyzing_Text/14 - TF-IDF Homework.ipynb".
The first task is:
Enough of this garbage, let's cluster
Using a simple counting vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Using a term frequency inverse document frequency vectorizer, cluster the documents into eight categories, telling me what the top terms are per category.
Can you write Python code for it?

#simple counting vectorizer,

from sklearn.cluster import KMeans
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)

count_vectorizer = CountVectorizer(stop_words='english')
X = count_vectorizer.fit_transform(All_speeches)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

# term frequency vectorizer,

vectorizer = TfidfVectorizer(use_idf=True, stop_words='english')
X = vectorizer.fit_transform(All_speeches)

number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)

print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :10]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))

#term frequency inverse document frequency vectorizer

def oh_tokenizer(str_input):
    words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
    return words

l2_vectorizer = TfidfVectorizer(use_idf=True, stop_words='english', tokenizer=oh_tokenizer)
X = l2_vectorizer.fit_transform(speeches_df['content'])
l2_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
for i in range(number_of_clusters):
    top_ten_words = [l2_df[ind] for ind in order_centroids[i, :9]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))


I am working on the file "14_Analyzing_Text/14 - TF-IDF Homework.ipynb".
The first task is:
Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and many don't mention "chairman" and how many mention neither "mr" nor "chairman"?
Can you write Python code for it?

speeches_df.info()

Top_100_tokens['No_chairman'] = Top_100_tokens['chairman'] == 0
Top_100_tokens[Top_100_tokens['No_chairman'] == True].count().head(1)

Top_100_tokens['no_mr'] = Top_100_tokens['mr'] == 0
Top_100_tokens[Top_100_tokens['no_mr'] == True].count().head(1)


I am working on the file "14_Analyzing_Text/14 - TF-IDF Homework.ipynb".
The first task is:
In class we had the texts variable. For the homework can just do speeches_df['content'] to get the same sort of list of stuff.
Take a look at the contents of the first 5 speeches
Can you write Python code for it?

All_speeches = speeches_df['content']
First_five_speeches = speeches_df['content'].head(5)
First_five_speeches


I am working on the file "14_Analyzing_Text/14 - TF-IDF Homework.ipynb".
The first task is:
Now search for something else! Another two terms that might show up. elections and chaos? Whatever you thnik might be interesting.
Can you write Python code for it?

All_tokens['chaos'] = All_tokens['chaos'].sort_values(ascending=False) >= 1
All_tokens[All_tokens['chaos'] == True].count().head(1)