I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Regularization.ipynb".
The first task is:
Observations:
- The value of $\lambda$ is a hyperparameter that you can tune using a dev set.
- L2 regularization makes your decision boundary smoother. If $\lambda$ is too large, it is also possible to "oversmooth", resulting in a model with high bias.
What is L2-regularization actually doing?:
L2-regularization relies on the assumption that a model with small weights is simpler than a model with large weights. Thus, by penalizing the square values of the weights in the cost function you drive all the weights to smaller values. It becomes too costly for the cost to have large weights! This leads to a smoother model in which the output changes more slowly as the input changes.

What you should remember -- the implications of L2-regularization on:
- The cost computation:
    - A regularization term is added to the cost
- The backpropagation function:
    - There are extra terms in the gradients with respect to weight matrices
- Weights end up smaller ("weight decay"):
    - Weights are pushed to smaller values.
3 - Dropout
Finally, dropout is a widely used regularization technique that is specific to deep learning.
It randomly shuts down some neurons in each iteration. Watch these two videos to see what this means!

<video width="620" height="440" src="images/dropout1_kiank.mp4" type="video/mp4"...
Can you write Python code for it?

# GRADED FUNCTION: forward_propagation_with_dropout

def forward_propagation_with_dropout(X, parameters, keep_prob=0.5):
    """
    Implements the forward propagation: LINEAR -> RELU + DROPOUT -> LINEAR -> RELU + DROPOUT -> LINEAR -> SIGMOID.

    Arguments:
    X -- input dataset, of shape (2, number of examples)
    parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":
                    W1 -- weight matrix of shape (20, 2)
                    b1 -- bias vector of shape (20, 1)
                    W2 -- weight matrix of shape (3, 20)
                    b2 -- bias vector of shape (3, 1)
                    W3 -- weight matrix of shape (1, 3)
                    b3 -- bias vector of shape (1, 1)
    keep_prob - probability of keeping a neuron active during drop-out, scalar

    Returns:
    A3 -- last activation value, output of the forward propagation, of shape (1,1)
    cache -- tuple, information stored for computing the backward propagation
    """

    np.random.seed(1)

    # retrieve parameters
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]
    W3 = parameters["W3"]
    b3 = parameters["b3"]

    # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID
    Z1 = np.dot(W1, X) + b1
    A1 = relu(Z1)
    ### START CODE HERE ### (approx. 4 lines)         # Steps 1-4 below correspond to the Steps 1-4 described above.
    D1 = np.random.rand(A1.shape[0], A1.shape[1])     # Step 1: initialize matrix D1 = np.random.rand(..., ...)
    D1 = D1 < keep_prob                            # Step 2: convert entries of D1 to 0 or 1 (using keep_prob as the threshold)
    A1 = A1 * D1                                      # Step 3: shut down some neurons of A1
    A1 = A1 / keep_prob                               # Step 4: scale the value of neurons that haven't been shut down
    ### END CODE HERE ###
    Z2 = np.dot(W2, A1) + b2
    A2 = relu(Z2)
    ### START CODE HERE ### (approx. 4 lines)
    D2 =...


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Regularization.ipynb".
The first task is:
Problem Statement: You have just been hired as an AI expert by the French Football Corporation. They would like you to recommend positions where France's goal keeper should kick the ball so that the French team's players can then hit it with their head.

  Figure 1 : Football field
 The goal keeper kicks the ball in the air, the players of each team are fighting to hit the ball with their head

They give you the following 2D dataset from France's past 10 games.
Can you write Python code for it?

train_X, train_Y, test_X, test_Y = load_2D_dataset()


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Regularization.ipynb".
The first task is:
The non-regularized model is obviously overfitting the training set. It is fitting the noisy points! Lets now look at two techniques to reduce overfitting.
2 - L2 Regularization
The standard way to avoid overfitting is called L2 regularization. It consists of appropriately modifying your cost function, from:
$$J = -\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small  y^{(i)}\log\left(a^{L}\right) + (1-y^{(i)})\log\left(1- a^{L}\right) \large{)} \tag{1}$$
To:
$$J_{regularized} = \small \underbrace{-\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small y^{(i)}\log\left(a^{L}\right) + (1-y^{(i)})\log\left(1- a^{L}\right) \large{)} }\text{cross-entropy cost} + \underbrace{\frac{1}{m} \frac{\lambda}{2} \sum\limits_l\sum\limits_k\sum\limits_j W{k,j}^{[l]2} }_\text{L2 regularization cost} \tag{2}$$
Let's modify your cost and observe the consequences.
Exercise: Implement compute_cost_with_regularization() which computes the cost given by formula (2). To calculate $\sum\limits_k\sum\limits_j W_{k,j}^{[l]2}$  , use :
python
np.sum(np.square(Wl))
Note that you have to do this for $W^{[1]}$, $W^{[2]}$ and $W^{[3]}$, then sum the three terms and multiply by $ \frac{1}{m} \frac{\lambda}{2} $.
Can you write Python code for it?

# GRADED FUNCTION: compute_cost_with_regularization

def compute_cost_with_regularization(A3, Y, parameters, lambd):
    """
    Implement the cost function with L2 regularization. See formula (2) above.

    Arguments:
    A3 -- post-activation, output of forward propagation, of shape (output size, number of examples)
    Y -- "true" labels vector, of shape (output size, number of examples)
    parameters -- python dictionary containing parameters of the model

    Returns:
    cost - value of the regularized loss function (formula (2))
    """
    m = Y.shape[1]
    W1 = parameters["W1"]
    W2 = parameters["W2"]
    W3 = parameters["W3"]

    cross_entropy_cost = compute_cost(A3, Y) # This gives you the cross-entropy part of the cost

    ### START CODE HERE ### (approx. 1 line)
    L2_regularization_cost = lambd * (np.sum(np.square(W1)) + np.sum(np.square(W2)) + np.sum(np.square(W3))) / (2 * m)
    ### END CODER HERE ###

    cost = cross_entropy_cost + L2_regularization_cost

    return cost

A3, Y_assess, parameters = compute_cost_with_regularization_test_case()

print("cost = " + str(compute_cost_with_regularization(A3, Y_assess, parameters, lambd = 0.1)))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Regularization.ipynb".
The first task is:
The train accuracy is 94.8% while the test accuracy is 91.5%. This is the baseline model (you will observe the impact of regularization on this model). Run the following code to plot the decision boundary of your model.
Can you write Python code for it?

plt.title("Model without regularization")
axes = plt.gca()
axes.set_xlim([-0.75, 0.40])
axes.set_ylim([-0.75, 0.65])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
As expected, you will not see 20! You got a tensor saying that the result is a tensor that does not have the shape attribute, and is of type "int32". All you did was put in the 'computation graph', but you have not run this computation yet. In order to actually multiply the two numbers, you will have to create a session and run it.
Can you write Python code for it?


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
As usual you flatten the image dataset, then normalize it by dividing by 255. On top of that, you will convert each label to a one-hot vector as shown in Figure 1. Run the cell below to do so.
Can you write Python code for it?

# Flatten the training and test images
X_train_flatten = X_train_orig.reshape(X_train_orig.shape[0], -1).T
X_test_flatten = X_test_orig.reshape(X_test_orig.shape[0], -1).T
# Normalize image vectors
X_train = X_train_flatten / 255.
X_test = X_test_flatten / 255.
# Convert training and test labels to one hot matrices
Y_train = convert_to_one_hot(Y_train_orig, 6)
Y_test = convert_to_one_hot(Y_test_orig, 6)

print("number of training examples = " + str(X_train.shape[1]))
print("number of test examples = " + str(X_test.shape[1]))
print("X_train shape: " + str(X_train.shape))
print("Y_train shape: " + str(Y_train.shape))
print("X_test shape: " + str(X_test.shape))
print("Y_test shape: " + str(Y_test.shape))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Change the index below and run the cell to visualize some examples in the dataset.
Can you write Python code for it?

# Example of a picture
index = 0
plt.imshow(X_train_orig[index])
print ("y = " + str(np.squeeze(Y_train_orig[:, index])))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Expected Output :

**result**

[[-2.15657382]
 [ 2.95891446]
 [-1.08926781]
 [-0.84538042]]

1.2 - Computing the sigmoid
Great! You just implemented a linear function. Tensorflow offers a variety of commonly used neural network functions like tf.sigmoid and tf.softmax. For this exercise lets compute the sigmoid function of an input.
You will do this exercise using a placeholder variable x. When running the session, you should use the feed dictionary to pass in the input z. In this exercise, you will have to (i) create a placeholder x, (ii) define the operations needed to compute the sigmoid using tf.sigmoid, and then (iii) run the session.
 Exercise : Implement the sigmoid function below. You should use the following:

tf.placeholder(tf.float32, name = "...")
tf.sigmoid(...)
sess.run(..., feed_dict = {x: z})

Note that there are two typical ways to create and use sessions in tensorflow:
Method 1:
```python
sess = tf.Session()
Run the variables initialization (if needed), run the operations
result = sess.run(..., feed_dict = {...})
sess.close() # Close the session
**Method 2:**python
with tf.Session() as sess:
    # run the variables initialization (if needed), run the operations
    result = sess.run(..., feed_dict = {...})
    # This takes care of closing the session for you :)
```
Can you write Python code for it?

# GRADED FUNCTION: sigmoid

def sigmoid(z):
    """
    Computes the sigmoid of z

    Arguments:
    z -- input value, scalar or vector

    Returns:
    results -- the sigmoid of z
    """

    ### START CODE HERE ### ( approx. 4 lines of code)
    # Create a placeholder for x. Name it 'x'.
    x = tf.placeholder(tf.float32, name="x")

    # compute sigmoid(x)
    sigmoid = tf.sigmoid(x)

    # Create a session, and run it. Please use the method 2 explained above.
    # You should use a feed_dict to pass z's value to x.
    with tf.Session() as sess:
        # Run session and call the output "result"
        result = result = sess.run(sigmoid, feed_dict = {x: z})

    ### END CODE HERE ###

    return result

print ("sigmoid(0) = " + str(sigmoid(0)))
print ("sigmoid(12) = " + str(sigmoid(12)))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Expected Output :

**sigmoid(0)**

0.5

**sigmoid(12)**

0.999994

To summarize, you how know how to:
1. Create placeholders
2. Specify the computation graph corresponding to operations you want to compute
3. Create the session
4. Run the session, using a feed dictionary if necessary to specify placeholder variables' values.
1.3 -  Computing the Cost
You can also use a built-in function to compute the cost of your neural network. So instead of needing to write code to compute this as a function of $a^{2}$ and $y^{(i)}$ for i=1...m:
$$ J = - \frac{1}{m}  \sum_{i = 1}^m  \large ( \small y^{(i)} \log a^{ [2] (i)} + (1-y^{(i)})\log (1-a^{ [2] (i)} )\large )\small\tag{2}$$
you can do it in one line of code in tensorflow!
Exercise: Implement the cross entropy loss. The function you will use is:

tf.nn.sigmoid_cross_entropy_with_logits(logits = ...,  labels = ...)

Your code should input z, compute the sigmoid (to get a) and then compute the cross entropy cost $J$. All this can be done using one call to tf.nn.sigmoid_cross_entropy_with_logits, which computes
$$- \frac{1}{m}  \sum_{i = 1}^m  \large ( \small y^{(i)} \log \sigma(z^{2}) + (1-y^{(i)})\log (1-\sigma(z^{2})\large )\small\tag{2}$$
Can you write Python code for it?

# GRADED FUNCTION: cost

def cost(logits, labels):
    """
    Computes the cost using the sigmoid cross entropy

    Arguments:
    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)
    labels -- vector of labels y (1 or 0)

    Note: What we've been calling "z" and "y" in this class are respectively called "logits" and "labels"
    in the TensorFlow documentation. So logits will feed into z, and labels into y.

    Returns:
    cost -- runs the session of the cost (formula (2))
    """

    ### START CODE HERE ###

    # Create the placeholders for "logits" (z) and "labels" (y) (approx. 2 lines)
    z = tf.placeholder(tf.float32, name="z")
    y = tf.placeholder(tf.float32, name="y")

    # Use the loss function (approx. 1 line)
    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits=z, labels=y)

    # Create a session (approx. 1 line). See method 1 above.
    sess = tf.Session()

    # Run the session (approx. 1 line).
    cost = sess.run(cost, feed_dict={z: logits, y: labels})

    # Close the session (approx. 1 line). See method 1 above.
    sess.close()

    ### END CODE HERE ###

    return cost

logits = sigmoid(np.array([0.2, 0.4, 0.7, 0.9]))
cost = cost(logits, np.array([0, 0, 1, 1]))
print ("cost = " + str(cost))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Expected Output:

            **Train Accuracy**

        0.999074

            **Test Accuracy**

        0.716667

Amazing, your algorithm can recognize a sign representing a figure between 0 and 5 with 71.7% accuracy.
Insights:
- Your model seems big enough to fit the training set well. However, given the difference between train and test accuracy, you could try to add L2 or dropout regularization to reduce overfitting.
- Think about the session as a block of code to train the model. Each time you run the session on a minibatch, it trains the parameters. In total you have run the session a large number of times (1500 epochs) until you obtained well trained parameters.
2.7 - Test with your own image (optional / ungraded exercise)
Congratulations on finishing this assignment. You can now take a picture of your hand and see the output of your model. To do that:
    1. Click on "File" in the upper bar of this notebook, then click "Open" to go on your Coursera Hub.
    2. Add your image to this Jupyter Notebook's directory, in the "images" folder
    3. Write your image's name in the following code
    4. Run the code and check if the algorithm is right!
Can you write Python code for it?

import scipy
from PIL import Image
from scipy import ndimage

## START CODE HERE ## (PUT YOUR IMAGE NAME)
my_image = "thumbs_up.jpg"
## END CODE HERE ##

# We preprocess your image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))
my_image = scipy.misc.imresize(image, size=(64, 64)).reshape((1, 64 * 64 * 3)).T
my_image_prediction = predict(my_image, parameters)

plt.imshow(image)
print("Your algorithm predicts: y = " + str(np.squeeze(my_image_prediction)))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Expected Output:

            **X**

        Tensor("Placeholder_1:0", shape=(12288, ?), dtype=float32) (not necessarily Placeholder_1)

            **Y**

        Tensor("Placeholder_2:0", shape=(6, ?), dtype=float32) (not necessarily Placeholder_2)

2.2 - Initializing the parameters
Your second task is to initialize the parameters in tensorflow.
Exercise: Implement the function below to initialize the parameters in tensorflow. You are going use Xavier Initialization for weights and Zero Initialization for biases. The shapes are given below. As an example, to help you, for W1 and b1 you could use:
python
W1 = tf.get_variable("W1", [25,12288], initializer = tf.contrib.layers.xavier_initializer(seed = 1))
b1 = tf.get_variable("b1", [25,1], initializer = tf.zeros_initializer())
Please use seed = 1 to make sure your results match ours.
Can you write Python code for it?

# GRADED FUNCTION: initialize_parameters

def initialize_parameters():
    """
    Initializes parameters to build a neural network with tensorflow. The shapes are:
                        W1 : [25, 12288]
                        b1 : [25, 1]
                        W2 : [12, 25]
                        b2 : [12, 1]
                        W3 : [6, 12]
                        b3 : [6, 1]

    Returns:
    parameters -- a dictionary of tensors containing W1, b1, W2, b2, W3, b3
    """

    tf.set_random_seed(1)                   # so that your "random" numbers match ours

    ### START CODE HERE ### (approx. 6 lines of code)
    W1 = tf.get_variable("W1", [25, 12288], initializer = tf.contrib.layers.xavier_initializer(seed=1))
    b1 = tf.get_variable("b1", [25, 1], initializer = tf.zeros_initializer())
    W2 = tf.get_variable("W2", [12, 25], initializer = tf.contrib.layers.xavier_initializer(seed=1))
    b2 = tf.get_variable("b2", [12, 1], initializer = tf.zeros_initializer())
    W3 = tf.get_variable("W3", [6, 12], initializer = tf.contrib.layers.xavier_initializer(seed=1))
    b3 = tf.get_variable("b3", [6, 1], initializer = tf.zeros_initializer())
    ### END CODE HERE ###

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2,
                  "W3": W3,
                  "b3": b3}

    return parameters

tf.reset_default_graph()
with tf.Session() as sess:
    parameters = initialize_parameters()
    print("W1 = " + str(parameters["W1"]))
    print("b1 = " + str(parameters["b1"]))
    print("W2 = " + str(parameters["W2"]))
    print("b2 = " + str(parameters["b2"]))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Expected Output:

            **Z3**

        Tensor("Add_2:0", shape=(6, ?), dtype=float32)

You may have noticed that the forward propagation doesn't output any cache. You will understand why below, when we get to brackpropagation.
2.4 Compute cost
As seen before, it is very easy to compute the cost using:
python
tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = ..., labels = ...))
Question: Implement the cost function below.
- It is important to know that the "logits" and "labels" inputs of tf.nn.softmax_cross_entropy_with_logits are expected to be of shape (number of examples, num_classes). We have thus transposed Z3 and Y for you.
- Besides, tf.reduce_mean basically does the summation over the examples.
Can you write Python code for it?

# GRADED FUNCTION: compute_cost

def compute_cost(Z3, Y):
    """
    Computes the cost

    Arguments:
    Z3 -- output of forward propagation (output of the last LINEAR unit), of shape (6, number of examples)
    Y -- "true" labels vector placeholder, same shape as Z3

    Returns:
    cost - Tensor of the cost function
    """

    # to fit the tensorflow requirement for tf.nn.softmax_cross_entropy_with_logits(...,...)
    logits = tf.transpose(Z3)
    labels = tf.transpose(Y)

    ### START CODE HERE ### (1 line of code)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels))
    ### END CODE HERE ###

    return cost

tf.reset_default_graph()

with tf.Session() as sess:
    X, Y = create_placeholders(12288, 6)
    parameters = initialize_parameters()
    Z3 = forward_propagation(X, parameters)
    cost = compute_cost(Z3, Y)
    print("cost = " + str(cost))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Great! To summarize, remember to initialize your variables, create a session and run the operations inside the session.
Next, you'll also have to know about placeholders. A placeholder is an object whose value you can specify only later.
To specify values for a placeholder, you can pass in values by using a "feed dictionary" (feed_dict variable). Below, we created a placeholder for x. This allows us to pass in a number later when we run the session.
Can you write Python code for it?

# Change the value of x in the feed_dict

x = tf.placeholder(tf.int64, name = 'x')
print(sess.run(2 * x, feed_dict = {x: 3}))
sess.close()


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Note that 12288 comes from $64 \times 64 \times 3$. Each image is square, 64 by 64 pixels, and 3 is for the RGB colors. Please make sure all these shapes make sense to you before continuing.
Your goal is to build an algorithm capable of recognizing a sign with high accuracy. To do so, you are going to build a tensorflow model that is almost the same as one you have previously built in numpy for cat recognition (but now using a softmax output). It is a great occasion to compare your numpy implementation to the tensorflow one.
The model is LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SOFTMAX. The SIGMOID output layer has been converted to a SOFTMAX. A SOFTMAX layer generalizes SIGMOID to when there are more than two classes.
2.1 - Create placeholders
Your first task is to create placeholders for X and Y. This will allow you to later pass your training data in when you run your session.
Exercise: Implement the function below to create the placeholders in tensorflow.
Can you write Python code for it?

# GRADED FUNCTION: create_placeholders

def create_placeholders(n_x, n_y):
    """
    Creates the placeholders for the tensorflow session.

    Arguments:
    n_x -- scalar, size of an image vector (num_px * num_px = 64 * 64 * 3 = 12288)
    n_y -- scalar, number of classes (from 0 to 5, so -> 6)

    Returns:
    X -- placeholder for the data input, of shape [n_x, None] and dtype "float"
    Y -- placeholder for the input labels, of shape [n_y, None] and dtype "float"

    Tips:
    - You will use None because it let's us be flexible on the number of examples you will for the placeholders.
      In fact, the number of examples during test/train is different.
    """

    ### START CODE HERE ### (approx. 2 lines)
    X = tf.placeholder(tf.float32, [n_x, None], name="X")
    Y = tf.placeholder(tf.float32, [n_y, None], name="Y")
    ### END CODE HERE ###

    return X, Y

X, Y = create_placeholders(12288, 6)
print("X = " + str(X))
print("Y = " + str(Y))


I am working on the file "Improving Deep Neural networks- Hyperparameter Tuning - Regularization and Optimization/Tensorflow Tutorial.ipynb".
The first task is:
Writing and running programs in TensorFlow has the following steps:

Create Tensors (variables) that are not yet executed/evaluated.
Write operations between those Tensors.
Initialize your Tensors.
Create a Session.
Run the Session. This will run the operations you'd written above.

Therefore, when we created a variable for the loss, we simply defined the loss as a function of other quantities, but did not evaluate its value. To evaluate it, we had to run init=tf.global_variables_initializer(). That initialized the loss variable, and in the last line we were finally able to evaluate the value of loss and print its value.
Now let us look at an easy example. Run the cell below:
Can you write Python code for it?

a = tf.constant(2)
b = tf.constant(10)
c = tf.multiply(a,b)
print(c)


I am working on the file "Improving accuracy of models.ipynb".
The first task is:
By including the additional modes, and using the higher order integration rule, we see that the modal model can give almost exact results for extinction.
The reactive part is also shown below. The remaining disagreement may be due to the relatively coarse mesh which has been used.
Can you write Python code for it?

plt.figure()
plt.plot(freqs*1e-9, extinction.imag/area)
plt.plot(freqs*1e-9, np.sum(extinction_modes.imag, axis=1)/area, '+')
plt.legend(['Modal decomposition', 'Direct calculation'])
plt.xlim(0, freqs[-1]*1e-9)
plt.xlabel('f (GHz)')
plt.title('Reactive stored energy')
plt.show()


I am working on the file "Improving accuracy of models.ipynb".
The first task is:
Extinction from model of modes
Now we calculate the extinction based on the scalar model for each mode, and also time this calculation
Can you write Python code for it?

for freq_count, s in sim.iter_freqs(freqs):
    V = sim.source_vector(plane_wave, s)
    I_modes = (1/(s - modes.s) + 1/modes.s)*modes.vl.dot(V)
    extinction_modes[freq_count] = V.vdot(modes.vr*I_modes)


I am working on the file "Improving accuracy of models.ipynb".
The first task is:
Find modes by contour integration
Next, we use a more advanced contour integration method to find the location of modes. This is required to find modes which are highly damped, and hence are further away from the $j\omega$ axis.
This method requires more computation than the simpler approach, but it yields more reliable results. It is also necessary to specify a region of complex frequency $s$ in which the modes are expected to be found.
Can you write Python code for it?

contour = RectangularContour(-1e11+1e9j, 5e11j)

estimates = sim.estimate_poles(contour)
refined = sim.refine_poles(estimates)
modes = refined.add_conjugates()

plt.figure()
plt.plot(modes.s.imag, np.abs(modes.s.real), 'x')
plt.xlabel('Frequency $j\omega$')
plt.ylabel('Damping rate $|\Omega|$')
plt.title('Complex eigenfrequencies of modes')
plt.tight_layout()
plt.show()


I am working on the file "Improving accuracy of models.ipynb".
The first task is:
Note the very large number of modes which were found, including some with extremely high damping.
Exact extinction calculation
Again, we calculate the exact result for comparison purposes
Can you write Python code for it?

num_freqs = 101
freqs = np.linspace(1e9, 30e9, num_freqs)

e_inc = np.array([1.0, 1.0, 0])
k_dir = np.array([0, 0, 1])
plane_wave = PlaneWaveSource(e_inc, k_dir, p_inc =1.0)

area = np.pi*(0.5*mesh.max_distance)**2

extinction = np.empty(num_freqs, np.complex128)
extinction_modes = np.empty((num_freqs, len(modes)), np.complex128)

for freq_count, s in sim.iter_freqs(freqs):
    Z = sim.impedance(s)
    V = sim.source_vector(plane_wave, s)
    extinction[freq_count] = np.vdot(V, Z.solve(V))


I am working on the file "Improving accuracy of models.ipynb".
The first task is:
We can now see that the extinction cross-section of each mode can be calculated separately
Can you write Python code for it?

plt.figure()
plt.plot(freqs*1e-9, extinction_modes.real/area);
plt.xlim(0, freqs[-1]*1e-9)
plt.xlabel('f (GHz)')
plt.ylabel('$Q_{ext}$')
plt.title('Extinction efficiency of each mode')
plt.tight_layout()
plt.show()


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/04 - Automatic differentiation with autograd.ipynb".
The first task is:
If we want to compute a function f with respect to x, we need a place to store it.
Communicating that we want plan to store a gradient is done by invoking 'attack_grad()' function.
Can you write Python code for it?


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/04 - Automatic differentiation with autograd.ipynb".
The first task is:
MXNet will build the graph only when it is explicitly said so in order to minimize the computational requirements.
Recoding the gradients it done as follow:
Can you write Python code for it?

with mx.autograd.record():
    y = x * 2
    z = y * x
    # Thus z = 2 * x * x
    v = y / x

z.backward(retain_graph=True)

print(z)

print(x.grad)

print(y)

y.backward(retain_graph=True)

x.grad

v.backward(retain_graph=True)

x.grad


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/05 - Problem Set.ipynb".
The first task is:
Problem 3: Create a 3x3x20 tensor such that at every x,y coordinate, moving through the z coordinate lists the Fibonacci sequence. So, at a z position of 0, the 3x3 matrix will be all 1s. At z-position 1, the 3x3 matrix will be all 1s. At z-position 2, the 3x3 matrix will be all 2s, at z-position 3, the 3x3 matrix will be all 3s and so forth.
Hint: Create the first 2 matrices by hand and then use element-wise operations in a loop to construct the rest of the tensor.
Can you write Python code for it?


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/05 - Problem Set.ipynb".
The first task is:
Problem 4: What is the sum of the vector you created? What is the mean?
Can you write Python code for it?


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/05 - Problem Set.ipynb".
The first task is:
Problem 8: Graph the parabola described in Problem 6 and inspect the slope of y when x = 0 and x = 7. Does it match up with your answer from Problem 6?
Can you write Python code for it?


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/05 - Problem Set.ipynb".
The first task is:
Problems from Automatic differentiation with autograd
Problem 7: The formula for a parabola is y=ax^2+bx+c. If a=5 and b = 13, what is the slope of y when x=0.  How about when x=7?
Can you write Python code for it?


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/05 - Problem Set.ipynb".
The first task is:
Problems from Linear Algebra
Problem 2: Create a 4x4 matrix of random values (where values are uniformly random on the iterval [0,1]. Then create an 4x4 identity matrix (an identity of size n is the n × n square matrix with ones on the main diagonal and zeros elsewhere). Multiply the two together and verify that you get the original matrix back.
Can you write Python code for it?

# Problem 2 Work Area
random_x = mx.nd.random_uniform(shape=[4, 4], low = 0, high = 1)
random_x

identity_m = mx.nd.one_hot(mx.nd.arange(4), depth=4)
identity_m

output = mx.nd.dot(random_x, identity_m)
output


I am working on the file "In Progress - Deep Learning - The Straight Dope/01 - Crash Course/05 - Problem Set.ipynb".
The first task is:
Problems using NDarray (Official Documentation)
Problem 1: Initialize an ndarray of dimension 1x256 on the GPU without overwriting its memory. Then, find the index corresponding to the maximum value in the array (argmax)
Can you write Python code for it?

# Problem 1 Work Area
x = mx.nd.empty(shape=[1, 256], ctx=mx.gpu(0), dtype='float32')
x

mx.nd.argmax(x, axis=1)


I am working on the file "In-class Lectures/April 06/Example of Exploratory Analysis.ipynb".
The first task is:
Notice that the mean of Cluster 0 (the red points) is in between the two clusters
Can you write Python code for it?

# let's create some helper functions
def calculate_cluster_means(X, labels):
    lbls = np.unique(labels)
    print "Cluster labels: {}".format(np.unique(lbls))

    cluster_means = [np.mean(X[labels==num, :], axis=0) for num in range(lbls[-1] + 1)]
    print "Cluster Means: {}".format(cluster_means)
    return cluster_means

def print_3_distances(noise_point, cluster_means):
    # euclidean
    dist = [euclidean(noise_point, cm) for cm in cluster_means]
    print "Euclidean distance: {}".format(dist)

    # chebyshev
    dist = [chebyshev(noise_point, cm) for cm in cluster_means]
    print "Chebysev distance: {}".format(dist)

    # cityblock
    dist = [cityblock(noise_point, cm) for cm in cluster_means]
    print "Cityblock (Manhattan) distance: {}".format(dist)

def plot_the_clusters(X, dbscan_model, noise_point=None):
    labels = dbscan_model.labels_
    clusters = len(set(labels))
    unique_labels = set(labels)
    colors = plt.cm.Spectral(np.linspace(0, 1, len(unique_labels)))

    plt.subplots(figsize=(12,8))

    for k, c in zip(unique_labels, colors):
        class_member_mask = (labels == k)
        xy = X[class_member_mask]
        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=c,
                markeredgecolor='k', markersize=14)

    if noise_point is not None:
        plt.plot(noise_point[0], noise_point[1], 'xr')

    plt.title("Total Clusters: {}".format(clusters), fontsize=14, y=1.01)

def do_yo_thang(X, dbscan_model, noise_point):
    cluster_means = calculate_cluster_means(X, dbscan_model.labels_)
    print_3_distances(noise_point, cluster_means)
    plot_the_clusters(X, dbscan_model, noise_point)

# Let's start playing with scalings
# First we'll do what he did in the book this will have the effect of
# weighting each feature about equally for euclidean distance

X_ss = StandardScaler().fit_transform(X)
db_ss = DBSCAN(eps=0.4, min_samples=3).fit(X_ss)
noise_point = X_ss[30, :]
do_yo_thang(X_ss, db_ss, noise_point)

# Let's make it a little harder now
noisy_prices...


I am working on the file "In-class Lectures/April 06/Example of Exploratory Analysis.ipynb".
The first task is:
test some different scalings
Can you write Python code for it?

# add another helper fxn
def makeX(days, prices):
    return np.concatenate([days, prices], axis=1)

prices_ss = StandardScaler().fit_transform(noisy_prices[:, None])
prices_rob = RobustScaler().fit_transform(noisy_prices[:, None])

days_mm4 = MinMaxScaler(feature_range=(-4,4)).fit_transform(days[:, None])

X_ssmm4 = makeX(days_mm4, prices_ss)
db = DBSCAN(eps=0.6, min_samples=5).fit(X_ssmm4)
# show with X_ss again

noise_point = X_ssmm4[30, :]
do_yo_thang(X_ssmm4, db, noise_point)
for index, xy in enumerate(zip(days_mm4, prices_ss)):
    plt.annotate('{}: ({:0.2f}, {:0.2f})'.format(index, xy[0][0], xy[1][0]), xytext=(xy[0]-0.45, xy[1]-0.15), xy=xy)

print_3_distances(X_ssmm4[50,:], calculate_cluster_means(X_ssmm4, db.labels_))
print
print_3_distances(X_ssmm4[53, :], calculate_cluster_means(X_ssmm4, db.labels_))

X_rbmm4 = makeX(days_mm4, prices_rob)
db = DBSCAN(eps=0.6, min_samples=5).fit(X_ssmm4)
# show with X_ss again

noise_point = X_rbmm4[30, :]
do_yo_thang(X_rbmm4, db, noise_point)
for index, xy in enumerate(zip(days_mm4, prices_rob)):
    plt.annotate('{}: ({:0.2f}, {:0.2f})'.format(index, xy[0][0], xy[1][0]), xytext=(xy[0]-0.45, xy[1]-0.08), xy=xy)

print_3_distances(X_rbmm4[50,:], calculate_cluster_means(X_ssmm4, db.labels_))
print
print_3_distances(X_rbmm4[53, :], calculate_cluster_means(X_ssmm4, db.labels_))


I am working on the file "In-class Lectures/March 09/Lecture_March_9.ipynb".
The first task is:
Exploratory Data Analysis (EDA) is where you should always begin
As well as where you should spend most of your time, see what's happening in the data. Your brain is better than most ML algorithms (at least for now)
Can you write Python code for it?

import matplotlib
import matplotlib.pyplot as plt

matplotlib.style.use('ggplot')

# Pandas has a ton of built-in visualizations
# Play and Learn
# http://pandas.pydata.org/pandas-docs/stable/visualization.html
df.plot.scatter(x='Start_Date', y='Price')

df['Price'].plot.box()

color = dict(boxes='DarkGreen', whiskers='DarkOrange', medians='DarkBlue', caps='Gray')
df['Price'].plot.box(color=color, sym='r+')


I am working on the file "In-class Lectures/March 09/Lecture_March_9.ipynb".
The first task is:
Let's see what happens when we manually add an outlier
Can you write Python code for it?

df = df.set_value(49, 'Price', 55)
# Time for a Google Investigation
# "IQR Outlier"


I am working on the file "InClassAssignment1/error-group-work-template.ipynb".
The first task is:
Finally the relative error.
Can you write Python code for it?

plt.plot(xInterval,(RelErr*100),'-r')
plt.title('Relative Error between Sin(x) and X')
plt.xlabel('X')
plt.ylabel('% Error')
plt.show()


I am working on the file "InClassAssignment1/error-group-work-template.ipynb".
The first task is:
Ideally eps should be 0, but instead we see the machine epsilon and while the value is small it can lead to issues.  Write a loop that multiplies the value c above by 10 and see how the error propagates.
Can you write Python code for it?

loopVal = c
maxVal = 30
for i in range (maxVal):
    loopVal = loopVal * 10

print 'After %s iterations,' % maxVal
print 'loopVal is %s.'% loopVal

epsLoop = (1.0 * 10.0**maxVal) - loopVal
print 'machine epsilon is %s', epsLoop


I am working on the file "InClassAssignment1/error-group-work-template.ipynb".
The first task is:
Model Error
Model error arises in various forms, here we are gonna take some population data and fit two different models and
analyze which model is better for the given data.  Take a look at the code below and comment on the results.
Can you write Python code for it?

# Model Error
time = [0, 1, 2, 3, 4, 5] # hours
growth = [20, 40, 75, 150, 297, 510] # Bacteria Population

time = np.array(time)
growth = np.array(growth)

# First we can just plot the data to visualize it
plt.plot(time,growth,'rs')
plt.title("Scatter plot for the Bacteria population growth over time")
plt.xlabel('Time (hrs)')
plt.ylabel('Population')
plt.show()

# Now we can use the Exponential Model, y = ab^x, to fit the data
a = 20.5122; b = 1.9238;
y = a*b**time[:]
ErrRelExp = abs(y-growth)/abs(growth)
plt.plot(time,growth,'rs',time,y,'-b')
plt.title("Expoenential model fit")
plt.xlabel('Time (hrs)')
plt.ylabel('Population')
plt.legend(["Data", "Exponential Fit"], loc=4)
plt.show()

# Now we can use the Power Model, y = ax^b, to fit the data
a = 32.5846; b = 1.572;
y = a*time[:]**b
ErrRelPow = abs(y-growth)/abs(growth)
plt.plot(time,growth,'rs',time,y,'-b')
plt.title("Power model fit")
plt.xlabel('Time (hrs)')
plt.ylabel('Population')
plt.legend(["Data", "Power Fit"], loc=4)
plt.show()

plt.plot(time,(ErrRelExp*100),"-r",time,(ErrRelPow*100),"-b")
plt.title("Relative Error of Exponential and Power Models")
plt.xlabel('Time (hrs)')
plt.ylabel('Relative Error (%)')
plt.legend(["Exp Model","Pow Model"], loc=1)
plt.show()

##Comments from Dan Judkins##
#1234567890123456789012345678901234567890123456789012345678901234567890123456789
# The Power Model does not appear to fit the data as provided as well as the the
# Exponential Model.However, the measurements accuracy is not given and time is
# short.  If the study were to be performed over a longer period, the power
# model may end up being the more appropriate model given resource constraints
# (i.e. space, food, etc.)


I am working on the file "InClassAssignment1/error-group-work-template.ipynb".
The first task is:
Now plot the absolute error
Can you write Python code for it?

AbsError = abs(ySine - yX)

plt.plot(xInterval,(AbsErr*100),'-g')
plt.title('Absolute Error between Sin(x) and X')
plt.xlabel('X')
plt.ylabel('Error')
plt.show()


I am working on the file "InClassAssignment1/error-group-work-template.ipynb".
The first task is:
The largest floating point number
Use the system library to find the largest floating point value.  Now try to compute some things with this number and see what happens.
Can you write Python code for it?

largestFloat = np.finfo(float).max

ExtraLarge = largestFloat + 10.0

Diff = ExtraLarge - largestFloat

print 'largest floating point number is %s' % largestFloat
print 'add 10.0 to largest float number is %s' % ExtraLarge
print 'Diff between the two is %s' % Diff


I am working on the file "In_Class/ICA1_MachineLearning.ipynb".
The first task is:

Back to Top
Using Linear Regression
In the videos, we derived the formula for calculating the optimal values of the regression weights (you must be connected to the internet for this equation to show up properly):
$$ w = (X^TX)^{-1}X^Ty $$
where $X$ is the matrix of values with a bias column of ones appended onto it. For the diabetes dataset one could construct this $X$ matrix by stacking a column of ones onto the ds.data matrix.
$$ X=\begin{bmatrix}
         & \vdots &        &  1 \
        \dotsb & \text{ds.data} & \dotsb &  \vdots\
         & \vdots &         &  1\
     \end{bmatrix}
$$
Question 1: For the diabetes dataset, how many elements will the vector $w$ contain?
Can you write Python code for it?

# Enter your answer here (or write code to calculate it)

# 11


I am working on the file "In_Class/ICA1_MachineLearning.ipynb".
The first task is:

Back to Top
Start of Live Session Coding
Exercise 2: Scikit-learn also has a linear regression fitting implementation. Look at the scikit learn API and learn to use the linear regression method. The API is here:

API Reference: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

Use the sklearn LinearRegression module to check your results from the previous question.
Question 2: Did you get the same parameters?
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

# write your code here, print the values of model by accessing
#    its properties that you looked up from the API

reg = LinearRegression()
reg.fit(ds.data, ds.target)

print('model coefficients are:', reg.coef_)
print('model intercept is', reg.intercept_)
print('Answer to question is', 'YES.')


I am working on the file "In_Class/ICA1_MachineLearning.ipynb".
The first task is:
Enter Team Member Names here (double click to edit):

Name 1: Ian Johnson
Name 2: Derek Phanekham
Name 3: Travis Siems

In Class Assignment One
In the following assignment you will be asked to fill in python code and derivations for a number of different problems. Please read all instructions carefully and turn in the rendered notebook (or HTML of the rendered notebook)  before the end of class (or right after class). The initial portion of this notebook is given before class and the remainder is given during class. Please answer the initial questions before class, to the best of your ability. Once class has started you may rework your answers as a team for the initial part of the assignment.

Contents

Loading the Data
Linear Regression
Using Scikit Learn for Regression
Linear Classification

Back to Top
Loading the Data
Please run the following code to read in the "diabetes" dataset from sklearn's data loading module.
This will load the data into the variable ds. ds is a bunch object with fields like ds.data and ds.target. The field ds.data is a numpy matrix of the continuous features in the dataset. The object is not a pandas dataframe. It is a numpy matrix. Each row is a set of observed instances, each column is a different feature. It also has a field called ds.target that is a continuous value we are trying to predict. Each entry in ds.target is a label for each row of the ds.data matrix.
Can you write Python code for it?

from sklearn.datasets import load_diabetes
import numpy as np
from __future__ import print_function

ds = load_diabetes()

# this holds the continuous feature data
# because ds.data is a matrix, there are some special properties we can access (like 'shape')
print('features shape:', ds.data.shape, 'format is:', ('rows','columns')) # there are 442 instances and 10 features per instance
print('range of target:', np.min(ds.target),np.max(ds.target))

from pprint import pprint

# we can set the fields inside of ds and set them to new variables in python
pprint(ds.data) # prints out elements of the matrix
pprint(ds.target) # prints the vector (all 442 items)


I am working on the file "In_Class/ICA1_MachineLearning.ipynb".
The first task is:
Exercise 1: In the following empty cell, use the given equation above (using numpy matrix operations) to find the values of the optimal vector $w$. You will need to be sure $X$ and $y$ are created like the instructor talked about in the video. Don't forget to include any modifications to $X$ to account for the bias term in $w$. You might be interested in the following functions:

import numpy as np
np.hstack((mat1,mat2)) stack two matrices horizontally, to create a new matrix
np.ones((rows,cols)) create a matrix full of ones
my_mat.T takes transpose of numpy matrix named my_mat
np.dot(mat1,mat2) or mat1 @ mat2 is matrix multiplication for two matrices
np.linalg.inv(mat) gets the inverse of the variable mat
Can you write Python code for it?

# Write you code here, print the values of the regression weights using the 'print()' function in python
X = np.hstack((ds.data, np.ones((len(ds.target),1))))
w = np.linalg.inv(X.T @ X) @ X.T @ ds.target
print(w)


I am working on the file "In_Class/ICA1_MachineLearning.ipynb".
The first task is:
Exercise 4: Now use linear regression to come up with a set of weights, w, that predict the class value. You can use numpy or sklearn, whichever you prefer. This is exactly like you did before for the diabetes dataset. However, instead of regressing to continuous values, you are just regressing to the integer value of the class (0 or 1), like we talked about in the video (using the hard limit funciton).
 - Note: If you are using numpy, remember to account for the bias term when constructing the feature matrix, X.
Can you write Python code for it?

# write your code here and print the values of the weights

reg = LinearRegression()
reg.fit(ds.data, ds.target)

# Print the weights of the linear classifier.
print('model coefficients are:', reg.coef_)
print('model intercept is', reg.intercept_)


I am working on the file "In_Class/ICA1_MachineLearning.ipynb".
The first task is:
Exercise 5: Finally, use a hard decision function on the output of the linear regression to make this a binary classifier. This is just like we talked about in the video, where the output of the linear regression passes through a function:

$\hat{y}=g(w^TX^T)$ where
$g(w^TX^T)$ for $w^TX^T < \alpha$ maps the predicted class to 0
$g(w^TX^T)$ for $w^TX^T \geq \alpha$ maps the predicted class to 1.

Here, alpha is a threshold for deciding the class.
Question 3: What value for $\alpha$ makes the most sense? What is the accuracy of the classifier given the $\alpha$ you chose?
Note: You can calculate the accuracy with the following code: accuracy = float(sum(yhat==y)) / len(y) assuming you choose variable names y and yhat for the target and prediction, respectively.
Can you write Python code for it?

# use this box to predict the classification output
y_hat = reg.predict(ds.data)

#Populate list of possible alpha values
alphas = [n/100.0 for n in range(-100, 100)]
ls = [ (float(sum((y_hat > a)==ds.target)) / len(ds.target), a) for a in alphas]

print('Percentage accuracy:', max(ls)[0])
print('Alpha value: ', max(ls)[1])


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:

Back to Top
Non-linear SVMs
Now let's explore the use of non-linear svms. More explicitly, using different kernels. Take a look at the example training and testing code below for the non-linear SVM. All parameters are left as default, except we change the kernel to be rbf. Run the block of code below.
Can you write Python code for it?

from sklearn.svm import SVC

clf = SVC(kernel='rbf')
clf.fit(Xpca,y)
yhat = clf.predict(Xpca)
print('Overall Accuracy is ',accuracy_score(y,yhat))


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:

Back to Top
Using Linear SVMs
Exercise 1: Use the block of code below to check if the number of parameters you calculated is equal to the number of parameters returned by sklearn's implementation of the Linear SVM. Was your calculation correct? If different, can you think of a reason why the parameters would not match?
Can you write Python code for it?

from sklearn.svm import LinearSVC
from sklearn.decomposition import RandomizedPCA

n_components = 100
pca = RandomizedPCA(n_components=n_components)
Xpca = pca.fit_transform(X)

clf = LinearSVC()
clf.fit(Xpca,y)

#===================================================================
# Enter you code below to calculate the number of parameters in the model

print(sum([len(x) for x in clf.coef_]) + len(clf.intercept_))
#===================================================================


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:
A. The highest accuracy we could achieve was 1.0, or 100% accuracy. This was achieved with a gamma value of 0.02
B. We would not expect this to generalize because this gamma value is large enough that the model is likely fitting to noise in the original dataset. Therefore, this model may not work well on other photos and the given gamma value may not be ideal for creating other models, depending on the number of attributes.

Final Question: Using the most accurate non-linear SVM you found in the previous question, how many parameter coefficients does the trained model contain?
181,292 parameter coefficients exist in the trained model.
Can you write Python code for it?

#===================================================
# Enter any scratchwork calculations you need below
svc = SVC(kernel=kern, gamma=.02)
svc.fit(Xpca, y)
print(sum([len(x) for x in svc.dual_coef_]) + len(svc.intercept_))


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:
Enter Team Member Names here (double click to edit):

Name 1: Ian Johnson
Name 2: Derek Phanekham
Name 3: Travis Siems

In Class Assignment Two
In the following assignment you will be asked to fill in python code and derivations for a number of different problems. Please read all instructions carefully and turn in the rendered notebook (or HTML of the rendered notebook)  before the end of class (or right after class). The initial portion of this notebook is given before class and the remainder is given during class. Please answer the initial questions before class, to the best of your ability. Once class has started you may rework your answers as a team for the initial part of the assignment.

Contents

Loading the Data
Linear SVMs
Using Linear SVMs
Non-linear SVMs

Back to Top
Loading the Data
Please run the following code to read in the "olivetti faces" dataset from sklearn's data loading module.
This will load the data into the variable ds. ds is a bunch object with fields like ds.data and ds.target. The field ds.data is a numpy matrix of the continuous features in the dataset. The object is not a pandas dataframe. It is a numpy matrix. Each row is a set of observed instances, each column is a different feature. It also has a field called ds.target that is an integer value we are trying to predict (i.e., a specific integer represents a specific person). Each entry in ds.target is a label for each row of the ds.data matrix.
Can you write Python code for it?

# fetch the images for the dataset
# this will take a long time the first run because it needs to download
# after the first time, the dataset will be save to your disk (in sklearn package somewhere)
# if this does not run, you may need additional libraries installed on your system (install at your own risk!!)
from sklearn.datasets import fetch_lfw_people

lfw_people = fetch_lfw_people(min_faces_per_person=20, resize=None)

# get some of the specifics of the dataset
X = lfw_people.data
y = lfw_people.target
names = lfw_people.target_names

n_samples, n_features = X.shape
_, h, w = lfw_people.images.shape
n_classes = len(names)

print("n_samples: {}".format(n_samples))
print("n_features: {}".format(n_features))
print("n_classes: {}".format(n_classes))
print("Original Image Sizes {} by {}".format(h,w))
print (125*94) # the size of the images are the size of the feature vectors


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:
Exercise 3: Use the starter code from above to calculate the accuracy for three different non-linear SVM kernels. That is, repeat the code above for different kernel parameters.  Which kernel is most accurate with the default parameters?
You might be interested in the documentation of the scikit-learn SVM implementation, available here:
- http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
Can you write Python code for it?

#===================================================
# Enter your code below
clf = SVC(kernel='rbf')
clf.fit(Xpca,y)
yhat = clf.predict(Xpca)
print('Overall Accuracy is ',accuracy_score(y,yhat), 'for the rbf kernel')

clf = SVC(kernel='poly')
clf.fit(Xpca,y)
yhat = clf.predict(Xpca)
print('Overall Accuracy is ',accuracy_score(y,yhat), 'for the poly kernel')

clf = SVC(kernel='sigmoid')
clf.fit(Xpca,y)
yhat = clf.predict(Xpca)
print('Overall Accuracy is ',accuracy_score(y,yhat), 'for the sigmoid kernel')
#===================================================


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:
Our calculation is the same. Between the 6200 items in the coefficient matrix and the 62 values in the intercept vector, the total number of weights computed is 6262, which is what we originally calculated

Exercise 2: Use the starter code below to calculate two quantities:
- Part A.: The overall accuracy of the trained linear svm on the training set
- Part B.: The mean, standard deviation, maximum, and minimum of the accuracy per class on the training set
You might be interested in the following documentation of the confusion matrix calculated by scikit-learn:
- http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
And an example matrix returned by the confusion matrix function:
<img src="http://scikit-learn.org/stable/_images/plot_confusion_matrix_001.png",width=400,height=400>
Can you write Python code for it?

from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
import numpy as np

yhat = clf.predict(Xpca)

#===================================================
# Enter your code below
print('Overall Accuracy is ',sum(yhat == y) / len(y))
#print(clf.score(Xpca, y))

list_accuracies = [x[i] / sum(x) for i, x in enumerate(confusion_matrix(y, yhat))]

print('The class accuracy is ',np.mean(list_accuracies), '+-', np.std(list_accuracies),end=' ')
print('(min,max) (',min(list_accuracies), ',' ,max(list_accuracies),')')

#===================================================


I am working on the file "In_Class/ICA2_MachineLearning.ipynb".
The first task is:
The most accurate kernel with default parameters is the poly kernel.

Exercise 4: Choose the most accurate kernel and manipulate the settings for gamma to make the classification more accurate.
- Part A: How accurate can you make it?
- Part B: Would you expect the results to generalize well? Why or why not?
Can you write Python code for it?


I am working on the file "In_Class/ICA2_MachineLearning_PartA.ipynb".
The first task is:
Question 1: For the faces dataset, describe what the data represents? That is, what is each column? What is each row? What do the unique class values represent?
Every column is a pixel location in a 125x94 photograph.
Each row is a single image of someone's face.
The unique class values are the names of the people in the photographs.

Back to Top
Linear Support Vector Machines
Question 2: If we were to train a linear Support Vector Machine (SVM) upon the faces data, how many parameters would need to be optimized in the model? That is, how many coefficients would need to be calculated?
11750
Can you write Python code for it?

# Enter any scratchwork or calculations here


I am working on the file "In_Class/ICA2_MachineLearning_PartA.ipynb".
The first task is:
Question 3:
- Part A: Given the number of parameters calculated above, would you expect the model to train quickly using batch optimization techniques? Why or why not?
- Part B: Is there a way to reduce training time?
- Part C: If we transformed the X data using principle components analysis (PCA) with 100 components, how many parameters would we need to find for a linear Support Vector Machine (SVM)?
Enter you answer here (double click)
A. No. Parallelizing wouldn't scale well to 11,750 threads.
B. Train on a subsample
C. 100
Can you write Python code for it?

# Enter any scratchwork or calculations here

print('Part C. With 100 features: ', '100')


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:

Back to Top
KDDCup Evaluation and Cross Validation
Can you write Python code for it?

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold, ShuffleSplit
from sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit

from sklearn.metrics import make_scorer, accuracy_score
from sklearn.metrics import precision_score, recall_score, f1_score

from sklearn.linear_model import LogisticRegression

# select model
clf = LogisticRegression()
#select cross validation
cv = KFold(n_splits=10)
# select evaluation criteria
my_scorer = make_scorer(accuracy_score)
# run model training and cross validation
per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=cv,
                                    scoring=my_scorer
                                   )

plt.bar(range(len(per_fold_eval_criteria)),per_fold_eval_criteria)
plt.ylim([min(per_fold_eval_criteria)-0.01,max(per_fold_eval_criteria)])


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
DARPA is also concerned about how much training data they will need from the analysts in order to have a high performing model. They would like to use the current dataset to help answer that question. The code below is written for you to help answer DARPA's question about how many examples will be needed for training. Examine the code and then answer the following question:
Question 6: Based on the analysis graphed below, how many positive examples are required to have a good tradeoff between bias and variance for the given evaluation criteria? Why?
Can you write Python code for it?

clf = LogisticRegression()

test_scores = []
train_sizes=np.linspace(5e-4,5e-3,10)

for size in train_sizes:
    cv = StratifiedShuffleSplit(n_splits=100,
                                train_size = size,
                                test_size = 1-size,
                               )
    test_scores.append(cross_val_score(estimator=clf,X=X,y=y,cv=cv,scoring=my_scorer))

plot_filled(np.array(test_scores), train_sizes*100, 'Percentage training data (%)')

print(.0015 * len(X))


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
Enter Team Member Names here (double click to edit):

Name 1: Ian Johnson
Name 2: Travis Siems
Name 3: Derek Phanekham

In Class Assignment Three
In the following assignment you will be asked to fill in python code and derivations for a number of different problems. Please read all instructions carefully and turn in the rendered notebook (or HTML of the rendered notebook)  before the end of class (or right after class). The initial portion of this notebook is given before class and the remainder is given during class. Please answer the initial questions before class, to the best of your ability. Once class has started you may rework your answers as a team for the initial part of the assignment.

Contents

Loading KDDCup Data
KDDCup Evaluation and Cross Validation
More Cross Validation
Statistical Comparison

Before coming to class, please make sure you have the latest version of scikit-learn. This notebook was created for version 0.18 and higher.

Back to Top
Loading KDDCup Data
Please run the following code to read in the "KDD Cup" dataset from sklearn's data loading module. It consists of examples of different simulated attacks for the 1998 DARPA Intrusion Detection System (IDS).
This will load the data into the variable ds. ds is a bunch object with fields like ds.data and ds.target. The field ds.data is a numpy matrix of the continuous features in the dataset. The object is not a pandas dataframe. It is a numpy matrix. Each row is a set of observed instances, each column is a different feature. It also has a field called ds.target that is an integer value we are trying to predict (i.e., a specific integer represents a specific person). Each entry in ds.target is a label for each row of the ds.data matrix.
Can you write Python code for it?

# fetch the dataset
from sklearn.datasets import  fetch_kddcup99
from sklearn import __version__ as sklearn_version

print('Sklearn Version:',sklearn_version)
ds =  fetch_kddcup99(subset='http')

import numpy as np
# get some of the specifics of the dataset
X = ds.data
y = ds.target != b'normal.'

n_samples, n_features = X.shape
n_classes = len(np.unique(y))

print("n_samples: {}".format(n_samples))
print("n_features: {}".format(n_features))
print("n_classes: {}".format(n_classes))


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
For this question, the circumstances for the DARPA KDD99 cup are changed in the following way:
- When the model for detecting attacks is deployed, we now think that it will often need to be retrained.
- DARPA anticipates that there will be a handful of different style attacks on their systems that have never been seen before. To detect these new attacks, they are employing programmers and analysts to find them manually every day.
- DARPA believes the perpetrators of these new attacks are more sophisticated, so finding the new attacks will take priority over detecting the older, known attacks.
- DARPA wants to use your learning algorithm for detecting only these new attacks, but the amount of training and testing data will be extremely small, because the analysts can only identify a handful of new style attacks each day.
- DARPA asks you if you think its a good idea to employ retraining your model each day to find these new attacks.
Question 5: How would you change the method of cross validation to answer this question from DARPA? That is, how can you change your cross validation method to better mirror how your system will be used and deployed by DARPA?
We would use the time-series approach discussed in the flipped lecture, where we use all existing data to try to classify the new data from every day.
At day 0 (the day that we start identifying the new type of attack), we would build a new dataset consisting of all of the new messages, including the new sophisticated attacks, which we would label as sophisticated attacks.
At day N, we would use all of the collected data from days 0..N (not including N) to build a model to classify the data from day N. We would then compare the model's output (for predicting day N) to the actual measured classes of the data from day N. We would compute the F1 score of the model with respect to identifying only sophisticated attacks, and use that to evaluate the model, or we could use a cost matrix where the cost of misidentifying a sophisiticated attack is much higher than...
Can you write Python code for it?

#plotting function for use in next question
# takes input 'test_scores', and an x-axis label
def plot_filled(test_scores,train_x_axis, xlabel=''):

    test_mean = np.percentile(test_scores,50, axis=1)
    test_max = np.percentile(test_scores,95, axis=1)
    test_min = np.percentile(test_scores,5, axis=1)

    plt.plot(train_x_axis, test_mean,
             color='blue', linestyle='--',
             marker='s', markersize=5,
             label='validation set')

    plt.fill_between(train_x_axis,
                     test_min,
                     test_max,
                     alpha=0.15, color='blue')

    plt.grid(True)
    plt.xlabel(xlabel)
    plt.ylabel('Evaluation Criterion')
    plt.legend(loc='lower right')
    plt.tight_layout()


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
It seems that approximately 0.15% of the data must be comprised of positive examples in order to optimize the tradeoff between vias and variance.
For the entire dataset, this means about 88 examples.

Back to Top
Statistical Comparison
Now lets create a few different models and see if any of them have statistically better performances.
We are creating three different classifiers below to compare to one another. For creating different training and testing splits, we are using stratified shuffle splits on the datasets.
Can you write Python code for it?

clf1 = LogisticRegression(C=100)
clf2 = LogisticRegression(C=1)
clf3 = LogisticRegression(C=0.1)

train_size = 0.003 # small training size
cv = StratifiedShuffleSplit(n_splits=10,train_size=train_size,test_size=1-train_size)

evals1 = cross_val_score(estimator=clf1,X=X,y=y,scoring=my_scorer,cv=cv)
evals2 = cross_val_score(estimator=clf2,X=X,y=y,scoring=my_scorer,cv=cv)
evals3 = cross_val_score(estimator=clf3,X=X,y=y,scoring=my_scorer,cv=cv)


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
Question 1: How many instances are in the binary classification problem loaded above? How many instances are in each class? Plot a pie chart or bar chart of the number of classes.
Can you write Python code for it?

from matplotlib import pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

#=== Fill in code below========
print('Total number of instances', len(y))
print('Number of instances in each class:',np.bincount(y))

vals = np.bincount(y)
plt.bar(range(len(vals)), vals)
plt.ylabel('Counts')
plt.title('Counts of Normal and Abnormal Events')
plt.xticks(range(2), ('Normal','Abnormal'))


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
Question 2 Is the code above a proper separation of training and testing sets for the given dataset? Why or why not?
For this dataset, due to the class imbalance, stratified K fold partitioning should be used to guarantee that each of the folds actually includes examples of each class. A normal K-fold partitioning will result in only a handful of folds including any of the abnormal "attack" messages.
Question 3: Is the evaluation metric chosen in the above code appropriate for the dataset? Why or Why not?
No, accuracy is not a good metric in this case because there is a significant class imbalance problem. Accuracy is not meaningful because there are so many normal messages in the data and very few abnormal messages in the data.

Exercise 1: If the code above is not a proper separation of the train or does not use the proper evaluation criteria, fix the code in the block below to use appropriate train/test separation and appropriate evaluation criterion (criteria).
Can you write Python code for it?

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold, ShuffleSplit
from sklearn.model_selection import StratifiedKFold, StratifiedShuffleSplit

from sklearn.metrics import make_scorer, accuracy_score
from sklearn.metrics import precision_score, recall_score, f1_score

from sklearn.linear_model import LogisticRegression
# these imports above might help you

#=====Write your code below here=================
# select model
clf = LogisticRegression()
#select cross validation
cv = StratifiedKFold(n_splits=10) ##CHANGED TO STRATIFIED K FOLD
# select evaluation criteria
my_scorer = make_scorer(f1_score) ##SWITCHED TO F1_SCORE
# run model training and cross validation
per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=cv,
                                    scoring=my_scorer
                                   )

plt.bar(range(len(per_fold_eval_criteria)),per_fold_eval_criteria)
plt.ylim([min(per_fold_eval_criteria)-0.01,max(per_fold_eval_criteria)])

print("Mean F1: ", np.mean(per_fold_eval_criteria))


I am working on the file "In_Class/ICA3_MachineLearning.ipynb".
The first task is:
Question 7: Given the code above, what statistical test is more appropriate for selecting confidence intervals, and why? Your options are:
- A: approximating the evaluation criterion as a binomial distribution and bounding by the variance (the first option we used in the flipped lecture video)
- B: approximating the bounds using the folds of the cross validation to get mean and variance (the second option we used in the flipped lecture video)
- C: Either are acceptable statistical tests for obtaining confidence intervals
Answer: B
A) is not an acceptable answer, because, since the training size is so small, the testing sets are not independent, and the binomial approximation requires that the datasets are independent.
B) is an acceptable answer, because we have 3 sets of scores for the three classifiers, so we can compute the mean and variance of the difference between these sets. This is the only acceptable test in this scenario.
C) is not an acceptable answer, since A is incorrect.

Final Exercise: With 95% confidence, perform the statistical test that you selected above. Is any model or set of models statistically the best performer(s)? Or can we not say if the models are different with greater than 95% confidence?
If you chose option A, use a multiplier of Z=1.96. The number of instances used in testing can be calculated from the variable train_size.
If you chose option B, use a multiplier of t=2.26 and k=10.
Can you write Python code for it?

#===================================================
# Enter your code below
diff12 = evals1 - evals2
diff13 = evals1 - evals3
diff23 = evals2 - evals3

sigma12 = np.sqrt(np.sum(diff12*diff12) * 1/(10-1))
sigma13 = np.sqrt(np.sum(diff13*diff13) * 1/(10-1))
sigma23 = np.sqrt(np.sum(diff23*diff23) * 1/(10-1))

d12 = (np.mean(diff12) + 1/(np.sqrt(10) * 2.26 * sigma12), np.mean(diff12) - 1/(np.sqrt(10) * 2.26 * sigma12))
d13 = (np.mean(diff13) + 1/(np.sqrt(10) * 2.26 * sigma13), np.mean(diff13) - 1/(np.sqrt(10) * 2.26 * sigma13))
d23 = (np.mean(diff23) + 1/(np.sqrt(10) * 2.26 * sigma23), np.mean(diff23) - 1/(np.sqrt(10) * 2.26 * sigma23))

print('Models 1 and 2 have statistically the best F1_score with 95% confidence (compared to model 3)')
print('Models 1 and 2 do not have statistically different F1 scores with 95% confidence.')
#===================================================


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Calculate the percent of each party in 2000
Can you write Python code for it?

def calculateNP00(row):
    return row['NP2000'] / row['Total2000']
def calculateDem00(row):
    return row['Dem2000'] / row['Total2000']
def calculateGOP00(row):
    return row['GOP2000'] / row['Total2000']

distributions00 = totalRow.compute([
        ('nonparty_distribution2000', agate.Formula(agate.Number(), calculateNP00)),
        ('demparty_distribution2000', agate.Formula(agate.Number(), calculateDem00)),
        ('gopparty_distribution2000', agate.Formula(agate.Number(), calculateGOP00)),
    ])


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Calculate the percent of each party in 2014
Can you write Python code for it?

def calculateNP14(row):
    return row['NP2014'] / row['Total2014']
def calculateDem14(row):
    return row['Dem2014'] / row['Total2014']
def calculateGOP14(row):
    return row['GOP2014'] / row['Total2014']

distributions14 = totalRow.compute([
        ('nonparty_distribution2014', agate.Formula(agate.Number(), calculateNP14)),
        ('demparty_distribution2014', agate.Formula(agate.Number(), calculateDem14)),
        ('gopparty_distribution2014', agate.Formula(agate.Number(), calculateGOP14)),
    ])


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Check out column names & types from parties table
Can you write Python code for it?


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Create agate Table of csv file from Nebraska registered voters data from 2000 to 2014
Can you write Python code for it?


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Let's see some stats about Nebraska political party distributions and calculate when the independent party would overtake the Democratic Party, given the growth rates continue.
Grab the total row form the parties table
Can you write Python code for it?

totalRow = parties.where(lambda row: row['County Name'] == 'Total State')
print(len(totalRow.rows))


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Lets round those values to one decimal place
Can you write Python code for it?

from decimal import Decimal

def round_total_change(row):
    return row['total_change'].quantize(Decimal('0.1'))

def round_nonparty_change(row):
    return row['nonparty_change'].quantize(Decimal('0.1'))

def round_gopparty_change(row):
    return row['gopparty_change'].quantize(Decimal('0.1'))

def round_demparty_change(row):
    return row['demparty_change'].quantize(Decimal('0.1'))

rounded_changes = changes0014.compute([
    ('total_change_round', agate.Formula(agate.Number(), round_total_change)),
    ('nonparty_change_round', agate.Formula(agate.Number(), round_nonparty_change)),
    ('gopparty_change_round', agate.Formula(agate.Number(), round_gopparty_change)),
    ('demparty_change_round', agate.Formula(agate.Number(), round_demparty_change)),
])


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
Ugh it's ugly lets see only the fields we need
Can you write Python code for it?

for_printing = rounded_changes.select(['County Name', 'total_change_round', 'nonparty_change_round', 'gopparty_change_round', 'demparty_change_round'])


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
View each party's percent in 2000
Can you write Python code for it?

printDistribution00 = distributions00.select(['nonparty_distribution2000', 'demparty_distribution2000', 'gopparty_distribution2000'])
printDistribution00.print_table()


I am working on the file "IndependentStory/agatePolitical.ipynb".
The first task is:
View each party's percent in 2014
Can you write Python code for it?

printDistribution14 = distributions14.select(['nonparty_distribution2014', 'demparty_distribution2014', 'gopparty_distribution2014'])
printDistribution14.print_table()


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
3.2 Bottlenecks
We used line profiler to identify the potential bottlenecks.
From the result, it can be observed that the biggest bottleneck in the fastica function are "np.tanh" ("g_logcosh" and "gprime_logcosh") and 'np.linalg.svd' function, which take about 90% percent of the time when running fastica function. The next bottleneck are 'preprocessing.scale'and matrix dot product.
3.3 Parallelism for tanh
After identifying “np.tanh” function as our biggest bottleneck, we trid to solve it by multi-core parallelism. To easily compare the performance, here we dealt with a “big” array ($10^5 \times 10^3$).
Can you write Python code for it?

import numpy as np
import numexpr as ne
# "big" array
TT = np.random.normal(size=(10**5,10**3))

%%time
res = np.tanh(TT)

from concurrent.futures import ThreadPoolExecutor
import multiprocessing as mp

%%time
with ThreadPoolExecutor(max_workers=4) as pool:
    res = pool.map(np.tanh, [i for i in TT])

%%time
with mp.Pool(processes=4) as pool:
    res = pool.map(np.tanh, [i for i in TT])


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
4.1.1 True Source Components
Can you write Python code for it?


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
4.1.2 ICs by FastICA
Can you write Python code for it?

test(fastICA_ne(X_test,f = "logcosh",n_comp =2, alpha = 1,maxit = 200, tol = 0.0001)['S'])


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
4.2 Real World Data
4.2.1 fMRI Data
The real-world data set is a subsample of real fMRI data, which is from the 1000 Connectome Project. The data is a 30000 by 20 matrix, where 30000 is the number of voxels in brain. We applied our FastICA algorithm to the dataset, and separated the original mixed data to 20 independent components. From the plots, it seems that our algorithm successfully separated the mixed data. In the plot of the original data, we can see that all 20 features are very noised and have no apparent patterns. However, in the plot of the 20 independent components after using FastICA algorithm, it appears that the pattern of the signals become more apparent, even though there are still a lot of noise present. Considering the fact that this data set is real-world data with many noises, we would conclude that there is still more modifications and tests to be conducted to our algorithm in order to achieve more sensible results. With this potential for improvement in mind, we decided to proceed with caution and continue to improve our algorithm later.
Can you write Python code for it?


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
4.2.2 Finance Data
Our second real data application is in finance. Specifically, our application follows the thoughts of paper “A first application of independent component analysis to extracting structure from stock returns”[3].
We looked at the 2010-2017 stock prices of six top technology companies: Intel Corporation (INTC), Cisco Systems, Inc. (CSCO), QUALCOMM Incorporated (QCOM), eBay Inc. (EBAY), Apple Inc. (AAPL), Amazon.com, Inc. (AMZN) (data comes from Yahoo Finance). To make data more stationary, here we focused on the geometric growth of stock returns: $log(p(t))-log(p(t-1))$. Then we treated the stock returns of each company as a mixed signal, and used ICA to find potential driving factors. (Data manipulation details are in the Examples.ipynb)
The stock return changes for the six technology companies from 2010-2017 is shows in the first plot below. Except two big jumps, their stock returns were relatively stable over time. The independent components obtained by our FastICA algorithm was show in the second plot below. Unlike the stock returns we observed, they were more fluctuant. Some of them offset the effect of others. Specifically, at the two stock return jump time points, there were two significant ICs jumps in different direction, suggesting interaction like income effect and substitution effect. Based on the ICs obtained through ICA, the reconstructed signals were shown in the third plot. It was almost the same as our observed stock returns, suggesting ICA did a good job.
Next, we took a close look at the most recent 100 trading days’ stock returns. The results were shown in the last eight small plots. Again, the ICA reconstructed signals almost resembled the patterns of the stock return data.
Can you write Python code for it?

%run Test/finance.py
X_finance.shape

np.random.seed(663)
ica_finance = fastICA_ne(np.array(X_finance),f = "logcosh",n_comp=6, alpha = 1,maxit = 200, tol = 0.0001)

%run Test/plot_finance.py
plot_finance(X_finance,ica_finance)


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
5. Comparative Analysis with Competing Algorihtms
Our FastICA algorithm is compared with the PCA and factor analysis.They all solve a similar problem of finding linear representation of the multivariate data. Principal component analysis is a latent linear variable model which however assumes equal noise variance for each feature. This extra assumption makes probabilistic PCA faster as it can be computed in closed form.
Factor analysis is a simple linear generative model with Gaussian latent variables. The observations are assumed to be caused by a linear transformation of lower dimensional latent factors and added Gaussian noise.
We compared FastICA algorithm with PCA and factor analysis in terms of their accuracy of performing the linear transformations. In order to check the accuracy, the recovered signals should be compared with the true sources, and the recover signals with more similar shapes to the true sources are considered having better performance for linear transformation. From the resulting plots below, FastICA algorithm produces the best results among the three algorithm, as the shapes of the recovered signals perfectly match the original true sources. PCA and factor analysis have similar performance, but both fail to separate the signals completely. In the plots for factor analysis and PCA, the yellow and blue signals are still mixed. Therefore, by comparing the results of the three algorithms, we find that FastICA algorithm works best for the blind source separation problem.
Can you write Python code for it?

%run Test/comparedata.py

S_ica = fastICA_ne(X_compare,f = "logcosh", alpha = 1,maxit = 200, tol = 0.0001)['S']

%run Test/plot_compare.py
plot_compare(X_compare,S_compare,S_ica)


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
According the results, changing functions (i.e. from numpy to scipy) didn’t give us significant improvement. However, after changing the order of while and for loop (better algorithm), the running time suddenly dropped (from fastICA_0 to fastICA_3). JIT (just-in-time compiler) didn't show any improvement. The reason maybe some functions are not supported in nopython mode. Cython didn't do a good job either (see our Test.ipybn). Numexpr further enhanced efficiency, as it successfully solved our tanh bottleneck.
4. Application
4.1 Simulated Data and Code Testing
We used simulated data sets to test our FastICA algorithm. Since ICA is used for separating mixed signals, we generated two signals (Source Matrix S and Mixing Matrix A), mixed them to get the mixed signal(X). Then, we apply our FastICA algorithm to X, and compared the plots of two signals that were produced by our algorithm with the plots of the two original signals, i.e. S and A. The fact that the shapes of those signals match confirms that our FastICA algorithm can successfully separate mixed signals. (We did code testing for all our FastICA scripts. For test details, please see Test_Optimization.ipynb)
Can you write Python code for it?


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
After appplying FastICA algorithm, we visualize each independent component in the following plot
Can you write Python code for it?

ica_fMRI = fastICA_ne(fMRI,f = "logcosh", alpha = 1,maxit = 200, tol = 0.0001)
S_fMRI = ica_fMRI['S']
fig = plt.figure(figsize=(10,6))
for i in range(20):
    ax = plt.subplot(5,4,i+1)
    plt.plot(np.arange(30000)+1, S_fMRI[:,i])
    plt.xticks([])
    plt.yticks([])
fig.suptitle('Visualization of the Independent Components by ICA', fontsize=15)
pass


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
Numexpr worked surprisingly well. As you can see, Numexpr using MKL can be up to 5x faster than numpy, successfully solving our "tanh" bottleneck. For most running times, VML did accelerate computations, and more threads made computations faster (although the results aren't stable).
3.4 Optimization for Performance
We tried several methods here: replacing preprocessing function with numpy.mean,replacing numpy functions with scipy functions, better algorithm, jit (just-in-time compiler), vectorization, Cython, and Numexpr. The efficiency of major update functions are shown below (All our algorithm scripts can be found in Source.ipynb).
Can you write Python code for it?

%run Test/fMRI.py
%run Source/fastICA_0.py
%run Source/fastICA_1.py
%run Source/fastICA_3.py
%run Source/fastICA_scipy.py
%run Source/fastICA_jit.py
%run Source/fastICA_ne.py

%timeit -r2 -n4 fastICA_0(fMRI,f = "logcosh",n_comp =20, alpha = 1,maxit = 200, tol = 0.0001)

%timeit -r2 -n4 fastICA_1(fMRI,f = "logcosh",n_comp =20, alpha = 1,maxit = 200, tol = 0.0001)

%timeit -r2 -n4 fastICA_scipy(fMRI,f = "logcosh",n_comp =20, alpha = 1,maxit = 200, tol = 0.0001)

%timeit -r2 -n4 fastICA_3(fMRI,f = "logcosh",n_comp =20, alpha = 1,maxit = 200, tol = 0.0001)

%timeit -r2 -n4 fastICA_jit(fMRI,f = "logcosh",n_comp =20, alpha = 1,maxit = 200, tol = 0.0001)

%timeit -r2 -n4 fastICA_ne(fMRI,f = "logcosh",n_comp =20, alpha = 1,maxit = 200, tol = 0.0001)


I am working on the file "Independent_Component_Analysis.ipynb".
The first task is:
The results above suggested no significant improvement. Although multiprocessing sped up CPU times, the large amount of system time offset its advantage. Therefore, we shifted to Numexpr. Numexpr is a fast numerical expression evaluator for NumPy. It avoids allocating memory for intermediate results, which results in better cache utilization and reduces memory access in general. Since ICA algorithm always deals with signals data (large arrays), we thought it would be a good fit. Specifically, we noticed that Numexpr has support for Intel's VML in order to accelerate the evaluation of transcendental functions on Intel CPUs. Here we’d like to try both with and without VML
Can you write Python code for it?

ne.use_vml=False

%%time
res = ne.evaluate('tanh(TT)')

ne.use_vml=True

%%time
res = ne.evaluate('tanh(TT)')

ne.set_num_threads(1)

%%time
res = ne.evaluate('tanh(TT)')

ne.detect_number_of_cores()

ne.set_num_threads(ne.detect_number_of_cores())

%%time
res = ne.evaluate('tanh(TT)')


I am working on the file "IndianPinesCNN.ipynb".
The first task is:
Define the Evaluation OP
Can you write Python code for it?

def evaluation(logits, labels):
    """Evaluate the quality of the logits at predicting the label.
    Args:
    logits: Logits tensor, float - [batch_size, NUM_CLASSES].
    labels: Labels tensor, int32 - [batch_size], with values in the
      range [0, NUM_CLASSES).
    Returns:
    A scalar int32 tensor with the number of examples (out of batch_size)
    that were predicted correctly.
    """
    # For a classifier model, we can use the in_top_k Op.
    # It returns a bool tensor with shape [batch_size] that is true for
    # the examples where the label is in the top k (here k=1)
    # of all logits for that example.
    correct = tf.nn.in_top_k(logits, labels, 1)
    # Return the number of true entries.
    return tf.reduce_sum(tf.cast(correct, tf.int32))


I am working on the file "IndianPinesCNN.ipynb".
The first task is:
Define the Training OP
Can you write Python code for it?

def training(loss, learning_rate):
    """Sets up the training Ops.
    Creates a summarizer to track the loss over time in TensorBoard.
    Creates an optimizer and applies the gradients to all trainable variables.
    The Op returned by this function is what must be passed to the
    `sess.run()` call to cause the model to train.
    Args:
    loss: Loss tensor, from loss().
    learning_rate: The learning rate to use for gradient descent.
    Returns:
    train_op: The Op for training.
    """
    # Add a scalar summary for the snapshot loss.
    tf.scalar_summary(loss.op.name, loss)
    # Create the gradient descent optimizer with the given learning rate.
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    # Create a variable to track the global step.
    global_step = tf.Variable(0, name='global_step', trainable=False)
    # Use the optimizer to apply the gradients that minimize the loss
    # (and also increment the global step counter) as a single training step.
    train_op = optimizer.minimize(loss, global_step=global_step)
    return train_op


I am working on the file "IndianPinesCNN.ipynb".
The first task is:
Define the loss function
Can you write Python code for it?

def loss(logits, labels):
    """Calculates the loss from the logits and the labels.
    Args:
    logits: Logits tensor, float - [batch_size, NUM_CLASSES].
    labels: Labels tensor, int32 - [batch_size].
    Returns:
    loss: Loss tensor of type float.
    """
    labels = tf.to_int64(labels)
    cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(
      logits, labels, name='xentropy')
    loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
    return loss


I am working on the file "India’s_batting_performance_2016.ipynb".
The first task is:
Create a Dateframe of unique players name and their maximum score
Can you write Python code for it?

ndf_16.Player.unique()

playernames = ndf_16.Player.unique()

runs=[]
for i in range(len(ndf_16)):
    try:
        r = np.int(ndf_16['Runs'][i])
    except:
        r= np.int(ndf_16.Runs.unique()[0].split("*")[0])
    runs.append(r)
modRun = pd.DataFrame(runs,columns=["modRun"])
modDf  = pd.concat([ndf_16,modRun],axis=1)

def PlayerMaxRun(playername):
    tmpPlayer     = modDf[modDf["Player"]==playername]
    tmpPlayer     = tmpPlayer.reset_index(drop=True)
    maxrun        = np.max(np.array(tmpPlayer["modRun"]))
    totalrun      = sum(np.array(tmpPlayer["modRun"]))
    return (maxrun,totalrun)

tb1=[]
rnn=[]
for i in playernames:
    [mxrn,trn] = PlayerMaxRun(i)
    tb1.append([i,mxrn,trn])
    rnn.append(mxrn)

tb1

dfx = pd.DataFrame(tb1,columns=['player_name','max_run','total_run'])
dfx


I am working on the file "India’s_batting_performance_2016.ipynb".
The first task is:
Find all rows in the year 2016 from new dataframe and remove 'DND' rows from the dataframe which appear in the 'Runs' columns
Can you write Python code for it?

df_16 = df[df["year"]=="2016"]

df_16=df_16.reset_index(drop=True)
df_16.columns

Runs = np.array(df_16["Runs"])

np.squeeze(np.where(Runs=="DNB"))

ndf_16=df_16[0:88]
ndf_16.head(5)


I am working on the file "India’s_batting_performance_2016.ipynb".
The first task is:
Load Dataset
Can you write Python code for it?

IB = pd.read_csv("india-batting.csv")

IB.head(5)

IB.columns


I am working on the file "India’s_batting_performance_2016.ipynb".
The first task is:
Split the year from 'Start Date' columns and create a new column name 'year'
Can you write Python code for it?

year=[]
for i in range(len(IB)):
    x = IB['Start Date'][i].split(" ")[-1]
    year.append(x)

year= pd.DataFrame(year,columns=["year"])
mr = [IB,year]

df=pd.concat(mr,axis=1)
df.head(5)


I am working on the file "Induced-gap-tuning.ipynb".
The first task is:
As you can see $\Delta=68.4$ meV is the parameter we need with the superconductor on the side if we want $\Delta_{ind}$.
Can you write Python code for it?

delta = 68.4
chosen_curve = Delta_ind[nearest(deltas, delta)].select(mu=(4, 6), **{'$\Delta_{ind}$': (0.23, 0.28)})
(chosen_curve * hv.HLine(0.25)).relabel(r'$\Delta={}$ meV, $t_s=7/8t$'.format(delta))


I am working on the file "Induced-gap-tuning.ipynb".
The first task is:
Make sure to add the correct path like:
sys.path.append("/path/where/to/ipynb/runs")

The code currently assumes the scripts live in ~/orbitalfield/.
Can you write Python code for it?

%%px --local
import numpy as np
import sys
import os
sys.path.append(os.path.join(os.path.expanduser('~'), 'orbitalfield'))
from fun import *
from fun2 import *
def evs(lead, p):
    h0 = lead.cell_hamiltonian(args=[p])
    t0 = lead.inter_cell_hopping(args=[p])
    ham = h0 + t0 + t0.conj().T
    ev = np.linalg.eigvalsh(ham)
    return np.abs(ev).min()

import holoviews as hv
from itertools import product
hv.notebook_extension()
%opts Contours (linewidth=1 color='k') [aspect='square']
%output size=100 dpi=100


I am working on the file "Induced-gap-tuning.ipynb".
The first task is:
Run the next cell and use the slider to find the correct $\Delta$.
Can you write Python code for it?

Delta_ind = {delta: hv.Contours((mus, evs), kdims=[dimensions.mu, dimensions.delta_ind])
      for evs, delta in zip(res, deltas)}

hm = hv.HoloMap(Delta_ind, kdims=[dimensions.delta])
hm.select(mu=(0, 20), **{'$\Delta_{ind}$': (0.2, 10)}) * hv.HLine(0.25)


I am working on the file "Induced-gap-tuning.ipynb".
The first task is:
Uncomment the lines for the wire that you want to use.
Can you write Python code for it?

%%px --local
constants.a = 10
p = make_params(t_interface=7/8*constants.t, Delta=68.4, r1=50, r2=70,
                phi=135, orbital=True, A_correction=False) #r2=70
p.V = lambda x, y, z: 2 / 50 * z

# p.angle = 0 # WIRE WITH SC ON TOP
p.angle = 45 # WIRE WITH SC ON SIDE
lead = make_3d_wire_external_sc(a=constants.a, r1=p.r1, r2=p.r2, angle=p.angle, phi=p.phi)


I am working on the file "Induced-gap-tuning.ipynb".
The first task is:
You can specify the intervals of $\mu$ and $\Delta$ that you want to check.
Can you write Python code for it?

mus = np.linspace(0, 20, 100)
deltas = np.linspace(60, 80, 50)
vals = list(product(deltas, mus))
v.scatter('vals', vals)
%px res = [evs(lead, p) for p.Delta, p.mu in vals]
res = np.reshape(v.gather('res', block=True), (len(deltas), len(mus)))


I am working on the file "Infection_vs_Inflammation/Code/01-Process_Data_Josh.ipynb".
The first task is:
T2 relaxation
Can you write Python code for it?

# Make list of all T2.txt files
T2_list = get_ipython().getoutput('ls ../Study_03_CBA/*T2.txt')

# Allocate variables needed for analysis
T2DF=pd.DataFrame()
TR=np.linspace(.012,.012*12,12)

# Fit T2 for all ROIs, slices and mice. construct dataframe
for names in T2_list:
    #Convert txt file to array
    YDataMatrix=txt_2_array(names)
    #Estimate T2
    T2time=fitT2(TR,YDataMatrix)
    #convert to data frame
    df_T2=pd.DataFrame(T2time.T,columns=["Infected","Healthy_Right","Sterile_Inflammation","Healthy_Left"])
    #df_T2=pd.DataFrame(T2time.T,columns=["ROI-1","ROI-2","ROI-3","ROI-4"])
    df_info=name_2_df(names)
    df_final=pd.concat([df_T2,df_info], axis=1)
    T2DF=T2DF.append(df_final,ignore_index=True)

# Plot distribution of estimated T2 for each slice
T2DF[T2DF.Slice==1].iloc[:,:4].plot.density(); title("Slice 01"); xlim((0.025,.15))
T2DF[T2DF.Slice==2].iloc[:,:4].plot.density(); title("Slice 02"); xlim((0.025,.15))
T2DF[T2DF.Slice==3].iloc[:,:4].plot.density(); title("Slice 03"); xlim((0.025,.15))
T2DF[T2DF.Slice==4].iloc[:,:4].plot.density(); title("Slice 04"); xlim((0.025,.15))
T2DF[T2DF.Slice==5].iloc[:,:4].plot.density(); title("Slice 05"); xlim((0.025,.15))

# Make list of all T2.txt files
CEST_list = get_ipython().getoutput('ls ../Study_03_CBA/*CEST.txt')
CEST_list[0]

# Allocate variables needed for analysis
YDataMatrix=txt_2_array(CEST_list)

# Fit T2 for all ROIs, slices and mice. construct dataframe
for names in T2_list:
    #Convert txt file to array
    YDataMatrix=txt_2_array(names)
    #Estimate T2
    T2time=fitT2(TR,YDataMatrix)
    #convert to data frame
    df_T2=pd.DataFrame(T2time.T,columns=["Infected","Healthy_Right","Sterile_Inflammation","Healthy_Left"])
    #df_T2=pd.DataFrame(T2time.T,columns=["ROI-1","ROI-2","ROI-3","ROI-4"])
    df_info=name_2_df(names)
    df_final=pd.concat([df_T2,df_info], axis=1)
    T2DF=T2DF.append(df_final,ignore_index=True)


I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
As we can observe as we eprform the dynamics of the Hopfield model, the total energy of the network is strongly decreasing. This is in accordance with the physical interpretation we had. But even if we always have a decreasing energy (even with wrong recovery), we can observe a typical behavior of the magnetization $m$ (or correlation) as function of $p/N$.
Can you write Python code for it?

%matplotlib inline
import numpy as np
import time
network_size=20
start_time = time.time()
#alpha_list = [0.01,0.05,0.08,0.10,0.12,0.15,0.20,0.30]
alpha_list = np.arange(0.07,0.22,0.010
                      )
m_list = []
hopfield_net = network.HopfieldNetwork(nr_neurons= network_size**2)
factory = pattern_tools.PatternFactory(network_size,network_size)
# create a checkerboard pattern and add it to the pattern list
#checkerboard = factory.create_checkerboard()
#pattern_list = [checkerboard]
pattern_list=[]
pattern_list.extend(factory.create_random_pattern_list(nr_patterns=int(alpha_list[-1]*network_size*network_size), on_probability=0.5))

for alpha in alpha_list:

# create an instance of the class HopfieldNetwork

# let the hopfield network "learn" the patterns. Note: they are not stored
# explicitly but only network weights are updated !
 hopfield_net.store_patterns(pattern_list[0:int(alpha*network_size*network_size)])

# create a noisy version of a pattern and use that to initialize the network
 noisy_init_state=pattern_tools.flip_n(pattern_list[0],nr_of_flips=int(network_size*network_size*0.15)) # 10% of noisy pixels

 hopfield_net.set_state_from_pattern(noisy_init_state)
 states_as_patterns_old =  [noisy_init_state]
 states_as_patterns_new = [pattern_list[0]]
# from this initial state, let the network dynamics evolve.
#hopfield_net.run(nr_steps=50)
 #while pattern_tools.compute_overlap(states_as_patterns_old[0],states_as_patterns_new[0]) < 0.95:
  #print pattern_tools.compute_overlap(states_as_patterns_old[0],states_as_patterns_new[0])
  #states_as_patterns_old = states_as_patterns_new
  #states_new = hopfield_net.run_with_monitoring(nr_steps=1)
  #states_as_patterns_new = factory.reshape_patterns(states_new)
# each network state is a vector. reshape it to the same shape used to create the patterns.
 hopfield_net.run(nr_steps=60)
 states = hopfield_net.run_with_monitoring(nr_steps=1)
 states_as_patterns = factory.reshape_patterns(states)
...


I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
As we have explained with the theory around $\alpha = 0.14$, at temperature $0$ , we should observe a phase transition. And that's exactly what the strong drop in $m$ is telling us. At a precize number of stored patterns the network does not work well anymore.
To conclude this observation we are going to ocmpute the pahse diagram with Glauber dynamics of the Hopfield network. Each point will be the phase transition in the $(\alpha,T)$ space.
Can you write Python code for it?

from wand.image import Image as WImage
img = WImage(filename='Phase_transition_with_size25.pdf')
img