I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Hands_on_Keras_Tutorial.ipynb".
The first task is:
Training
Keras models are trained on Numpy arrays of input data and labels. For training a model, you will typically use the  fit function. Read its documentation here.
Can you write Python code for it?

# For a single-input model with 2 classes (binary classification):

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Generate dummy data
import numpy as np
data = np.random.random((1000, 100))
labels = np.random.randint(2, size=(1000, 1))

# Train the model, iterating on the data in batches of 32 samples
model.fit(data, labels, epochs=10, batch_size=32)

# For a single-input model with 10 classes (categorical classification):

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(10, activation='softmax'))
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Generate dummy data
import numpy as np
data = np.random.random((1000, 100))
labels = np.random.randint(10, size=(1000, 1))

# Convert labels to categorical one-hot encoding
one_hot_labels = keras.utils.to_categorical(labels, num_classes=10)

# Train the model, iterating on the data in batches of 32 samples
model.fit(data, one_hot_labels, epochs=10, batch_size=32)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Construct model
Can you write Python code for it?

pred = multilayer_perceptron(x, weights, biases)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Cost and Optimization Functions
We'll use Tensorflow's built-in functions for this part (check out the documentation for a lot more options and discussion on this):
Can you write Python code for it?

# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = pred,labels =  y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Data Format
The data is stored in a vector format, although the original data was a 2-dimensional matirx with values representing how much pigment was at a certain location. Let's explore this:
Can you write Python code for it?

type(mnist)

type(mnist.train.images)

#mnist.train.images[0]
mnist.train.images[2].shape


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Get the Data
We will be using the famous MNIST data set of handwritten digits.
The images which we will be working with are black and white images of size 28 x 28 pixels, or 784 pixels total. Our features will be the pixel values for each pixel. Either the pixel is "white" (blank with a 0), or there is some pixel value.
We will try to correctly predict what number is written down based solely on the image data in the form of an array. This type of problem (Image Recognition) is a great use case for Deep Learning Methods!
This data is to Deep Learning what the iris data set is to typical machine learning algorithms.
Let's get the data:
Can you write Python code for it?

# Import MINST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Initialization of Variables
Now initialize all those tf.Variable objects we created earlier. This will be the first thing we run when training our model:
Can you write Python code for it?

# Initializing the variables
init = tf.initialize_all_variables()


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Model Evaluations
Tensorflow comes with some built-in functions to help evaluate our model, including tf.equal and tf.cast with tf.reduce_mean.
tf.equal()
This is essentially just a check of predictions == y_test. In our case since we know the format of the labels is a 1 in an array of zeroes, we can compare argmax() location of that 1. Remember that y here is still that placeholder we created at the very beginning, we will perform a series of operations to get a Tensor that we can eventually fill in the test data for with an evaluation method. What we are currently running will still be empty of test data:
Can you write Python code for it?

# Test model
correct_predictions = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))

print(correct_predictions[0])


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Network Parameters
Here we have parameters which will directly define our Neural Network, these would be adjusted depending on what your data looked like and what kind of a net you would want to build. Basically just some numbers we will eventually use to define some variables later on in our model:
Can you write Python code for it?

# Network Parameters
n_hidden_1 = 256 # 1st layer number of features
n_hidden_2 = 256 # 2nd layer number of features
n_input = 784 # MNIST data input (img shape: 28*28)
n_classes = 10 # MNIST total classes (0-9 digits)
n_samples = mnist.train.num_examples

###  TensorFlow Graph Input

x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Running the Session
Now it is time to run our session! Pay attention to how we have two loops, the outer loop which runs the epochs, and the inner loop which runs the batches for each epoch of training. Let's breakdown each step!
Can you write Python code for it?

# Launch the session
sess = tf.InteractiveSession()

# Intialize all the variables
sess.run(init)

# Training Epochs
# Essentially the max amount of loops possible before we stop
# May stop earlier if cost/loss limit was set
for epoch in range(training_epochs):

    # Start with cost = 0.0
    avg_cost = 0.0

    # Convert total number of batches to integer
    total_batch = int(n_samples/batch_size)

    # Loop over all batches
    for i in range(total_batch):

        # Grab the next batch of training data and labels
        batch_x, batch_y = mnist.train.next_batch(batch_size)

        # Feed dictionary for optimization and loss value
        # Returns a tuple, but we only need 'c' the cost
        # So we set an underscore as a "throwaway"
        _, c = sess.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y})

        # Compute average loss
        avg_cost += c / total_batch

    print("Epoch: {} cost={:.4f}".format(epoch+1,avg_cost))

print("Model has completed {} Epochs of Training".format(training_epochs))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Training the Model
Can you write Python code for it?

Xsamp,ysamp = mnist.train.next_batch(1)

# Remember indexing starts at zero!
print(ysamp)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/TF on MNIST with Multilayer Perception.ipynb".
The first task is:
Weights and Bias
In order for our tensorflow model to work we need to create two dictionaries containing our weight and bias objects for the model. We can use the tf.variable object type. This is different from a constant because TensorFlow's Graph Object becomes aware of the states of all the variables. A Variable is a modifiable tensor that lives in TensorFlow's graph of interacting operations. It can be used and even modified by the computation. We will generally have the model parameters be Variables. From the documentation string:
A variable maintains state in the graph across calls to `run()`. You add a variable to the graph by constructing an instance of the class `Variable`.

The `Variable()` constructor requires an initial value for the variable, which can be a `Tensor` of any type and shape. The initial value defines the type and shape of the variable. After construction, the type and shape of the variable are fixed. The value can be changed using one of the assign methods.

We'll use tf's built-in random_normal method to create the random values for our weights and biases (you could also just pass ones as the initial biases).
Can you write Python code for it?

weights = {
    'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_hidden_2, n_classes]))
}

biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1])),
    'b2': tf.Variable(tf.random_normal([n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}


I am working on the file "NEU/Singh_Palod_DL/Autoencoders/Autoencoder for Text in TensorFlow.ipynb".
The first task is:
Since our compressed data is in probabilities, we'll convert  to whole nums to look up words
Can you write Python code for it?

test_data_compressed[test_data_compressed>0] = 1

test_data_compressed

test_data


I am working on the file "NEU/Singh_Palod_DL/Autoencoders/Autoencoder for Text in TensorFlow.ipynb".
The first task is:
Tadaa!!! And here's our prediction
This show's how well our compression is able to recover data

Remember that Autoencoders are lossy compression which means you will never be able to full reconstruct that data
Can you write Python code for it?

sent = np.ndarray.tolist(test_data_compressed)[0]
print(' '.join([reverse_vocabulary[i] if sent[i] == 1. else "" for i in range(len(sent))]))


I am working on the file "NEU/Singh_Palod_DL/Generative Adversarial Networks/GANS Mode Collapse.ipynb".
The first task is:
After checking images in out dir we see that the GAN mode collapsed
Fix for that is to let discriminator see ground truth in mini batches
 Here's a YouTube of I made from all 1000 images at 100ms delay
Can you write Python code for it?

from IPython.display import HTML

# Youtube
HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/ktxhiKhWoEE?rel=0&controls=0&showinfo=0" frameborder="0" allowfullscreen></iframe>')


I am working on the file "NEU/Singh_Palod_DL/Generative Adversarial Networks/GANS for Text.ipynb".
The first task is:
Check if the generator is saved properly
Can you write Python code for it?

saver.restore(sess,  './tf_model/generator-1000000')

samples = sess.run(G_sample, feed_dict={Z: sample_Z(1, Z_dim)})
for sample in samples:
    print(sample)
    sample[sample == 1.] = 1
    sample[sample < 1.] = 0
    print(np.where(sample == 0)[0])
    print(decode_sentence(sample))


I am working on the file "NEU/Singh_Palod_DL/Generative Adversarial Networks/GANS for Text.ipynb".
The first task is:
Missed out on 5 words
Can you write Python code for it?


I am working on the file "NEU/Singh_Palod_DL/Generative Adversarial Networks/GANS for Text.ipynb".
The first task is:
Reasons why sentences seem nonsensical:

We are representing our words in discreet 1 or 0, hint: Use pretrained word embeddings which we can do via our Word2Vec notebook
Notice how we were passing in the sentences data without sequence so A B C would be represented similiar to C A B
We are storing any information over time in our network (hint: LSTM) which we should do if we are to generate sequential data

Training word2vec using skipgram on twitter dataset
<img src="https://i.imgur.com/5tOMur7.png">
Can you write Python code for it?

with codecs.open("../data/sentiment.vec", 'r', encoding='utf-8') as f:
    next(f)
    word_vectors_twitter = f.read().split('\n')

word_vectors_twitter[0]

word_vectors = {}
for lin in word_vectors_twitter:
    line = lin.split()
#     print(line)
    try:
        word_vectors[line[0]] = np.array(line[1:])
    except:
        print(line, lin)

len(list(word_vectors.keys()))


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization Diff weights (1).ipynb".
The first task is:
Visualizing the fifth convnet/decoded/output layer_5 with sample test image
Can you write Python code for it?

i = 3
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout5_f(x)),0),0)
check.shape

#Plotting conv_5
for i in range(4):
    #i = 3
    x = x_test[i:i+1]
    check = np.squeeze(np.squeeze(np.array(convout1_f(x)),0),0)

    temp = x[0,:,:,:]
    fig, axes = plt.subplots(1, 1, figsize=(3, 3))
    plt.imshow(temp)
    plt.show()

    k = 0
    while k < check.shape[2]:
        #plt.figure()
        #plt.subplot(231 + i)
        fig, axes = plt.subplots(2, 4, figsize=(5, 5))
        for i in range(2):
            for j in range(4):
                axes[i,j].imshow(check[:,:,k])
                k += 1
        #axes[0, 0].imshow(R, cmap='jet')
        #plt.imshow(check[:,:,i])

    plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization Diff weights (1).ipynb".
The first task is:
Visualizing the first convnet/output layer_1 with sample first test image
Can you write Python code for it?

np.squeeze(np.squeeze(np.array(convout1_f(x)),0),0).shape

#Plotting conv_1
for i in range(4):
    #i = 3
    x = x_test[i:i+1]
    check = np.squeeze(np.squeeze(np.array(convout1_f(x)),0),0)

    temp = x[0,:,:,:]
    fig, axes = plt.subplots(1, 1, figsize=(3, 3))
    plt.imshow(temp)
    plt.show()

    k = 0
    while k < check.shape[2]:
        #plt.figure()
        #plt.subplot(231 + i)
        fig, axes = plt.subplots(4, 4, figsize=(5, 5))
        for i in range(4):
            for j in range(4):
                axes[i,j].imshow(check[:,:,k], cmap = 'gray')
                k += 1
        #axes[0, 0].imshow(R, cmap='jet')
        #plt.imshow(check[:,:,i])

    plt.show()

check.shape


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization Diff weights (1).ipynb".
The first task is:
Visualizing the sixth convnet/decoded/output layer_6 with sample test image
Can you write Python code for it?

i = 3
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout6_f(x)),0),0)
check.shape

#Plotting conv_6
for i in range(4):
    #i = 3
    x = x_test[i:i+1]
    check = np.squeeze(np.squeeze(np.array(convout1_f(x)),0),0)

    temp = x[0,:,:,:]
    fig, axes = plt.subplots(1, 1, figsize=(3, 3))
    plt.imshow(temp)
    plt.show()

    k = 0
    while k < check.shape[2]:
        #plt.figure()
        #plt.subplot(231 + i)
        fig, axes = plt.subplots(4, 4, figsize=(5, 5))
        for i in range(4):
            for j in range(4):
                axes[i,j].imshow(check[:,:,k])
                k += 1
        #axes[0, 0].imshow(R, cmap='jet')
        #plt.imshow(check[:,:,i])

    plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization.ipynb".
The first task is:
Plotting the third convnet/output layer_3 with sample test image
Can you write Python code for it?

i = 1
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout3_f(x)),0),0)
check.shape

#Plotting conv_3

temp = x[0,:,:,:]
fig, axes = plt.subplots(1, 1, figsize=(3, 3))
plt.imshow(temp)
plt.show()

k = 0
while k < check.shape[2]:
    #plt.figure()
    #plt.subplot(231 + i)
    fig, axes = plt.subplots(2, 4, figsize=(5, 5))
    for i in range(2):
        for j in range(4):
            axes[i,j].imshow(check[:,:,k])
            k += 1
    #axes[0, 0].imshow(R, cmap='jet')
    #plt.imshow(check[:,:,i])

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization.ipynb".
The first task is:
Plotting the weights from the first layer
Can you write Python code for it?

import matplotlib.pyplot as plt

n = 10

for i in range(n):
    fig = plt.figure(figsize=(4, 10))
    conv_1 = np.asarray(autoencoder.layers[1].get_weights())[0][:,:,0,i]
    ax = fig.add_subplot(111)
    plt.imshow(conv_1.transpose())
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    plt.show()

from keras import backend as K

# K.learning_phase() is a flag that indicates if the network is in training or
# predict phase. It allow layer (e.g. Dropout) to only be applied during training
inputs = [K.learning_phase()] + autoencoder.inputs

_layer1_f = K.function(inputs, [x2])
def convout1_f(X):
    # The [0] is to disable the training phase flag
    return _layer1_f([0] + [X])

_layer2_f = K.function(inputs, [x4])
def convout2_f(X):
    # The [0] is to disable the training phase flag
    return _layer2_f([0] + [X])

_layer3_f = K.function(inputs, [encoded])
def convout3_f(X):
    # The [0] is to disable the training phase flag
    return _layer3_f([0] + [X])

_up_layer1_f = K.function(inputs, [x6])
def convout4_f(X):
    # The [0] is to disable the training phase flag
    return _up_layer1_f([0] + [X])

_up_layer2_f = K.function(inputs, [x8])
def convout5_f(X):
    # The [0] is to disable the training phase flag
    return _up_layer2_f([0] + [X])

_up_layer3_f = K.function(inputs, [x10])
def convout6_f(X):
    # The [0] is to disable the training phase flag
    return _up_layer3_f([0] + [X])

_up_layer4_f = K.function(inputs, [decoded])
def convout7_f(X):
    # The [0] is to disable the training phase flag
    return _up_layer4_f([0] + [X])

x2

i = 1
x = x_test[i:i+1]


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization.ipynb".
The first task is:
Visualizing the fifth convnet/decoded/output layer_5 with sample test image
Can you write Python code for it?

i = 1
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout5_f(x)),0),0)
check.shape

#Plotting conv_5

temp = x[0,:,:,:]
fig, axes = plt.subplots(1, 1, figsize=(3, 3))
plt.imshow(temp)
plt.show()

k = 0
while k < check.shape[2]:
    #plt.figure()
    #plt.subplot(231 + i)
    fig, axes = plt.subplots(2, 4, figsize=(5, 5))
    for i in range(2):
        for j in range(4):
            axes[i,j].imshow(check[:,:,k])
            k += 1
    #axes[0, 0].imshow(R, cmap='jet')
    #plt.imshow(check[:,:,i])

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization.ipynb".
The first task is:
Visualizing the final decoded/output layer with sample test image
Can you write Python code for it?

i = 1
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout7_f(x)),0),0)
check.shape

#Plotting final layer

temp = x[0,:,:,:]
#plt.imshow(temp)
#fig = plt.figure()
#fig.set_figheight(3)
#fig.set_figwidth(3)
#plt.gray()
#ax.get_xaxis().set_visible(False)
#ax.get_yaxis().set_visible(False)
fig, axes = plt.subplots(1, 1, figsize=(3, 3))
plt.imshow(temp)
plt.show()

k = 2
while k < check.shape[2]:
    #plt.figure()
    #plt.subplot(231 + i)
    fig, axes = plt.subplots(1, 1, figsize=(3, 3))
    plt.imshow(check[:,:,:])
    k += 1
    #axes[0, 0].imshow(R, cmap='jet')
    #plt.imshow(check[:,:,i])

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization.ipynb".
The first task is:
Visualizing the sixth convnet/decoded/output layer_6 with sample test image
Can you write Python code for it?

i = 1
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout6_f(x)),0),0)
check.shape

#Plotting conv_6

temp = x[0,:,:,:]
fig, axes = plt.subplots(1, 1, figsize=(3, 3))
plt.imshow(temp)
plt.show()

k = 0
while k < check.shape[2]:
    #plt.figure()
    #plt.subplot(231 + i)
    fig, axes = plt.subplots(4, 4, figsize=(5, 5))
    for i in range(4):
        for j in range(4):
            axes[i,j].imshow(check[:,:,k])
            k += 1
    #axes[0, 0].imshow(R, cmap='jet')
    #plt.imshow(check[:,:,i])

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
As you would expect, there are some variables that correlate, such as density and residual sugar. Also volatile acidity and type are more closely connected than you originally could have guessed by looking at the two data sets separately, and it was kind of to be expected that free sulfur dioxide and total sulfur dioxide were going to correlate.
Data Preprocessing
Create a column to distinguish between red and white by giving red value 1 and white value 0. Why not simply label them 'red' and 'white'? This is because neural networks only works with numerical data and not labels, so it outputs probabilities which we have to later compute to one of the labels. (it's not that difficult!)
Can you write Python code for it?

# Add `type` column to `red` with value 1
red['type'] = 1

# Add `type` column to `white` with value 0
white['type'] = 0

# Row bind white to red
wines = red.append(white, ignore_index=True)

wines.tail()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
Correlation Matrix
Since it can be somewhat difficult to interpret graphs, it’s also a good idea to plot a correlation matrix. This will give insights more quickly about which variables correlate:
Can you write Python code for it?

import seaborn as sns
corr = red.append(white, ignore_index=True).corr()
fig, ax = plt.subplots(1, 1, figsize=(10, 8))
sns.heatmap(corr,
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values,cmap="autumn",linewidths=.2)
plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
Here’s a short description of each variable:
1) Fixed acidity: acids are major wine properties and contribute greatly to the wine’s taste. Usually, the total acidity is divided into two groups: the volatile acids and the nonvolatile or fixed acids. Among the fixed acids that you can find in wines are the following: tartaric, malic, citric, and succinic.
2) Volatile acidity: the volatile acidity is basically the process of wine turning into vinegar. In the U.S, the legal limits of Volatile Acidity are 1.2 g/L for red table wine and 1.1 g/L for white table wine.
3) Citric acid is one of the fixed acids that you’ll find in wines. It’s expressed in g/dm3dm3 in the two data sets.
4) Residual sugar typically refers to the sugar remaining after fermentation stops, or is stopped. It’s expressed in g/dm3dm3 in the red and white data.
5) Chlorides can be a major contributor to saltiness in wine. Here, you’ll see that it’s expressed in.
6) Free sulfur dioxide: the part of the sulphur dioxide that is added to a wine and that is lost into it is said to be bound, while the active part is said to be free. Winemaker will always try to get the highest proportion of free sulphur to bind. This variables is expressed in mg/dm3dm3 in the data.
7) Total sulfur dioxide is the sum of the bound and the free sulfur dioxide (SO2). Here, it’s expressed in mg/dm3dm3. There are legal limits for sulfur levels in wines: in the EU, red wines can only have 160mg/L, while white and rose wines can have about 210mg/L. Sweet wines are allowed to have 400mg/L. For the US, the legal limits are set at 350mg/L and for Australia, this is 250mg/L.
8) Density is generally used as a measure of the conversion of sugar to alcohol. Here, it’s expressed in g/cm3cm3.
9) Sulphates are to wine as gluten is to food. You might already know sulphites from the headaches that they can cause. They are a regular part of the winemaking around the world and are considered necessary. In this case, they are expressed in g(potassiumsulphatepotassiumsulphate)/dm3dm3.
10)...
Can you write Python code for it?

# Print info on white wine
white.info()

red.info()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
Now that we have our data preprocessed, we can move on to the real work: building our own neural network to classify wines.
Model Data
A quick way to get started is to use the Keras Sequential model: it’s a linear stack of layers. You can easily create the model by passing a list of layer instances to the constructor, which you set up by running: model = Sequential()
Can you write Python code for it?

# Import `Sequential` from `keras.models`
from keras.models import Sequential

# Import `Dense` from `keras.layers`
from keras.layers import Dense

# Initialize the constructor
model = Sequential()

# Add an input layer
model.add(Dense(12, activation='relu', input_shape=(11,)))

# Add one hidden layer
model.add(Dense(8, activation='relu'))

# Add an output layer
model.add(Dense(1, activation='sigmoid'))

# Model output shape
model.output_shape

# Model summary
model.summary()

# Model config
model.get_config()

# List all weight tensors
model.get_weights()

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X_train, y_train,epochs=10, batch_size=1, verbose=1)


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
One would notice that most of the wines made have a 9~10% alcohol present in them. Ofcourse some would notice that 10-12% is also frequent though not as much as 9%. Moreover the y-axis scales are different because of the disproportionate dataset (unbalanced) observations between red and white.
Sulphates
Next, one thing that interests me is the relation between the sulphates and the quality of the wine. As you may know, sulphates can cause people to have headaches and I’m wondering if this infuences the quality of the wine. What’s more, I often hear that women especially don’t want to drink wine exactly because it causes headaches. Maybe this affects the ratings for the red wine?
Can you write Python code for it?

fig, ax = plt.subplots(1, 2, figsize=(15, 5))

ax[0].scatter(red['quality'], red["sulphates"], color="red", label="Red wine")
ax[1].scatter(white['quality'], white['sulphates'], color="white", edgecolors="black", lw=0.5, label="White wine")

ax[0].set_xlabel("Quality")
ax[1].set_xlabel("Quality")
ax[0].set_ylabel("Sulphate")
ax[1].set_ylabel("Sulphate")
ax[0].set_xlim([0,10])
ax[1].set_xlim([0,10])
ax[0].set_ylim([0,2.5])
ax[1].set_ylim([0,2.5])
fig.subplots_adjust(wspace=0.5)
ax[0].legend(loc='best')
ax[1].legend(loc='best')
fig.suptitle("Wine Quality v/s Sulphate")

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
Our red wine dataframe has relatively lower (1599) observations than that of wine (4898). All the values are float except for the quality variable which is ratings given by wine experts on a scale of 1-10.
Can you write Python code for it?

# Print info on red wine
red.head()

#Print a random sample of 5 obs in white wine dataset
white.sample(5)

# Double check for null values in `red`
pd.isnull(red).sum()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
Predict Values
Let’s put your model to use! You can make predictions for the labels of the test set with it. Just use predict() and pass the test set to it to predict the labels for the data. In this case, the result is stored in y_pred:
Can you write Python code for it?

y_pred = model.predict(X_test)

# round predictions
y_pred = [round(x[0]) for x in y_pred]

y_pred[:5]

y_test[:5]

score = model.evaluate(X_test, y_test,verbose=1)

print(score)

# evaluate the model
scores = model.evaluate(X_test, y_test)
print("\n%s: %.2f%%" % (model.metrics_names[0], scores[0]*100))
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

# Import the modules from `sklearn.metrics`
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, cohen_kappa_score

# Confusion matrix
confusion_matrix(y_test, y_pred)

# Precision
precision_score(y_test, y_pred)

# Recall
recall_score(y_test, y_pred)

# F1 score
f1_score(y_test,y_pred)

# Cohen's kappa
cohen_kappa_score(y_test, y_pred)


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
So, from the graphs above we can see that for the same values (most of it) the ratings stay constant which doesn't necessarily affect the quality as i guessed earlier. Although, we can see that the red wine has higher amount of sulphates in them which explains why drinking red wine causes headache and women prefer white over red.
Acidity
Apart from the sulphates, acidity is one of the important wine characteristics that is necessary to achieve quality wines. Great wines often balance out acidity, tannin, alcohol and sweetness. Some more research taught me that in quantities of 0.2 to 0.4 g/L, volatile acidity doesn’t affect a wine’s quality. At higher levels, however, volatile acidity can give wine a sharp, vinegary tactile sensation. Extreme volatile acidity signifies a seriously flawed wine.
Can you write Python code for it?

import numpy as np

np.random.seed(570)

redlabels = np.unique(red['quality'])
whitelabels = np.unique(white['quality'])

fig, ax = plt.subplots(1, 2, figsize=(10, 8))
redcolors = np.random.rand(6,4)
whitecolors = np.append(redcolors, np.random.rand(1,4), axis=0)

for i in range(len(redcolors)):
    redy = red['alcohol'][red.quality == redlabels[i]]
    redx = red['volatile acidity'][red.quality == redlabels[i]]
    ax[0].scatter(redx, redy, c=redcolors[i])
for i in range(len(whitecolors)):
    whitey = white['alcohol'][white.quality == whitelabels[i]]
    whitex = white['volatile acidity'][white.quality == whitelabels[i]]
    ax[1].scatter(whitex, whitey, c=whitecolors[i])

ax[0].set_title("Red Wine")
ax[1].set_title("White Wine")
ax[0].set_xlim([0,1.5])
ax[1].set_xlim([0,1.5])
ax[0].set_ylim([6,15.5])
ax[1].set_ylim([6,15.5])
ax[0].set_xlabel("Volatile Acidity")
ax[0].set_ylabel("Alcohol")
ax[1].set_xlabel("Volatile Acidity")
ax[1].set_ylabel("Alcohol")
ax[0].legend(redlabels, loc='best', bbox_to_anchor=(1.3, 1))
ax[1].legend(whitelabels, loc='best', bbox_to_anchor=(1.3, 1))
fig.suptitle("Alcohol - Volatile Acidity")
fig.subplots_adjust(top=.85, wspace=0.7)

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/Keras Tutorial.ipynb".
The first task is:
Visualizing The Data
One way to do this is by looking at the distribution of some of the dataset’s variables and make scatter plots to see possible correlations
Can you write Python code for it?

import matplotlib.pyplot as plt

#Split with same y-axis
fig, ax = plt.subplots(1, 2,figsize=(10, 8))

ax[0].hist(red.alcohol, 15, facecolor='red',  ec="black", lw=0.5, alpha=0.5)
ax[1].hist(white.alcohol, 15, facecolor='white', ec="black", lw=0.5, alpha=0.5)

fig.subplots_adjust(left=0, right=1, bottom=0, top=0.8, hspace=0.05, wspace=0.2)

ax[0].set_title("Red Wine")
ax[1].set_title("White Wine")
#ax[0].set_ylim([0, 800])
ax[0].set_xlabel("Alcohol (% Vol)")
ax[0].set_ylabel("Frequency")
ax[1].set_xlabel("Alcohol (% Vol)")
ax[1].set_ylabel("Frequency")
#ax[1].set_ylim([0, 800])

fig.suptitle("Distribution of Alcohol in % Vol")

plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/t-SNE.ipynb".
The first task is:
Each row represents a different person and each column is on of many physical measurments lke the position of their arm, or forearm and each person gets one of 5 labels (classes) like sitting, standing, jumping, running and jogging.
Can you write Python code for it?

#List all fators from our response variable
dataframe_all.classe.unique()


I am working on the file "NEU/Tejas_Bawaskar _DL/t-SNE.ipynb".
The first task is:
Scatter plot the sample points among 5 classes
Can you write Python code for it?

markers=('s', 'd', 'o', '^', 'v')
color_map = {0:'red', 1:'blue', 2:'lightgreen', 3:'purple', 4:'cyan'}

plt.figure()
plt.figure(figsize=(10,10))
for idx, cl in enumerate(np.unique(y_test)):
    plt.scatter(x=x_test_2d[y_test==cl,0], y=x_test_2d[y_test==cl,1], c=color_map[idx], marker=markers[idx], label=cl)
plt.xlabel('X in t-SNE')
plt.ylabel('Y in t-SNE')
plt.legend(loc='upper right')
plt.title('t-SNE visualization of test data')
plt.show()


I am working on the file "NEU/Tejas_Bawaskar _DL/t-SNE.ipynb".
The first task is:
Step 4: get class labels y and then encode it into number
Can you write Python code for it?

# get class label data
y = dataframe_all.iloc[:,-1].values
# encode the class label
class_labels = np.unique(y)
label_encoder = LabelEncoder()
y = label_encoder.fit_transform(y)


I am working on the file "NEU/Tejas_Bawaskar _DL/t-SNE.ipynb".
The first task is:
Step 5: split the data into training set and test set
Can you write Python code for it?

test_percentage = 0.3
x_train, x_test, y_train, y_test = train_test_split(x_std, y, test_size = test_percentage, random_state = 0)


I am working on the file "NEU/Vikram_Balakrishnan_DL/Phase_0/1. variables.ipynb".
The first task is:
Running the graph in a tf session
Can you write Python code for it?

with tf.Session() as session:
    session.run(model)
    print(session.run(y))


I am working on the file "NEU/Vikram_Balakrishnan_DL/Phase_0/1. variables.ipynb".
The first task is:
Section 2 - moving average
Can you write Python code for it?

import tensorflow as tf

x = tf.Variable(0, name='x')

model = tf.global_variables_initializer()

with tf.Session() as session:
    session.run(model)
    for i in range(5):
        x = x + 1
        avg = x/(i+1)
        print('x = ',session.run(x))
        print('moving avg = ',session.run(avg))


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
Download a collection of historical data
This time we'll make a query based on a longitude, latitude point and using a time range.
Can you write Python code for it?

# Our specified time
#dt = datetime(2012, 10, 29, 15) # Superstorm Sandy
#dt = datetime(2016, 6, 18, 1)
dt = datetime(2016, 6, 8, 18)
query = rs.query()
query.lonlat_point(-73.687, 41.175).time_range(dt, dt + timedelta(hours=1))


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
Finally, we plot them up using matplotlib and cartopy. We create a helper function for making a map to keep things simpler later.
Can you write Python code for it?

import matplotlib.pyplot as plt
import cartopy

def new_map(fig, lon, lat):
    # Create projection centered on the radar. This allows us to use x
    # and y relative to the radar.
    proj = cartopy.crs.LambertConformal(central_longitude=lon, central_latitude=lat)

    # New axes with the specified projection
    ax = fig.add_subplot(1, 1, 1, projection=proj)

    # Add coastlines
    ax.coastlines('50m', 'black', linewidth=2, zorder=2)

    # Grab state borders
    state_borders = cartopy.feature.NaturalEarthFeature(
        category='cultural', name='admin_1_states_provinces_lines',
        scale='50m', facecolor='none')
    ax.add_feature(state_borders, edgecolor='black', linewidth=1, zorder=3)

    return ax


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
Grab the first dataset so that we can get the longitude and latitude of the station and make a map for plotting. We'll go ahead and specify some longitude and latitude bounds for the map.
Can you write Python code for it?

ds = list(cat.datasets.values())[0]
data = Dataset(ds.access_urls['CdmRemote'])
# Pull out the data of interest
sweep = 0
rng = data.variables['distanceR_HI'][:]
az = data.variables['azimuthR_HI'][sweep]
ref_var = data.variables['Reflectivity_HI']

# Convert data to float and coordinates to Cartesian
ref = raw_to_masked_float(ref_var, ref_var[sweep])
x, y = polar_to_cartesian(az, rng)


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
Make the request, which returns an instance of TDSCatalog; this handles parsing the returned XML information.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
The CDMRemote reader provides an interface that is almost identical to the usual python NetCDF interface. We pull out the variables we need for azimuth and range, as well as the data itself.
Can you write Python code for it?

sweep = 0
ref_var = data.variables['Reflectivity_HI']
ref_data = ref_var[sweep]
rng = data.variables['distanceR_HI'][:]
az = data.variables['azimuthR_HI'][sweep]


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
The specified longitude, latitude are in NY and the TDS helpfully finds the closest station to that point.  We can see that for this time range we obtained multiple datasets.
Can you write Python code for it?

cat = rs.get_catalog(query)
cat.datasets


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
Use the function to make a new map and plot a colormapped view of the data
Can you write Python code for it?

fig = plt.figure(figsize=(10, 10))
ax = new_map(fig, data.StationLongitude, data.StationLatitude)

# Set limits in lat/lon space
ax.set_extent([-77, -70, 38, 43])

# Add ocean and land background
ocean = cartopy.feature.NaturalEarthFeature('physical', 'ocean', scale='50m',
                                            edgecolor='face',
                                            facecolor=cartopy.feature.COLORS['water'])
land = cartopy.feature.NaturalEarthFeature('physical', 'land', scale='50m',
                                           edgecolor='face',
                                           facecolor=cartopy.feature.COLORS['land'])

ax.add_feature(ocean, zorder=-1)
ax.add_feature(land, zorder=-1)
ax.pcolormesh(x, y, ref, cmap=ref_cmap, norm=ref_norm, zorder=0);

meshes = []
for item in sorted(cat.datasets.items()):
    # After looping over the list of sorted datasets, pull the actual Dataset object out
    # of our list of items and access over CDMRemote
    ds = item[1]
    data = Dataset(ds.access_urls['CdmRemote'])

    # Pull out the data of interest
    sweep = 0
    rng = data.variables['distanceR_HI'][:]
    az = data.variables['azimuthR_HI'][sweep]
    ref_var = data.variables['Reflectivity_HI']

    # Convert data to float and coordinates to Cartesian
    ref = raw_to_masked_float(ref_var, ref_var[sweep])
    x, y = polar_to_cartesian(az, rng)

    # Plot the data and the timestamp
    mesh = ax.pcolormesh(x, y, ref, cmap=ref_cmap, norm=ref_norm, zorder=0)
    text = ax.text(0.65, 0.03, data.time_coverage_start, transform=ax.transAxes,
                   fontdict={'size':16})

    # Collect the things we've plotted so we can animate
    meshes.append((mesh, text))

# Set up matplotlib to do the conversion to HTML5 video
import matplotlib
matplotlib.rcParams['animation.html'] = 'html5'

# Create an animation
from matplotlib.animation import ArtistAnimation
ArtistAnimation(fig, meshes)


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
We can look at the datasets on the catalog to see what data we found by the query. We find one volume in the return, since we asked for the volume nearest to a single time.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
We can use the RadarServer instance to check our query, to make sure we have required parameters and that we have chosen valid station(s) and variable(s)
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
We define some helper functions to make working with the data easier. One takes the raw data and converts it to floating point values with the missing data points appropriately marked. The other helps with converting the polar coordinates (azimuth and range) to Cartesian (x and y).
Can you write Python code for it?

import numpy as np
def raw_to_masked_float(var, data):
    # Values come back signed. If the _Unsigned attribute is set, we need to convert
    # from the range [-127, 128] to [0, 255].
    if var._Unsigned:
        data = data & 255

    # Mask missing points
    data = np.ma.array(data, mask=data==0)

    # Convert to float using the scale and offset
    return data * var.scale_factor + var.add_offset

def polar_to_cartesian(az, rng):
    az_rad = np.deg2rad(az)[:, None]
    x = rng * np.sin(az_rad)
    y = rng * np.cos(az_rad)
    return x, y


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-Copy1-checkpoint.ipynb".
The first task is:
We'll use the CDMRemote reader in Siphon and pass it the appropriate access URL.
Can you write Python code for it?

from siphon.cdmr import Dataset
data = Dataset(ds.access_urls['CdmRemote'])


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
Download a collection of historical data
This time we'll make a query based on a longitude, latitude point and using a time range.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
Finally, we plot them up using matplotlib and cartopy. We create a helper function for making a map to keep things simpler later.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
Grab the first dataset so that we can get the longitude and latitude of the station and make a map for plotting. We'll go ahead and specify some longitude and latitude bounds for the map.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
Make the request, which returns an instance of TDSCatalog; this handles parsing the returned XML information.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
MetPy is a Python package for meteorology (Documentation: http://metpy.readthedocs.org and GitHub: http://github.com/MetPy/MetPy). We import MetPy and use it to get the colortable and value mapping information for the NWS Reflectivity data.
Can you write Python code for it?

from metpy.plots import ctables  # For NWS colortable
ref_norm, ref_cmap = ctables.registry.get_with_steps('NWSReflectivity', 5, 5)


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
The CDMRemote reader provides an interface that is almost identical to the usual python NetCDF interface. We pull out the variables we need for azimuth and range, as well as the data itself.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
The specified longitude, latitude are in NY and the TDS helpfully finds the closest station to that point.  We can see that for this time range we obtained multiple datasets.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
Then convert the raw data to floating point values and the polar coordinates to Cartesian.
Can you write Python code for it?

ref = raw_to_masked_float(ref_var, ref_data)
x, y = polar_to_cartesian(az, rng)


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
We can look at the datasets on the catalog to see what data we found by the query. We find one volume in the return, since we asked for the volume nearest to a single time.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
We can pull that dataset out of the dictionary and look at the available access URLs. We see URLs for OPeNDAP, CDMRemote, and HTTPServer (direct download).
Can you write Python code for it?

ds = list(catalog.datasets.values())[0]
ds.access_urls


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
We can use the RadarServer instance to check our query, to make sure we have required parameters and that we have chosen valid station(s) and variable(s)
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_NEXRAD-checkpoint.ipynb".
The first task is:
We define some helper functions to make working with the data easier. One takes the raw data and converts it to floating point values with the missing data points appropriately marked. The other helps with converting the polar coordinates (azimuth and range) to Cartesian (x and y).
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
Download a collection of historical data
This time we'll make a query based on a longitude, latitude point and using a time range.
Can you write Python code for it?

query = rs.query()
#dt = datetime(2012, 10, 29, 15) # Our specified time
dt = datetime(2016, 6, 8, 18) # Our specified time
query.lonlat_point(-73.687, 41.175).time_range(dt, dt + timedelta(hours=1))


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
Finally, we plot them up using matplotlib and cartopy. We create a helper function for making a map to keep things simpler later.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
First we'll create an instance of RadarServer to point to the appropriate radar server access URL.
Can you write Python code for it?

# The S3 URL did not work for me, despite .edu domain
#url = 'http://thredds-aws.unidata.ucar.edu/thredds/radarServer/nexrad/level2/S3/'

#Trying motherlode URL
url = 'http://thredds.ucar.edu/thredds/radarServer/nexrad/level2/IDD/'
from siphon.radarserver import RadarServer
rs = RadarServer(url)


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
Next, we'll create a new query object to help request the data. Using the chaining methods, let's ask for the latest data at the radar KLVX (Louisville, KY). We see that when the query is represented as a string, it shows the encoded URL.
Can you write Python code for it?

from datetime import datetime, timedelta
query = rs.query()
query.stations('KLVX').time(datetime.utcnow())


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
Now we can loop over the collection of returned datasets and plot them. As we plot, we collect the returned plot objects so that we can use them to make an animated plot. We also add a timestamp for each plot.
Can you write Python code for it?

meshes = []
for item in sorted(cat.datasets.items()):
    # After looping over the list of sorted datasets, pull the actual Dataset object out
    # of our list of items and access over CDMRemote
    ds = item[1]
    data = Dataset(ds.access_urls['CdmRemote'])

    # Pull out the data of interest
    sweep = 0
    rng = data.variables['distanceR_HI'][:]
    az = data.variables['azimuthR_HI'][sweep]
    ref_var = data.variables['Reflectivity_HI']

    # Convert data to float and coordinates to Cartesian
    ref = raw_to_masked_float(ref_var, ref_var[sweep])
    x, y = polar_to_cartesian(az, rng)

    # Plot the data and the timestamp
    mesh = ax.pcolormesh(x, y, ref, cmap=ref_cmap, norm=ref_norm, zorder=0)
    text = ax.text(0.65, 0.03, data.time_coverage_start, transform=ax.transAxes,
                   fontdict={'size':16})

    # Collect the things we've plotted so we can animate
    meshes.append((mesh, text))


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
The CDMRemote reader provides an interface that is almost identical to the usual python NetCDF interface. We pull out the variables we need for azimuth and range, as well as the data itself.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
The specified longitude, latitude are in NY and the TDS helpfully finds the closest station to that point.  We can see that for this time range we obtained multiple datasets.
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
Use the function to make a new map and plot a colormapped view of the data
Can you write Python code for it?

fig = plt.figure(figsize=(10, 10))
ax = new_map(fig, data.StationLongitude, data.StationLatitude)

# Set limits in lat/lon space
ax.set_extent([-77, -70, 38, 42])

# Add ocean and land background
ocean = cartopy.feature.NaturalEarthFeature('physical', 'ocean', scale='50m',
                                            edgecolor='face',
                                            facecolor=cartopy.feature.COLORS['water'])
land = cartopy.feature.NaturalEarthFeature('physical', 'land', scale='50m',
                                           edgecolor='face',
                                           facecolor=cartopy.feature.COLORS['land'])

ax.add_feature(ocean, zorder=-1)
ax.add_feature(land, zorder=-1)
#ax = new_map(fig, data.StationLongitude, data.StationLatitude)
ax.pcolormesh(x, y, ref, cmap=ref_cmap, norm=ref_norm, zorder=0);


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
We can pull that dataset out of the dictionary and look at the available access URLs. We see URLs for OPeNDAP, CDMRemote, and HTTPServer (direct download).
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
We can use the RadarServer instance to check our query, to make sure we have required parameters and that we have chosen valid station(s) and variable(s)
Can you write Python code for it?


I am working on the file "NEXRAD/.ipynb_checkpoints/THREDDS_Radar_Server-checkpoint.ipynb".
The first task is:
We'll use the CDMRemote reader in Siphon and pass it the appropriate access URL.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
Download a collection of historical data
This time we'll make a query based on a longitude, latitude point and using a time range.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
Finally, we plot them up using matplotlib and cartopy. We create a helper function for making a map to keep things simpler later.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
MetPy is a Python package for meteorology (Documentation: http://metpy.readthedocs.org and GitHub: http://github.com/MetPy/MetPy). We import MetPy and use it to get the colortable and value mapping information for the NWS Reflectivity data.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
Next, we'll create a new query object to help request the data. Using the chaining methods, let's ask for the latest data at the radar KLVX (Louisville, KY). We see that when the query is represented as a string, it shows the encoded URL.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
The CDMRemote reader provides an interface that is almost identical to the usual python NetCDF interface. We pull out the variables we need for azimuth and range, as well as the data itself.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
The specified longitude, latitude are in NY and the TDS helpfully finds the closest station to that point.  We can see that for this time range we obtained multiple datasets.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
Then convert the raw data to floating point values and the polar coordinates to Cartesian.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
Use the function to make a new map and plot a colormapped view of the data
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
We can look at the datasets on the catalog to see what data we found by the query. We find one volume in the return, since we asked for the volume nearest to a single time.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD-Copy1.ipynb".
The first task is:
We'll use the CDMRemote reader in Siphon and pass it the appropriate access URL.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
Finally, we plot them up using matplotlib and cartopy. We create a helper function for making a map to keep things simpler later.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
First we'll create an instance of RadarServer to point to the appropriate radar server access URL.
Can you write Python code for it?

# The archive of data on S3 URL did not work for me, despite .edu domain
#url = 'http://thredds-aws.unidata.ucar.edu/thredds/radarServer/nexrad/level2/S3/'

#Trying motherlode URL
url = 'http://thredds.ucar.edu/thredds/radarServer/nexrad/level2/IDD/'
from siphon.radarserver import RadarServer
rs = RadarServer(url)


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
Grab the first dataset so that we can get the longitude and latitude of the station and make a map for plotting. We'll go ahead and specify some longitude and latitude bounds for the map.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
MetPy is a Python package for meteorology (Documentation: http://metpy.readthedocs.org and GitHub: http://github.com/MetPy/MetPy). We import MetPy and use it to get the colortable and value mapping information for the NWS Reflectivity data.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
Next, we'll create a new query object to help request the data. Using the chaining methods, let's ask for the latest data at the radar KLVX (Louisville, KY). We see that when the query is represented as a string, it shows the encoded URL.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
The CDMRemote reader provides an interface that is almost identical to the usual python NetCDF interface. We pull out the variables we need for azimuth and range, as well as the data itself.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
The specified longitude, latitude are in NY and the TDS helpfully finds the closest station to that point.  We can see that for this time range we obtained multiple datasets.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
Then convert the raw data to floating point values and the polar coordinates to Cartesian.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
Use the function to make a new map and plot a colormapped view of the data
Can you write Python code for it?

fig = plt.figure(figsize=(10, 10))
ax = new_map(fig, data.StationLongitude, data.StationLatitude)

# Set limits in lat/lon space
ax.set_extent([-77, -70, 38, 43])

# Add ocean and land background
ocean = cartopy.feature.NaturalEarthFeature('physical', 'ocean', scale='50m',
                                            edgecolor='face',
                                            facecolor=cartopy.feature.COLORS['water'])
land = cartopy.feature.NaturalEarthFeature('physical', 'land', scale='50m',
                                           edgecolor='face',
                                           facecolor=cartopy.feature.COLORS['land'])

ax.add_feature(ocean, zorder=-1)
ax.add_feature(land, zorder=-1)
ax.pcolormesh(x, y, ref, cmap=ref_cmap, norm=ref_norm, zorder=0);


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
We can look at the datasets on the catalog to see what data we found by the query. We find one volume in the return, since we asked for the volume nearest to a single time.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
We can use the RadarServer instance to check our query, to make sure we have required parameters and that we have chosen valid station(s) and variable(s)
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
We define some helper functions to make working with the data easier. One takes the raw data and converts it to floating point values with the missing data points appropriately marked. The other helps with converting the polar coordinates (azimuth and range) to Cartesian (x and y).
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_NEXRAD.ipynb".
The first task is:
We'll use the CDMRemote reader in Siphon and pass it the appropriate access URL.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
First we'll create an instance of RadarServer to point to the appropriate radar server access URL.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
Grab the first dataset so that we can get the longitude and latitude of the station and make a map for plotting. We'll go ahead and specify some longitude and latitude bounds for the map.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
MetPy is a Python package for meteorology (Documentation: http://metpy.readthedocs.org and GitHub: http://github.com/MetPy/MetPy). We import MetPy and use it to get the colortable and value mapping information for the NWS Reflectivity data.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
The CDMRemote reader provides an interface that is almost identical to the usual python NetCDF interface. We pull out the variables we need for azimuth and range, as well as the data itself.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
The specified longitude, latitude are in NY and the TDS helpfully finds the closest station to that point.  We can see that for this time range we obtained multiple datasets.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
Then convert the raw data to floating point values and the polar coordinates to Cartesian.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
Use the function to make a new map and plot a colormapped view of the data
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
Using matplotlib, we can take a collection of Artists that have been plotted and turn them into an animation. With matplotlib 1.5 (1.5-rc2 is available now!), this animation can be converted to HTML5 video viewable in the notebook.
Can you write Python code for it?

# Set up matplotlib to do the conversion to HTML5 video
import matplotlib
matplotlib.rcParams['animation.html'] = 'html5'

# Create an animation
from matplotlib.animation import ArtistAnimation
ArtistAnimation(fig, meshes)


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
We can look at the datasets on the catalog to see what data we found by the query. We find one volume in the return, since we asked for the volume nearest to a single time.
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
We can pull that dataset out of the dictionary and look at the available access URLs. We see URLs for OPeNDAP, CDMRemote, and HTTPServer (direct download).
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
We define some helper functions to make working with the data easier. One takes the raw data and converts it to floating point values with the missing data points appropriately marked. The other helps with converting the polar coordinates (azimuth and range) to Cartesian (x and y).
Can you write Python code for it?


I am working on the file "NEXRAD/THREDDS_Radar_Server.ipynb".
The first task is:
We'll use the CDMRemote reader in Siphon and pass it the appropriate access URL.
Can you write Python code for it?


I am working on the file "NIPS Topic Modeling.ipynb".
The first task is:
Document cleaning

Tokenizing just on spaces
no lemmatization or stemming
removing non-ascci characters
removing stop words
removing words not in Glove vocab
removing non-alpha words (e.g. Letter and symbols)
removing short words < 2 characters long
lowercasing all words
Can you write Python code for it?

corpus = []
nips_path = "/Users/michael/Documents/GaussianLDA/data/"
for folder in os.listdir(nips_path)[1:]:
    for doc in os.listdir(nips_path + folder):
        with open(nips_path + folder + "/" + doc, 'r') as f:
            txt = f.read().split()
            txt = map(lambda x: x.lower(), txt)  # Lowercasing each word
            txt = filter(lambda word: [letter for letter in word if ord(letter) < 128 ], txt)  # Checking each word for ascci error
            txt = filter(lambda x: x not in stops, txt)  # Removing stop words
            txt = filter(lambda x: x.isalpha(), txt)  # Removing non-letter words (eg numbers and symbols)
            txt = filter(lambda x: len(x) > 2, txt)  # removing super short words and single letters
            txt = filter(lambda x: x in wv_vocab, txt)
            txt = ' '.join(txt)
            corpus.append(txt)

print "Number of documents in corpus: {}".format(len(corpus))

reload(FastGaussianLDA2)
topics = 50
dim = 50
run_num = 1
outputfile = "/Users/michael/Documents/GaussianLDA/output/NIPS_{}_{}T_{}D_".format(str(run_num),
                                                                                str(topics),
                                                                                str(dim))
lda = FastGaussianLDA2.Gauss_LDA(topics, corpus, word_vector_model=wvmodel, alpha=.5, outputfile=outputfile)
lda.fit(50)  # Number of samples to run


I am working on the file "NIPS Topic Modeling.ipynb".
The first task is:
Sets of vocab to filter on: NLTK StopWords and Glove vocab
Can you write Python code for it?

wv_vocab = set(wvmodel.vocab.keys())
stops = set(corpus.stopwords.words(fileids="english"))


I am working on the file "NISTFund/NISTFund.ipynb".
The first task is:
Lengthy, detailed explanation of how to create or modify the webscraping functions make_conv_lst, make_pd_conv_lst in the case of changes made on the NIST website, breaking previous links; drop down below this section if it had worked above
If you obtain errors as such, this means that the webmasters of the NIST site made changes.  Then you'd have to manually go in and write another webscraping procedure.  However, the Python function I created at least gives a guideline or outline of how one should proceed.  Basically, it's a combination of using the Developer Inspection tool of your web browser (of choice) and using BeautifulSoup to traverse the HTML code of the webpage.
EY : 20170424 note: I had imagined before of a general Python function/webscraper, that could search out the key words or terms so that even if the webmaster(s) change(s) the webpage, it'll dynamically and automatically scrape for the table of unit conversions.  I don't know how to do that; please let me know if you do (email, twitter, github comment, etc.)
Otherwise, I go manually to the webpage and look at the link I want to scrape, directly.  I see
NIST Guide to the SI, Appendix B.8: Factors for Units Listed Alphabetically
 NIST Guide to the SI, Appendix B.9: Factors for units listed by kind of quantity or field of science
Share
Can you write Python code for it?

convBS = scraped_BS("https://www.nist.gov/physical-measurement-laboratory/nist-guide-si-appendix-b8")

convBS.soup.find_all("div",{"class":"table-inner"})

convBS.soup.find_all("table");
print(len( convBS.soup.find_all("table")) )  # there are 26 letters in the alphabet; NIST has entries for 21 of them

convBS.convtbls = convBS.soup.find_all("table")
convdata=[]
convdata2=[]
headers = convBS.convtbls[0].find_all('tr')[1].find_all('th')
headers = [ele.text.replace(' ','') for ele in headers]

headers

for tbl in convBS.convtbls:
    for row in tbl.find_all('tr'):
        if row.find_all('td') != []:
            if row.text != '':
                rowsplit = row.text.replace("\n",'',1).split('\n')
                try:
                    rowsplit = [pt.replace(u'\xa0',u' ').strip() for pt in rowsplit]
                except UnicodeDecodeError as err:
                    print rowsplit
                    Break
                    raise err
                convdata.append( rowsplit )
                if len(row.find_all('td')) == (len(headers)+1):
                    convdata2.append( row.find_all('td'))

print(len(convdata));
print(len(convdata2))

convdata3 = []
for row in convdata2:
    rowout = []
    rowout.append( row[0].text.strip())
    rowout.append( row[1].text.strip())
    value = (row[2].text+row[3].text).strip().replace(u'\xa0',' ').replace(u'\n',' ').replace(' ','')

    rowout.append(Decimal( value ))
    convdata3.append(rowout)

print(len(convdata3))

pd.DataFrame(convdata3,columns=headers).head()

print(len(convdata))


I am working on the file "NISTFund/NISTFund.ipynb".
The first task is:
NIST Official conversions
To start creating to table (when it doesn't exist yet), run the following:
Can you write Python code for it?

DF_conv=make_pd_conv_lst()

from NISTFund import scraped_BS, NISTCONValpha
convBS = scraped_BS(NISTCONValpha)

convBS.soup.find_all("table",{"class":'texttable'})


I am working on the file "NISTFund/NISTFund.ipynb".
The first task is:
The above 3 lines are what init_FundConst essentially does.
Can you write Python code for it?


I am working on the file "NISTFund/NISTFund.ipynb".
The first task is:
retrieve_file will directly download the ASCII file that NIST uses in their website for Fundamental constants.  By default, it'll check, and create if necessary, for subdirectory ./rawdata/ and put the ASCII text file there.
Can you write Python code for it?

urladdr = retrieve_file(); print(urladdr);


I am working on the file "NISTFund/NISTFund.ipynb".
The first task is:
scraping_allascii will parse this ASCII table.  Then, the following step will put it into a pandas DataFrame with the correct header.
Can you write Python code for it?

os.path.isfile('./rawdata/allascii.txt')

lines,title,src,header,rawtbl,tbl=scraping_allascii()

FundConst = pd.DataFrame(tbl, columns=header)


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Data Reading
Let's read a NIX file created by RELACS - source code for RELACS with NIX support on GitHub
Can you write Python code for it?


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Data Writing
Let's start the real deal by creating a new NIX file
Can you write Python code for it?

nf = nix.File.open('nix-demo-1.nix.h5', nix.FileMode.Overwrite)

step = 0.1
x = np.arange(0, 100, step)
y = np.sin(x)
print(x[:10])
print(y[:10])
plt.plot(x, y)

block = nf.create_block('test data set', 'nix.test.dataset')
print(block)

array = block.create_data_array('the_data', 'sin_data', nix.DataType.Double, y.shape)
array.data[:] = y

print(array.data[:10])

dim = array.create_sampled_dimension(1, step)

dim.label = 'x'
dim.unit = 's'


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Explore the data by listing all the blocks and all the DataArrays inside each block...
Can you write Python code for it?

for b in nf.blocks:
    print(b.id, b.name)

block = nf.blocks[0]

for da in block.data_arrays:
    print(da.id, da.name, da.type)


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Guppy has recorded some events for us. They are stored in multi_tags.
Can you write Python code for it?

for t in block.multi_tags:
    print("%s %s [%s]"  % (t.name, t.type, t.id))

mt = block.multi_tags['76ebce5e-79aa-4886-af10-20e13bd76066']
pos, ext = mt.positions, mt.extents
print(pos.data.shape)
for k in range(pos.data.shape[1]):
    print('tag: %d' % k)
    print(' pos: %s' % str(pos.data[:, k]))
    print(' ext: %s' % str(ext.data[:, k]))
    print('')


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Lets plot something now! We need the start point, which is either the offset stored in the SampleDimension or 0
Can you write Python code for it?

x_start = d1.offset or 0

howmany = 10000
x = np.arange(0, howmany) * d1.sampling_interval
y = v1.data[:howmany]
plt.plot(x, y)
plt.xlabel('%s [%s]' % (d1.label, d1.unit))
plt.ylabel('%s [%s]' % (v1.label, v1.unit))
plt.title('%s [%s]' % (v1.name, v1.type))
plt.xlim([np.min(x), np.max(x)])


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Ok, fetch the first tagged event, get the index manually (there is also an utility function for it, but we wanna demonstrate how it can be done) and then plot it.
Can you write Python code for it?

f_pos_t = pos.data[3, 0]
f_pos_i = filter(lambda x: x[1] == f_pos_t, enumerate(video.dimensions[3].ticks))[0][0]
f_tagged = video.data[:, :, 1, f_pos_i]
plt.imshow(f_tagged)


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
See what belongs together by inspecting MutliTags
Can you write Python code for it?

for mt in block.multi_tags:
    print('* ', mt.name, mt.type, mt.id)
    print('-'*10)
    for ref in mt.references:
        print(ref.name, ref.id)
    print('')

for s in nf.sections:
    print("* %s: %s" % (s.name, s.type))
    for sub in s.sections:
        print("|- %s: %s" % (sub.name, sub.type))
        for subsub in sub.sections:
            print("| |-%s, %s [%d]" % (subsub.name, subsub.type, len(subsub)))
            for p in subsub.props:
                print("| | |~ %s: %s %s" % (p.name, p.values[0].value, p.unit or ""))
            print("| |")
        print("| ")

v1 = block.data_arrays['456a51b4-5d5d-4896-8e4b-e24c82c0eac2']

print(v1.data.shape)

d1 = v1.dimensions[0]
print (d1.dimension_type)

print(d1.label, d1.unit, d1.sampling_interval)


I am working on the file "NIX INCF TF Meeting 2014.ipynb".
The first task is:
Video Data Example
Let's look at some video data recorded via the guppy tool.
Can you write Python code for it?

import cv2

nf = nix.File.open("2014-08-22_0.h5", nix.FileMode.ReadOnly)
block = nf.blocks[0]

for da in block.data_arrays:
    print("%10s\t%s \t [%s]" % (da.name, da.type, da.id))

video = block.data_arrays['5eeae436-db18-48e1-a7ed-2b73deb25014']

print(video.data.shape)


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
(10 pts) Write a function iterative
that outputs accuracy –– a numpy array of relative errors $\big{\frac{\|x_{k+1} - x\|_2}{\|x\|_2}\big}$ after num_iter iterations using $\tau_k$ from the previous task. Note: The only loop you are allowed to use here is a loop for $k$.
Can you write Python code for it?

# INPUT:  N - int (positive), alpha - float (positive), num_iter - integer (positive),
#  y - np.array (shape: Nx1, convolved signal), s - np.array (shape: Nx1, original signal)
# OUTPUT: rel_error - np.array size (num_iter x 1)

def iterative(N, num_iter, y, s, alpha):
    # Your code is here
    return rel_error


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
(2 pts) Set num_iter=1000, x=s[::20] and do a convergence plot for $\alpha = \frac{1}{2}$ and $\alpha = \frac{1}{5}$.
Can you write Python code for it?


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
(3 pts) Plot the first $100$ points of the result and the first $100$ points of your signal on the same figure. Do the same plots for $\alpha = \frac{1}{5}$, $\alpha = \frac{1}{100}$ using plt.subplots in matplotlib. Each subplot should contain first $100$ points of initial and convolved signals for some $\alpha$. Make sure that you got results that look like smoothed initial signal.

(2 pts) Play the resulting signal. In order to do so you should also scale the frequency (rate), which is one of the inputs in Audio.
Note that you cannot play a signal which is too small.
Can you write Python code for it?

audio_alpha1 = convolution(audio, len(audio), 0.2)
audio_alpha2 = convolution(audio, len(audio), 0.01)
# plotting
fig = plt.figure(figsize=(15,10))
fig.tight_layout()
ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)

ax1.plot(audio[:100:], label="Original")
ax1.plot(audio_alpha1[:100:], label="Convolved")
ax1.set_ylabel("Amplitude", size=13)
ax1.set_xlabel("Time", size=13)
ax1.set_title("Original and convolved signals (alpha = 0.2)", size=13)
ax1.legend(loc="upper right")

ax2.plot(audio[:100:], label="Original")
ax2.plot(audio_alpha2[:100:], label="Convolved")
ax2.set_ylabel("Amplitude", size=13)
ax2.set_xlabel("Time", size=13)
ax2.set_title("Original and convolved signals (alpha = 0.01)", size=13)
ax2.legend(loc="upper right")

# As we put every 4 sample from original audio track we need to descrease sample rate in 4 times
Audio(audio_alpha1, rate=rate/4)

Audio(audio_alpha2, rate=rate/4)


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
(3 pts) Set x=s[::20], num_iter=1000 and $\alpha=\frac{1}{5}$. Explain what happens with the convergence if you add small random noise of amplitude $10^{-3}\max(x)$  to $y$. The answer to this question should be an explanation supported by plots and/or tables.
Can you write Python code for it?


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
2. (7 pts) Implement the Strassen algorithm.
Can you write Python code for it?

def strassen(A, B):
    """
    Implement Strassen algorithm for matrix multiplication

    Parameters: Matrices A, B

    Returns: Matrix C = AB
    """
#     Your code is here
    return C


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
3. (3 pts) Plot the following two figures for your favorite picture
* How relative error of approximation depends on the rank of approximation?
* How compression rate in terms of storing information ((singular vectors + singular numbers) / total size of image) depends on the rank of approximation?
Can you write Python code for it?


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
3. (5 pts) Compare three approaches: naive multiplication, Strassen algorithm and standard NumPy function.
Provide a plot in log-scale of dependence between the matrix size and the runtime of multiplication. You will have three lines, do not forget to add legend, axis labels and other attributes (see our requirements)
Consider the matrix size in the range of 100 to 700 with step 100, e.g. $n=100, 200,\ldots, 700$.
Justify the results theoretically (e.g., use the known formulas for total multiplication complexity of naive and Strassen algorithms).
Can you write Python code for it?


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
Convolution (10 pts)

(5 pts) Write a function convolution (see below)
that takes the signal you want to convolve and multiply it by Toeplitz matrix T (for matvec operations use @ symbol).
Can you write Python code for it?

# INPUT: signal - np.array (shape: Nx1), N - int (positive), alpha - float (positive)
# OUTPUT: convolved_signal - np.array (shape: Nx1)

def convolution(signal, N, alpha):
    T = gen_toeplitz(N, alpha)
    convolved_signal = np.matmul(T, signal)
    return convolved_signal


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
Problem 2 (Theoretical tasks)  45 pts
1.
- (5 pts) Prove that $\| U A \|_F = \| A U \|_F = \| A \|_F$ for any unitary matrix $U$.
- (5 pts) Prove that $\| Ux \|_2 = \| x \|_2$ for any $x$ iff $U$ is unitary.
- (5 pts) Prove that $\| U A \|_2 = \| A U \|_2 = \| A \|_2$ for any unitary $U$.
2.
- (5 pts) Using the results from the previous subproblem, prove that $\| A \|F \le \sqrt{\mathrm{rank}(A)} \| A \|_2$. _Hint: SVD will help you.
- (5 pts) Show that for any $m, n$ and $k \le \min(m, n)$ there exists $A \in \mathbb{R}^{m \times n}: \mathrm{rank}(A) = k$, such that $\| A \|_F = \sqrt{\mathrm{rank}(A)} \| A \|_2$. In other words, show that the previous inequality is not strict.
- (5 pts) Prove that if $\mathrm{rank}(A) = 1$, then $\| A \|_F  = \| A \|_2$.
- (5 pts) Prove that $\| A B \|_F \le \| A \|_2 \| B \|_F$.
3.
* (3 pts) Differentiate with respect to $A$ the function
$$
f(A) = \mathrm{sin}(x^\top A B C D x),
$$
where $x$ is a vector and $A, B, C, D$ are square matrices.

(7 pts) Differentiate with respect to $y, A, X$ the function
$$f(y, A, X) = \mathrm{tr}(\mathrm{diag}(y) A X),$$
where $y \in \mathbb{R}^n$ and $A, X \in \mathbb{R}^{n \times n}$. Here

$$
\mathrm{diag}(y)_{i, j} =
    \begin{cases}
      y_i, & \text{if}\ i = j \
      0, & \text{otherwise}
    \end{cases}
$$
Proofs
1.
- (5 pts) Prove that $\| U A \|F = \| A U \|_F = \| A \|_F$ for any unitary matrix $U$.

According to the corollary for the Frobenius norm:
$$\| UA \|_F^2 = \mbox{trace }[( U A )^{}\, UA] = \mbox{trace }[A^{}U^{} \, UA] = \mbox{trace }[A^{}A] = \|A\|^2_F$$
Since matrix trace is invariant under cyclic permutations:
$$\| AU \|_F^2 = \mbox{trace }[( A U )^{}\, A U] = \mbox{trace }[U^{}A^{} \, AU] = \mbox{trace }[U U^{} \, A^{} A] = \mbox{trace }[A^{}A] = \|A\|^2_F $$
q.e.d
- (5 pts) Prove that $\| Ux \|_2 = \| x \|_2$ for any $x$ iff $U$ is unitary.

Suppose $U$ is unitary and $y=Ux$, then:
$$ \| Ux \|^2_2 = \sum_i \| y_i \|^2 = \sum_i y_i y_i^{} = \langle Ux,Ux \rangle = \langle x, U^{}Ux \rangle = \langle x,...
Can you write Python code for it?

def naive_multiplication(A, B):
    """
    Implement naive matrix multiplication with explicit for cycles

    Parameters: Matrices A, B

    Returns: Matrix C = AB
    """
#     Your code is here
    return C


I am working on the file "NLAhw/hw1/PS1.ipynb".
The first task is:
Problem 4 (SVD) 20 pts
In this assignment you are supposed to study how SVD could be used in image compression.
1. (2 pts) Compute the singular values of some predownloaded image (via the code provided below) and plot them. Do not forget to use logarithmic scale.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
from PIL import Image, ImageDraw
import requests
import numpy as np

url = 'https://pbs.twimg.com/profile_images/1658625695/my_photo_400x400.jpg'  # Ivan
url = 'https://i.chzbgr.com/full/5536320768/h88BAB406/' # Insight
# url = '' # your favorite picture, please!

face_raw = Image.open(requests.get(url, stream=True).raw)
face = np.array(face_raw).astype(np.uint8)

plt.imshow(face_raw)
plt.xticks(())
plt.yticks(())
plt.title('Original Picture')
plt.show()

# Your code is here


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
(1 pts) Print names of the top-10 authors according to PageRank over DBLP when $d=0.85$. Comment on your findings.
Can you write Python code for it?

top10_2 = sorted(range(len(x_k)), reverse=True, key=lambda k: x_k[k])[:10]
for i,val in enumerate(top10_2):
    print(str(i + 1) + ". ", authors[val])


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
(2 pts) Consider the same graph, but with a directed edge that goes from the node 3 to the node 4 being removed. Plot residuals as in the previous task and discuss the convergence. Now, run the power method with num_iter=100 for 10 different initial guesses and print/plot the resulting approximated eigenvectors. Why do they depend on the initial guess?
Can you write Python code for it?

G = np.matrix([[0, 0, 1, 0, 0],
               [1, 0, 1, 0, 0],
               [0, 1, 0, 0, 0],
               [0, 0, 0, 0, 1],
               [0, 0, 0, 0, 0]])
A = pagerank_matrix(G)
ev = np.linalg.eig(A)
print(sorted([abs(x) for x in ev[0]], reverse=True))

x0 = np.random.rand(G.shape[0], 1)
x, l, res = power_method(A, x0, 100)

plt.figure(figsize=(14, 7))
plt.loglog(res/res[0])
plt.ylabel("Residual", size=14)
plt.xlabel("Iteration number", size=14)
plt.title(r'Convergence plot for the largest eigenvalue (one edge is removed)', size=14)
plt.grid()
plt.show()

np.set_printoptions(precision=4)
for i in range(10):
    x0 = np.random.rand(G.shape[0], 1)
    if (i == 2):
        x0[0] = 0
        x0[1] = 0
        x0[2] = 0
        x0[3] = 1
        x0[4] = -1
    x, l, res = power_method(A, x0, 100)
    print("Eigen vector: ")
    print(x.T)


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
(5 pts) Find the second largest in the absolute value eigenvalue of the obtained matrix $A_d$. How and why is it connected to the damping factor $d$? What is the convergence rate of the PageRank algorithm when using damping factor?

Usually, graphs that arise in various areas are sparse (social, web, road networks, etc.) and, thus, computation of a matrix-vector product for corresponding PageRank matrix $A$ is much cheaper than $\mathcal{O}(N^2)$. However, if $A_d$ is calculated directly, it becomes dense and, therefore, $\mathcal{O}(N^2)$ cost grows prohibitively large for  big $N$.
Can you write Python code for it?

G = np.matrix([[0, 0, 1, 0, 0],
               [1, 0, 1, 0, 0],
               [0, 1, 0, 0, 0],
               [0, 0, 0, 0, 1],
               [0, 0, 0, 1, 0]])
A = pagerank_matrix(G)
A_d = d * A + (1 - d)/A.shape[0] * np.ones(A.shape)

ev = np.linalg.eig(A)
print("Before regularization")
print('l = ', sorted([abs(x) for x in ev[0]], reverse=True))
ev = np.linalg.eig(A_d)
print("After regularization")
print('l = ', sorted([abs(x) for x in ev[0]], reverse=True))


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
(5 pts) Implement the power method for a given matrix $A$, an initial guess $x_0$ and a number of iterations num_iter. It should be organized as a function power_method(A, x0, num_iter) that outputs approximation to eigenvector $x$, eigenvalue $\lambda$ and history of residuals ${\|Ax_k - \lambda_k x_k\|_2}$. Make sure that the method conveges to the correct solution on a matrix $\begin{bmatrix} 2 & -1 \ -1 & 2 \end{bmatrix}$ which is known to have the largest eigenvalue equal to $3$.
Can you write Python code for it?

A = np.matrix([[2, -1], [-1, 2]])
x0 = np.matrix([0, 1])
x, l, res = power_method(A, x0, 10)
print(x, l)


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Calculate top 10 nearest neighbours with the corresponding cosine similarities for the words {numerical, linear, algebra} and insert them in the correspoding functions in pset2.py.

Problem 4 (eigenvalues)  45 pts
1. Theoretical tasks

(5 pts) Prove that normal matrix is Hermitian iff its eigenvalues are real. Prove that normal matrix is unitary iff its eigenvalues satisfy $|\lambda| = 1$.

a)

normal matrix is Hermitian then eigenvalues are real
$$
A^{}A = AA^{}, A = A^{} \
Ax = \lambda x, x^{} A^{} = \overline{\lambda} x^ \
x^Ax = x^ \lambda x = \lambda \|x\|^2_2 \
(x^ A x)^ = (x^ \lambda x)^,\, x^ A^ x = \overline{\lambda} \|x\|^2_2 = x^ A x \
\lambda \|x\|^2_2 = \overline{\lambda} \|x\|^2_2 \
\lambda = \overline{\lambda} \to \lambda_i \in \mathbb{R} \
$$
eigenvalues are real then normal matrix is Hermitian
$$
x^Ax = \lambda \|x\|^2_2 \
x^ A^ x = \overline{\lambda} \|x\|^2_2 = \lambda \|x\|^2_2 \
x^(A-A^)x = \lambda \|x\|^2_2 - \lambda\|x\|^2_2 = 0 \
x \neq 0 \to A = A^ \
$$ for each non-zero eigenvector we will get $A=A^$ then $A$ is Hermitian

b)

normal matrix is unitary then eigenvalues satisfy $|\lambda| = 1$
$$
(Ax, Ax) = (\lambda x, \lambda x) = \lambda^2 \|x\|^2_2 \to \
x^ A^ A x = \lambda^2\|x\|^2_2 \to x^ x = \lambda^2 \|x\|^2_2 \to \lambda^2 = 1 \to |\lambda| = 1
$$
eigenvalues satisfy $|\lambda| = 1$ then normal matrix is unitary
$$
x^ A^ A x = \lambda^2 x^ I x \to x^(A^ A - I)x = 0
$$
for every non-zero eigen-vector $x: A^ A = I = A A^$ hence normal matrix is unitary q.e.d

(5 pts) The following problem illustrates instability of the Jordan form. Find theoretically the eigenvalues of the perturbed Jordan block:
$$
    J(\varepsilon) =
    \begin{bmatrix}
     \lambda & 1 & & & 0 \
     & \lambda & 1 & & \
     &  & \ddots & \ddots & \
     & & & \lambda & 1 \
     \varepsilon & & & & \lambda  \
    \end{bmatrix}{n\times n}
$$ Comment how eigenvalues of $J(0)$ are perturbed for large $n$.
$$
det(J(\varepsilon) - \alpha I) = 0 \
J(\varepsilon) =
    \begin{bmatrix}
   ...
Can you write Python code for it?

# implement the functions in the pset2.py file
from pset2 import pagerank_matrix, power_method, pagerank_matvec

G = np.matrix([[0, 0, 1, 0, 0],
               [1, 0, 1, 0, 0],
               [0, 1, 0, 0, 0],
               [0, 0, 0, 0, 1],
               [0, 0, 0, 1, 0]])
A = pagerank_matrix(G)
print("Matrix A:")
print(A)

ev = np.linalg.eig(A)
print(ev[0])
print("Largest eigenvalue is 1 with multiplicity 2")


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Construct the word vocabulary from the obtained sentences which enumerates words which occur more than $r=200$ times in the corpus.
Can you write Python code for it?

def create_vocabulary(sentences, r=200):
    vocabulary = {}
    # Your code is here
    return vocabulary

vocab = create_vocabulary(sentences)


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Download dataset enwik8 of compressed Wikipedia articles and preprocess raw data with Perl script main_.pl. This script will clean all unnecessary symbols, make all words to lowercase, and produce only sentences with words.
wget http://mattmahoney.net/dc/enwik8.zip
unzip enwik8.zip
mkdir data
perl main_.pl enwik8 > data/enwik8.txt
Can you write Python code for it?

# Load enwik 8

import re
file = open("data/enwik8.txt", "r")
doclist = [line for line in file]
docstr = ''.join(doclist)
sentences = re.split(r'[.!?]', docstr)
sentences = [sentence.split() for sentence in sentences if len(sentence) > 1]

print (sentences[1249])


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Here it can be seen that after regularization $\lambda_2' = d \lambda_2 $, where $d$ is damping factor.
The convergence rate is $q = d \lambda_2 $
There(p. 46) it is proven that:
If the spectrum of the stochastic matrix $S$ is ${1, λ_2, λ_3,...,λ_n}$, then
the spectrum of the Google matrix $G$ is ${1, αλ_2, αλ_3, . . . , αλ_n}$

(2 pts) Implement fast matrix-vector product for $A_d$ as a function pagerank_matvec(A, d, x), which takes a PageRank matrix $A$ (in sparse format, e.g., csr_matrix), damping factor $d$ and a vector $x$ as an input and returns $A_dx$ as an output. <br>
Done in pset2.py

(1 pts) Generate a random adjacency matrix of size $10000 \times 10000$ with only 100 non-zero elements and compare pagerank_matvec performance with direct evaluation of $A_dx$.
Can you write Python code for it?

import random
import time

N = 10000
G = np.zeros((N, N))
subG = (np.random.random((10, 10)) - 0.5).clip(0,1)
subG[subG.nonzero()] = 1
p1, p3 = random.randint(1, N - 10), random.randint(1, N - 10)
p2, p4 = p1 + 10, p3 + 10
G[p1:p2,p3:p4] = subG
x = np.random.rand(N, 1)
G = sp.sparse.csr_matrix(G)

A = pagerank_matrix(G)
d = 0.97

%%time
y = pagerank_matvec(A, d, x)

%%time
y = np.dot((d * A + (1 - d)/A.shape[0] * np.ones(A.shape)), x)


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
In out algorithm we know that the matrix is banded and apply much faster algorithm which is linear of size of matrix and quadratic of band size.
Numpy algorithm is not aware of this and thus does much worse.
2. Completing the proof of existence of LU
Some details in lecture proofs about $LU$ were omitted. Let us complete them here.
- (5 pts) Prove that if $LU$ decomposition exists, then matrix is strictly regular.
- (5 pts) Prove that if $A$ is a strictly regular matrix, then $A_1 = D - \frac 1a b c^T$ (see lectures for notations) is also strictly regular.
Can you write Python code for it?


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Problem 2 (QR decomposition) 30 pts
1. Standard Gram-Schmidt algorithm
Our goal is to orthogonalize a system of linearly independent vectors $v_1,\dots,v_n$.
The standard algorithm for this task is the Gram-Schmidt process:
$$
\begin{split}
u_1 &= v_1, \
u_2 &= v_2 - \frac{(v_2, u_1)}{(u_1, u_1)} u_1, \
\dots \
u_n &= v_n - \frac{(v_n, u_1)}{(u_1, u_1)} u_1 - \frac{(v_n, u_2)}{(u_2, u_2)} u_2 - \dots - \frac{(v_n, u_{n-1})}{(u_{n-1}, u_{n-1})} u_{n-1}.
\end{split}
$$
Obtained $u_1, \dots, u_n$ are orthogonal vectors in exact arithmetics. Then to make the system orthonormal you should divide each of the vectors by its norm: $u_i := u_i/\|u_i\|$.
The Gram-Schmidt process can be considered as a QR decomposition. Let us show that.

(2 pts) Write out what is matrices $Q$ and $R$ obtained in the process above.

(5 pts) Implement in the pset2.py the described Gram-Schmidt algorithm as a function gram_schmidt_qr(A) that takes a rectangular matrix A and outputs Q,R.

(3 pts) Create a square Vandermonde matrix $V\in\mathbb{R}^{n\times n},\ n = 20$ defined by the vector $x$: x = np.linspace(0,1,n) (components of $x$ are spaced uniformly between 0 and 1).
The loss of orthogonality can be described by the following error: $\|Q^{\top}Q-I\|_2$, where $Q^{\top}Q$ is called a Gram matrix. Compute QR decomposition of the created matrix $V$ with function that you have implemented and calculate error $\|Q^{\top}Q-I\|_2$. Comment on the result.

(5 pts) The observed loss of orthogonality is a problem of this particular algorithm. Luckily, there is a simple improvement to the algorithm above that reduces the loss of orthogonality. Implement this modification in the pset2.py as a function modified_gram_schmidt_qr(A) such that input and output are similar to gram_schmidt_qr(A).

(3 pts) Compute QR decomposition of the matrix $V$ from the previous task with the function modified_gram_schmidt_qr(A).
Compute error $\|Q^{\top}Q-I\|_2$. Compare this error to the error obtained with a "pure" Gram-Schmidt and comment on the result.
Can you write Python code for it?

# Implement the functions in the ```pset2.py``` file
from pset2 import gram_schmidt_qr
from pset2 import modified_gram_schmidt_qr

# Your code is here


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Problem 3 (Word2Vec as Matrix Factorization) 45 pts
In this assignment you are supposed to apply SVD to training your own word embedding model which maps English words to vectors of real numbers.
Skip-Gram Negative Sampling (SGNS) word embedding model, commonly known as word2vec (Mikolov et al., 2013), is usually optimized by stochastic gradient descent. However, the optimization of SGNS objective can be viewed as implicit matrix factorization objective as was shown in (Levy and Goldberg, 2015).
1. Notation
Assume we have a text corpus given as a sequence of words ${w_1,w_2,\dots,w_n}$ where $n$ may be larger than $10^{12}$ and $w_i \in \mathcal{V}$ belongs to a vocabulary of words $\mathcal{V}$. A word $c \in \mathcal{V}$ is called a context of word $w_i$ if they are found together in the text. More formally, given some measure $L$ of closeness between two words (typical choice is $L=2$), a word $c \in \mathcal{V}$ is called a context if $c \in {w_{i-L}, \dots, w_{i-1}, w_{i+1}, \dots, w_{i+L} }$ Let $\mathbf{w},\mathbf{c}\in\mathbb{R}^d$ be the word embeddings of word $w$ and context $c$, respectively. Assume they are specified by the mapping  $\Phi:\mathcal{V}\rightarrow\mathbb{R}^d$, so $\mathbf{w}=\Phi(w)$. The ultimate goal of SGNS word embedding model is to fit a good mapping $\Phi$.
Let $\mathcal{D}$ be a multiset of all word-contexts pairs observed in the corpus. In the SGNS model, the probability that word-context pair $(w,c)$ is observed in the corpus is modeled as the following distribution:
$$
P(#(w,c)\neq 0|w,c) = \sigma(\mathbf{w}^\top \mathbf{c}) = \frac{1}{1 + \exp(-\mathbf{w}^\top \mathbf{c})},
$$
where $#(w,c)$ is the number of times the pair $(w,c)$ appears in $\mathcal{D}$ and $\mathbf{w}^\top\mathbf{c}$ is the scalar product of vectors $\mathbf{w}$ and $\mathbf{c}$. Two important quantities which we will also use further are the number of times the word $w$ and the context $c$ appear in $\mathcal{D}$, which can be computed as
$$
#(w) = \sum_{c\in\mathcal{V}} #(w,c), \quad #(c) =...
Can you write Python code for it?


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
So convergence ratio is less than 1 hence it converges. The result of process depends on the initial vector as it might to the closest collinear vector (as it can be seen when $i = 2$).
In order to avoid this problem Larry Page and Sergey Brin proposed to use the following regularization technique:
$$
A_d = dA + \frac{1-d}{N} \begin{pmatrix} 1 & \dots & 1 \ \vdots & & \vdots \ 1 & \dots & 1 \end{pmatrix},
$$
where $d$ is a small parameter in $[0,1]$ (typically $d=0.85$), which is called damping factor, $A$ is of size $N\times N$. Now $A_d$ is the matrix with multiplicity of the largest eigenvalue equal to 1.
Recall that computing the eigenvector of the PageRank matrix, which corresponds to the largest eigenvalue, has the following interpretation. Consider a person who stays in a random node of a graph (i.e. opens a random web page); at each step s/he follows one of the outcoming edges uniformly at random (i.e. opens one of the links). So the person randomly walks through the graph and the eigenvector we are looking for is exactly his/her stationary distribution â€” for each node it tells you the probability of visiting this particular node. Therefore, if the person has started from a part of the graph which is not connected with the other part, he will never get there.  In the regularized model, the person at each step follows one of the outcoming links with probability $d$ OR teleports to a random node from the whole graph with probability $(1-d)$.

(2 pts) Now, run the power method with $A_d$ and plot residuals $\|A_d x_k - \lambda_k x_k\|_2$ as a function of $k$ for $d=0.97$, num_iter=100 and a random initial guess x0.
Can you write Python code for it?

d = 0.97

A_d = d * A + (1 - d)/A.shape[0] * np.ones(A.shape)

x0 = np.random.rand(A_d.shape[0], 1)
x, l, res = power_method(A_d, x0, 100)

plt.figure(figsize=(14, 7))
plt.loglog(res/res[0])
plt.ylabel("Residual", size=14)
plt.xlabel("Iteration number", size=14)
plt.title(r'Convergence plot for the largest eigenvalue (with regularization)', size=14)
plt.grid()
plt.show()


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Task 2 (practical) 40 pts
Can you write Python code for it?

import os
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from scipy.sparse.linalg import svds


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
To find good word embeddings, Levy and Goldberg, 2015 proposed to find rank-$d$ SVD of Shifted Positive Pointwise Mutual Information (SPPMI) matrix

$$
U \Sigma V^\top \approx \text{SPPMI},
$$
where $\text{SPPMI}(w, c) = \max\left(\text{SPMI}(w, c), 0 \right)$ and $\text{SPMI}(w, c)$ is the element of the matrix $\text{SPPMI}$ at position $(w, c)$.
Then use $W=U\sqrt{\Sigma}$ as word embedding matrix. Your task is to reproduce their results. Write function constructs $\text{SPPMI}$ matrix, computes its SVD and produces word-vectors matrix $W$. Pay attention that $\text{SPPMI}$ matrix is sparse!
Can you write Python code for it?

def compute_embeddings(D, k, d=200):
    # Your code is here
    return embedding_matrix

k = 5 # negative sampling parameter
W = compute_embeddings(D, k)


I am working on the file "NLAhw/hw2/PS2.ipynb".
The first task is:
Write class WordVectors using provided template.
Can you write Python code for it?

class WordVectors:

    def __init__(self, vocabulary, embedding_matrix):
        self.vocab = vocabulary
        self.W = embedding_matrix
        self.inv_vocab = {v: k for k, v in self.vocab.items()}

    def word_vector(self, word):
        """
        Takes word and returns its word vector.
        """
#         Your code is here
        return word_vector

    def nearest_words(self, word, top_n=10):
        """
        Takes word from the vocabulary and returns its top_n
        nearest neighbors in terms of cosine similarity.
        """
#         Your code is here
        return neighbors

model = WordVectors(vocab, W)

model.nearest_words("anarchism")

model.nearest_words("ussr")

model.nearest_words("rap")


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
(2 pts) Consider the same graph, but with a directed edge that goes from the node 3 to the node 4 being removed. Plot residuals as in the previous task and discuss the convergence. Now, run the power method with num_iter=100 for 10 different initial guesses and print/plot the resulting approximated eigenvectors. Why do they depend on the initial guess?
Can you write Python code for it?


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
(2 pts) Run the power method for the graph presented above and plot residuals $\|Ax_k - \lambda_k x_k\|_2$ as a function of $k$ for num_iter=100 and random initial guess x0.  Explain the absence of convergence.

Convergence ratio $q$ is equal to 1 (as two largest eigenvalues are ones). As $q$ should be less than 1 hence there will be no convergence.
Can you write Python code for it?

A = pagerank_matrix(G)
x0 = np.random.rand(G.shape[0], 1)
x, l, res = power_method(A, x0, 100)

plt.figure(figsize=(14, 7))
plt.loglog(res/res[0])
plt.ylabel("Residual", size=14)
plt.xlabel("Iteration number", size=14)
plt.title(r'Convergence plot for the largest eigenvalue', size=14)
plt.grid()
plt.show()


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
(5 pts) Find the second largest in the absolute value eigenvalue of the obtained matrix $A_d$. How and why is it connected to the damping factor $d$? What is the convergence rate of the PageRank algorithm when using damping factor?

Usually, graphs that arise in various areas are sparse (social, web, road networks, etc.) and, thus, computation of a matrix-vector product for corresponding PageRank matrix $A$ is much cheaper than $\mathcal{O}(N^2)$. However, if $A_d$ is calculated directly, it becomes dense and, therefore, $\mathcal{O}(N^2)$ cost grows prohibitively large for  big $N$.
Can you write Python code for it?