I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Padding sequences
To deal with both short and very long reviews, we'll pad or truncate all our reviews to a specific length. For reviews shorter than some seq_length, we'll pad with 0s. For reviews longer than seq_length, we can truncate them to the first seq_length words. A good seq_length, in this case, is 200.

Exercise: Define a function that returns an array features that contains the padded data, of a standard size, that we'll pass to the network.
* The data should come from review_ints, since we want to feed integers to the network.
* Each row should be seq_length elements long.
* For reviews shorter than seq_length words, left pad with 0s. That is, if the review is ['best', 'movie', 'ever'], [117, 18, 128] as integers, the row will look like [0, 0, 0, ..., 0, 117, 18, 128].
* For reviews longer than seq_length, use only the first seq_length words as the feature vector.

As a small example, if the seq_length=10 and an input review is:
[117, 18, 128]
The resultant, padded sequence should be:
[0, 0, 0, 0, 0, 0, 0, 117, 18, 128]
Your final features array should be a 2D array, with as many rows as there are reviews, and as many columns as the specified seq_length.
This isn't trivial and there are a bunch of ways to do this. But, if you're going to be building your own deep learning networks, you're going to have to get used to preparing your data.
Can you write Python code for it?

def pad_features(reviews_ints, seq_length):
    ''' Return features of review_ints, where each review is padded with 0's
        or truncated to the input seq_length.
    '''

    # Getting the correct rows x cols shape
    features = np.zeros((len(reviews_ints), seq_length), dtype=int)

    # For each review, I grab that review and
    for i, row in enumerate(reviews_ints):
        features[i, -len(row):] = np.array(row)[:seq_length]

    return features

# Test your implementation!

seq_length = 200

features = pad_features(reviews_ints, seq_length=seq_length)
features = features.astype(int)

## Test statements - do not change - ##
assert len(features) == len(reviews_ints), "Your features should have as many rows as reviews."
assert len(features[0]) == seq_length, "Each feature row should contain seq_length values."

# Print first 10 values of the first 30 batches
print(features[:30,:10])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Removing Outliers
As an additional pre-processing step, we want to make sure that our reviews are in good shape for standard processing. That is, our network will expect a standard input text size, and so, we'll want to shape our reviews into a specific length. We'll approach this task in two main steps:

Getting rid of extremely long or short reviews; the outliers
Padding/truncating the remaining data so that we have reviews of the same length.

Before we pad our review text, we should check for reviews of extremely short or long lengths; outliers that may mess with our training.
Can you write Python code for it?

# Outlier review stats
review_lens = Counter([len(x) for x in reviews_ints])
print("Zero-length reviews: {}".format(review_lens[0]))
print("Maximum review length: {}".format(max(review_lens)))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Sentiment Network with PyTorch
Below is where you'll define the network.
<img src="assets/network_diagram.png" width=40%>
The layers are as follows:
1. An embedding layer that converts our word tokens (integers) into embeddings of a specific size.
2. An LSTM layer defined by a hidden_state size and number of layers
3. A fully-connected output layer that maps the LSTM layer outputs to a desired output_size
4. A sigmoid activation layer which turns all outputs into a value 0-1; return only the last sigmoid output as the output of this network.
The Embedding Layer
We need to add an embedding layer because there are 74000+ words in our vocabulary. It is massively inefficient to one-hot encode that many classes. So, instead of one-hot encoding, we can have an embedding layer and use that layer as a lookup table. You could train an embedding layer using Word2Vec, then load it here. But, it's fine to just make a new layer, using it for only dimensionality reduction, and let the network learn the weights.
The LSTM Layer(s)
We'll create an LSTM to use in our recurrent network, which takes in an input_size, a hidden_dim, a number of layers, a dropout probability (for dropout between multiple layers), and a batch_first parameter.
Most of the time, you're network will have better performance with more layers; between 2-3. Adding more layers allows the network to learn really complex relationships.

Exercise: Complete the __init__, forward, and init_hidden functions for the SentimentRNN model class.

Note: init_hidden should initialize the hidden and cell state of an lstm layer to all zeros, and move those state to GPU, if available.
Can you write Python code for it?

# First checking if GPU is available
train_on_gpu=torch.cuda.is_available()

if(train_on_gpu):
    print('Training on GPU.')
else:
    print('No GPU available, training on CPU.')

train_on_gpu = False

import torch.nn as nn

class SentimentRNN(nn.Module):
    """
    The RNN model that will be used to perform Sentiment analysis.
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Initialize the model by setting up the layers.
        """
        super(SentimentRNN, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        # Embedding and LSTM layers
        self.embedding = nn.Embedding(num_embeddings=vocab_size,
                                      embedding_dim=embedding_dim)
        self.lstm = nn.LSTM(input_size=embedding_dim,
                            hidden_size=hidden_dim,
                            num_layers=n_layers,
                            dropout=drop_prob,
                            batch_first=True)

        # Dropout layer
        self.dropout = nn.Dropout(p=0.3)

        # Linear and sigmoid layers
        self.fc = nn.Linear(in_features=hidden_dim,
                            out_features=output_size)
        self.sig = nn.Sigmoid()

    def forward(self, x, hidden):
        """
        Perform a forward pass of our model on some input and hidden state.
        """
        batch_size = x.size(0)

        # Embeddings and lstm_out
        embeds = self.embedding(x)
        lstm_out, hidden = self.lstm(embeds, hidden)

        # Stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        # Dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)
        # Sigmoid function
        sig_out = self.sig(out)

        # Reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1)
       ...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Test your code
As a text that you've implemented the dictionary correctly, print out the number of unique words in your vocabulary and the contents of the first, tokenized review.
Can you write Python code for it?

# Stats about vocabulary
print('Unique words: ', len((vocab_to_int)))  # should ~ 74000+
print()

# Print tokens in first review
print('Tokenized review: \n', reviews_ints[:1])


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/06-Sentiment Prediction with RNNs/Sentiment_analysis_with_RNNs.ipynb".
The first task is:
Training, Validation, Test
With our data in nice shape, we'll split it into training, validation, and test sets.

Exercise: Create the training, validation, and test sets.
* You'll need to create sets for the features and the labels, train_x and train_y, for example.
* Define a split fraction, split_frac as the fraction of data to keep in the training set. Usually this is set to 0.8 or 0.9.
* Whatever data is left will be split in half to create the validation and testing data.
Can you write Python code for it?

split_frac = 0.8

## Split data into training, validation, and test data (features and labels, x and y)

split_idx = int(len(features)*0.8)
train_x, remaining_x = features[:split_idx], features[split_idx:]
train_y, remaining_y = encoded_labels[:split_idx], encoded_labels[split_idx:]

test_idx = int(len(remaining_x)*0.5)
val_x, test_x = remaining_x[:test_idx], remaining_x[test_idx:]
val_y, test_y = remaining_y[:test_idx], remaining_y[test_idx:]

## Print out the shapes of your resultant feature data
print("\t\t\tFeature Shapes:")
print("Train set: \t\t{}".format(train_x.shape),
      "\nValidation set: \t{}".format(val_x.shape),
      "\nTest set: \t\t{}".format(test_x.shape))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/07-Deploying PyTorch Models/.ipynb_checkpoints/Deploying_PyTorch_models_in_production-checkpoint.ipynb".
The first task is:
PyTorch deployment requires using an unstable version of PyTorch (1.0.0+).
In order to install this version, use "Preview" option when choosing PyTorch version.
https://pytorch.org/
Can you write Python code for it?

# Let's create an example model using ResNet-18
model = torchvision.models.resnet18()

model

# Creating a sample of the input
# It will be used to pass it to the network to build the dimensions
sample = torch.rand(size=(1, 3, 224, 224))

# Creating so called "traced Torch script"
traced_script_module = torch.jit.trace(model, sample)

traced_script_module

# The TracedModule is capable of making predictions
sample_prediction = traced_script_module(torch.ones(size=(1, 3, 224, 224)))

sample_prediction.shape

# Serializing the the script module
traced_script_module.save('./models_deployment/model.pt')


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/07-Deploying PyTorch Models/Deploying_PyTorch_models_in_production.ipynb".
The first task is:
PyTorch deployment requires using an unstable version of PyTorch (1.0.0+).
In order to install this version, use "Preview" option when choosing PyTorch version.
https://pytorch.org/
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/07-Deploying PyTorch Models/Deploying_PyTorch_models_in_production.ipynb".
The first task is:
Requirements
Can you write Python code for it?


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/Final Lab/Image Classifier Project.ipynb".
The first task is:
Building and training the classifier
Now that the data is ready, it's time to build and train the classifier. As usual, you should use one of the pretrained models from torchvision.models to get the image features. Build and train a new feed-forward classifier using those features.
We're going to leave this part up to you. If you want to talk through it with someone, chat with your fellow students! You can also ask questions on the forums or join the instructors in office hours.
Refer to the rubric for guidance on successfully completing this section. Things you'll need to do:

Load a pre-trained network (If you need a starting point, the VGG networks work great and are straightforward to use)
Define a new, untrained feed-forward network as a classifier, using ReLU activations and dropout
Train the classifier layers using backpropagation using the pre-trained network to get the features
Track the loss and accuracy on the validation set to determine the best hyperparameters

We've left a cell open for you below, but use as many as you need. Our advice is to break the problem up into smaller parts you can run separately. Check that each part is doing what you expect, then move on to the next. You'll likely find that as you work through each part, you'll need to go back and modify your previous code. This is totally normal!
When training make sure you're updating only the weights of the feed-forward network. You should be able to get the validation accuracy above 70% if you build everything right. Make sure to try different hyperparameters (learning rate, units in the classifier, epochs, etc) to find the best model. Save those hyperparameters to use as default values in the next part of the project.
Can you write Python code for it?

# TODO: Build and train your network
def build_model(class_to_idx):
    model = models.resnet18(pretrained=True)

    # Freezing the features
    for param in model.parameters():
        param.requires_grad_(False)
    from collections import OrderedDict

    model.fc = nn.Sequential(OrderedDict([
            ('fc1', nn.Linear(512, 512)),
            ('relu', nn.ReLU()),
            ('fc2', nn.Linear(512, 102)),
            ('output', nn.LogSoftmax(dim=1))

           ]))

    # Setting trainable parameters
    for trainable_params in model.fc.parameters():
        trainable_params.requires_grad_(True)
    model.class_to_idx = class_to_idx
    print('Model created!')
    return model

model = build_model(class_to_idx=class_to_idx)

# Check if CUDA is available
train_on_gpu = torch.cuda.is_available()

if train_on_gpu:
    model.cuda()

train_on_gpu

# Specify loss function (categorical cross-entropy)
criterion = nn.NLLLoss()

# Specify optimizer
learning_rate = 0.0001
optimizer = optim.Adam(params=model.fc.parameters(),
                       lr=learning_rate)

from tqdm import tqdm_notebook as tqdm

best_weights_path = './models/lab_best_model_state_dict.pth'

# Number of epochs to train the model
n_epochs = 1000

valid_loss_min = np.Inf # track change in validation loss

for epoch in range(1, n_epochs + 1):

    # Keep track of training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    # train the model
    model.train()
    for data, target in tqdm(train_dataloader):
        # Move tensors to GPU if CUDA is available
        if train_on_gpu:
            data, target = data.cuda(), target.cuda()
        # Clear the gradients of all optimized variables
        optimizer.zero_grad()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the batch loss
        loss = criterion(output, target)
        # Backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        #...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/Final Lab/Image Classifier Project.ipynb".
The first task is:
Class Prediction
Once you can get images in the correct format, it's time to write a function for making predictions with your model. A common practice is to predict the top 5 or so (usually called top-$K$) most probable classes. You'll want to calculate the class probabilities then find the $K$ largest values.
To get the top $K$ largest values in a tensor use x.topk(k). This method returns both the highest k probabilities and the indices of those probabilities corresponding to the classes. You need to convert from these indices to the actual class labels using class_to_idx which hopefully you added to the model or from an ImageFolder you used to load the data (see here). Make sure to invert the dictionary so you get a mapping from index to class as well.
Again, this method should take a path to an image and a model checkpoint, then return the probabilities and classes.
```python
probs, classes = predict(image_path, model)
print(probs)
print(classes)

[ 0.01558163  0.01541934  0.01452626  0.01443549  0.01407339]
['70', '3', '45', '62', '55']
```
Can you write Python code for it?

def predict(image_path, model, top_num=5):
    # Process image
    image = process_image(image=Image.open(image_path))

    # Numpy -> Tensor
    image_tensor = torch.from_numpy(image).type(torch.FloatTensor)
    # Adding additional dimension to the it
    model_input = image_tensor.unsqueeze(0)

    # Calculating the probabilities
    probabilities = torch.exp(model.forward(model_input))

    # Top probabilities
    top_probabilities, top_labels = probabilities.topk(top_num)
    top_probabilities = top_probabilities.detach().numpy().tolist()[0]
    top_labels = top_labels.detach().numpy().tolist()[0]

    # Convert indices to classes
    idx_to_class = {val: key for key, val in
                                      model.class_to_idx.items()}
    top_k_labels = [idx_to_class[lab] for lab in top_labels]
    top_flowers = [cat_to_name[idx_to_class[lab]] for lab in top_labels]
    return top_probabilities, top_k_labels, top_flowers

def predict(image_path, model, topk=5, use_gpu=True):
    ''' Predict the class (or classes) of an image using a trained deep learning model.
    '''
    model.eval()
    train_on_gpu = torch.cuda.is_available()

    model = model.cuda() if train_on_gpu else model.cpu()

    image = Image.open(image_path)
    np_array = process_image(image)
    tensor = torch.from_numpy(np_array)

    tensor_on_device = tensor.float().cuda() if train_on_gpu else tensor
    with torch.no_grad():
        var_inputs = torch.autograd.Variable(tensor_on_device)
    var_inputs = var_inputs.unsqueeze(0)
    output = model.forward(var_inputs)
    predictions = torch.exp(output).data.topk(topk)
    probabilities = predictions[0].cpu() if use_gpu else predictions[0]
    classes = predictions[1].cpu() if use_gpu else predictions[1]
    class_to_idx_inverted = {model.class_to_idx[k]: k for k in model.class_to_idx}
    mapped_classes = []
    for label in classes.numpy()[0]:
        mapped_classes.append(class_to_idx_inverted[label])
    return probabilities.numpy()[0],...


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/Final Lab/Image Classifier Project.ipynb".
The first task is:
Inference for classification
Now you'll write a function to use a trained network for inference. That is, you'll pass an image into the network and predict the class of the flower in the image. Write a function called predict that takes an image and a model, then returns the top $K$ most likely classes along with the probabilities. It should look like
```python
probs, classes = predict(image_path, model)
print(probs)
print(classes)

[ 0.01558163  0.01541934  0.01452626  0.01443549  0.01407339]
['70', '3', '45', '62', '55']
```

First you'll need to handle processing the input image such that it can be used in your network.
Image Preprocessing
You'll want to use PIL to load the image (documentation). It's best to write a function that preprocesses the image so it can be used as input for the model. This function should process the images in the same manner used for training.
First, resize the images where the shortest side is 256 pixels, keeping the aspect ratio. This can be done with the thumbnail or resize methods. Then you'll need to crop out the center 224x224 portion of the image.
Color channels of images are typically encoded as integers 0-255, but the model expected floats 0-1. You'll need to convert the values. It's easiest with a Numpy array, which you can get from a PIL image like so np_image = np.array(pil_image).
As before, the network expects the images to be normalized in a specific way. For the means, it's [0.485, 0.456, 0.406] and for the standard deviations [0.229, 0.224, 0.225]. You'll want to subtract the means from each color channel, then divide by the standard deviation.
And finally, PyTorch expects the color channel to be the first dimension but it's the third dimension in the PIL image and Numpy array. You can reorder dimensions using ndarray.transpose. The color channel needs to be first and retain the order of the other two dimensions.
Can you write Python code for it?

def process_image(image):
    ''' Scales, crops, and normalizes a PIL image for a PyTorch model,
        returns an Numpy array
    '''
    size = 224
    # TODO: Process a PIL image for use in a PyTorch model
    width, height = image.size

    if height > width:
        height = int(max(height * size / width, 1))
        width = int(size)
    else:
        width = int(max(width * size / height, 1))
        height = int(size)

    resized_image = image.resize((width, height))

    # Calculating points for cropping the image, left bottom and top right
    x_left = (width - size) / 2
    y_bottom = (height - size) / 2
    x_right = x_left + size
    y_top = y_bottom + size
    # Cropping the image
    cropped_image = image.crop((x_left,
                                y_bottom,
                                x_right,
                                y_top))
    np_image = np.array(cropped_image) / 255.
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    np_image_array = (np_image - mean) / std
    np_image_array = np_image.transpose((2, 0, 1))

    return np_image_array

process_image(image = Image.open('./flower_data/valid/10/image_07094.jpg'))


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/Final Lab/Image Classifier Project.ipynb".
The first task is:
Label mapping
You'll also need to load in a mapping from category label to category name. You can find this in the file cat_to_name.json. It's a JSON object which you can read in with the json module. This will give you a dictionary mapping the integer encoded categories to the actual names of the flowers.
Can you write Python code for it?

import json

with open('cat_to_name.json', 'r') as f:
    cat_to_name = json.load(f)

cat_to_name


I am working on the file "18-11-22-Deep-Learning-with-PyTorch/Final Lab/Image Classifier Project.ipynb".
The first task is:
Sanity Checking
Now that you can use a trained model for predictions, check to make sure it makes sense. Even if the validation accuracy is high, it's always good to check that there aren't obvious bugs. Use matplotlib to plot the probabilities for the top 5 classes as a bar graph, along with the input image. It should look like this:
<img src='images/inference_example.png' width=300px>
You can convert from the class integer encoding to actual flower names with the cat_to_name.json file (should have been loaded earlier in the notebook). To show a PyTorch tensor as an image, use the imshow function defined above.
Can you write Python code for it?

# TODO: Display an image along with the top 5 classes
image_path = './flower_data/valid/10/image_07094.jpg'
probabilities, classes = predict(image_path, model)

max_index = np.argmax(probabilities)
max_probability = probabilities[max_index]
label = classes[max_index]

fig = plt.figure(figsize=(10,10))
ax1 = plt.subplot2grid((15,9), (0,0),
                       colspan=9,
                       rowspan=9)
ax2 = plt.subplot2grid((15,9),
                       (9,2),
                       colspan=5,
                       rowspan=5)

image = Image.open(image_path)
ax1.axis('off')
ax1.set_title(cat_to_name[label])
ax1.imshow(image)
labels = []
for i_class in classes:
    labels.append(cat_to_name[i_class])
y_pos = np.arange(5)
ax2.set_yticks(y_pos)
ax2.set_yticklabels(labels)
ax2.invert_yaxis()
ax2.set_xlabel('Probability')
ax2.barh(y_pos,
         probabilities,
         xerr=0,
         align='center')

plt.show()


I am working on the file "18-pyspark-01.ipynb".
The first task is:
1. Getting Started
Spark stores data in memory. This memory space is represented by variable sc (SparkContext).
Can you write Python code for it?

!cypress-kinit
!klist

import sys
import os

sys.path.insert(0, '/usr/hdp/current/spark2-client/python')
sys.path.insert(0, '/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip')

os.environ['SPARK_HOME'] = '/usr/hdp/current/spark2-client/'
os.environ['SPARK_CONF_DIR'] = '/etc/hadoop/synced_conf/spark2/'
os.environ['PYSPARK_PYTHON'] = '/software/anaconda3/4.2.0/bin/python'

import pyspark
conf = pyspark.SparkConf()
conf.setMaster("yarn")
conf.set("spark.driver.memory","4g")
conf.set("spark.executor.memory","60g")
conf.set("spark.num.executors","3")
conf.set("spark.executor.cores","12")

sc = pyspark.SparkContext(conf=conf)

sc

textFile = sc.textFile("/repository/gutenberg-shakespeare.txt")

print (textFile)


I am working on the file "18-pyspark-01.ipynb".
The first task is:
2. What does Spark do with my data?
Storage Level:
- Does RDD use disk?
- Does RDD use memory?
- Does RDD use off-heap memory?
- Should an RDD be serialized (while persisting)?
- How many replicas (default: 1) to use (can only be less than 40)?
Can you write Python code for it?

textFile.getStorageLevel()

textFile.getNumPartitions()

textFile.cache()

textFile.getStorageLevel()


I am working on the file "18-pyspark-01.ipynb".
The first task is:
By default, each transformed RDD may be recomputed each time you run an action on it.
It is also possible to persist RDD in memory using persist() or cache()
persist() allows you to specify level of storage for RDD
cache() only persists RDD in memory
To retire RDD from memory, unpersist() is called

3. WordCount
Data operations in Spark are categorized into two groups, transformation and action.
- A transformation creates new dataset from existing data. Examples of transformation include map, filter, reduceByKey, and sort.
- An action returns a value to the driver program (aka memory space of this notebook) after running a computation on the data set. Examples of action include count, collect, reduce, and save.
"All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program." -- Spark Documentation
RDD Operations in Spark
Transformations:

map(f: T -> U) : RDD[T] -> RDD[U]
filter(f: T -> Bool) : RDD[T] -> RDD[T]
flatMap(f: T -> Seq[U]) : RDD[T] -> RDD[U]
sample(fraction: Float) : RDD[T] -> RDD[T] (deterministic sampling)
groupByKey() : RDD[(K,V)] -> RDD[(K, Seq[V])]
reduceByKey(f: (V,V) -> V) : RDD[(K,V)] -> RDD[(K,V)]
union() : (RDD[T], RDD[T]) -> RDD[T]
join() : (RDD[(K,V)], RDD[(K,W)]) -> RDD[(K,(V,W))]
cogroup() : (RDD[(K,V)], RDD[(K,W)] -> RDD[(K, (Seq[V],Seq[W]))]
crossProduct() : (RDD[T], RDD[U]) -> RDD[(T,U)]
mapValues(f: V -> W) : RDD[(K,V)] -> RDD[(K,W)] (preserves partitioning)
sort(c: Comparator[K]) :  RDD[(K,V)] -> RDD[(K,V)]
partitionBy(p: Partitioner[K]) : RDD[(K,V)] -> RDD[(K,V)]

Actions:

count() : RDD[T] -> Long
collect() : RDD[T] -> Seq[T]
reduce(f: (T,T) -> T) : RDD[T] -> T
lookup(k : K) : RDD[(K,V)] -> Seq[V] (on hash/range partitionied RDDs)
save(path: String) : Outputs RDD to a storage system
Can you write Python code for it?

textFile = sc.textFile("/repository/gutenberg-shakespeare.txt")

textFile

%%time
textFile.count()

wordcount = textFile.flatMap(lambda line: line.split(" ")) \
            .map(lambda word: (word, 1)) \
            .reduceByKey(lambda a, b: a + b)

wordcount

!hdfs dfs -rm -r intro-to-spark
!hdfs dfs -mkdir intro-to-spark

wordcount.saveAsTextFile("intro-to-spark/output-wordcount-01")

!hdfs dfs -cat intro-to-spark/output-wordcount-01/part-00000 \
    2>/dev/null | head -n 20


I am working on the file "18-pyspark-01.ipynb".
The first task is:
Challenge

Augment the mapping process of WordCount with a function to filter out punctuations and capitalization from the unique words

To stop the Spark job, call sc.stop()
Can you write Python code for it?


I am working on the file "18-pyspark-03.ipynb".
The first task is:
Airlines Data
Spark SQL
- Spark module for structured data processing
- provide more information about the structure of both the data and the computation being performed for additional optimization
- execute SQL queries written using either a basic SQL syntax or HiveQL
DataFrame
- distributed collection of data organized into named columns
- conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood
- can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs.
Can you write Python code for it?

sqlContext = pyspark.SQLContext(sc)
sqlContext

airlines = sqlContext.read.format("com.databricks.spark.csv")\
    .option("header", "true")\
    .option("inferschema", "true")\
    .load("/repository/airlines/data/")\
    .cache()

%%time
airlines.count()

%%time
airlines.count()

airlines.printSchema()


I am working on the file "18-pyspark-03.ipynb".
The first task is:
How do you display full carrier names?
Can you write Python code for it?

carriers = sqlContext.read.format("com.databricks.spark.csv")\
    .option("header", "true")\
    .option("inferschema", "true")\
    .load("/repository/airlines/metadata/carriers.csv")\
    .cache()
carriers.registerTempTable("carriers")

carriers.printSchema()

%%time
carrierFlightCountFullName = sqlContext.sql("SELECT c.Description, a.UniqueCarrier, COUNT(a.UniqueCarrier) AS FlightCount \
                                    FROM airlines AS a \
                                    INNER JOIN carriers AS c \
                                    ON c.Code = a.UniqueCarrier \
                                    GROUP BY a.UniqueCarrier, c.Description \
                                    ORDER BY a.UniqueCarrier")
carrierFlightCountFullName.show()


I am working on the file "18-pyspark-03.ipynb".
The first task is:
How many unique airlines are there?
Can you write Python code for it?

uniqueAirline = sqlContext.sql("SELECT DISTINCT UniqueCarrier \
                                FROM airlines")
uniqueAirline.show()


I am working on the file "18-pyspark-03.ipynb".
The first task is:
What is the averaged departure delay time for each airline?
Can you write Python code for it?

%%time
avgDepartureDelay = sqlContext.sql("SELECT FIRST(c.Description), FIRST(a.UniqueCarrier), AVG(a.DepDelay) AS AvgDepDelay \
                                    FROM airlines AS a \
                                    INNER JOIN carriers AS c \
                                    ON c.Code = a.UniqueCarrier \
                                    GROUP BY a.UniqueCarrier \
                                    ORDER BY a.UniqueCarrier")
avgDepartureDelay.show()

airlines.unpersist()

sc.stop()


I am working on the file "18-pyspark-03.ipynb".
The first task is:
You can interact with a DataFrame via SQLContext using SQL statements by registerting the DataFrame as a table
Can you write Python code for it?


I am working on the file "18. 분류의 기초/04. 분류(classification) 성능 평가.ipynb".
The first task is:
AUC (Area Under the Curve)
AUC는 ROC curve의 면적을 뜻한다. Fall-Out 대비 Recall 값이 클 수록 AUC가 1에 가까운 값이며 우수한 모형이다.
Can you write Python code for it?

from sklearn.metrics import auc
auc(fpr0, tpr0), auc(fpr1, tpr1), auc(fpr2, tpr2)


I am working on the file "18. 분류의 기초/04. 분류(classification) 성능 평가.ipynb".
The first task is:
Multi-Class 예제
Can you write Python code for it?

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
iris = load_iris()
model = LogisticRegression().fit(iris.data, iris.target)

from sklearn.metrics import roc_curve
fpr0, tpr0, thresholds0 = roc_curve(iris.target, model.decision_function(iris.data)[:, 0], pos_label=0)
fpr1, tpr1, thresholds1 = roc_curve(iris.target, model.decision_function(iris.data)[:, 1], pos_label=1)
fpr2, tpr2, thresholds2 = roc_curve(iris.target, model.decision_function(iris.data)[:, 2], pos_label=2)

fpr0, tpr0, thresholds0

plt.plot(fpr0, tpr0, "r-", label="class 0 ")
plt.plot(fpr1, tpr1, "g-", label="class 1")
plt.plot(fpr2, tpr2, "b-", label="class 2")
plt.plot([0, 1], [0, 1], 'k--', label="random guess")
plt.xlim(-0.05, 1.0)
plt.ylim(0, 1.05)
plt.xlabel('False Positive Rate (Fall-Out)')
plt.ylabel('True Positive Rate (Recall)')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()

print(confusion_matrix(iris.target, model1.predict(iris.data)))

print(classification_report(iris.target, model1.predict(iris.data)))

from sklearn.preprocessing import label_binarize
yb0 = label_binarize(iris.target, classes=[0, 1, 2])
yb1 = label_binarize(model1.predict(iris.data), classes=[0, 1, 2])

print(yb0[:, 0].sum(), yb1[:, 0].sum())
plt.plot(yb0[:, 0], 'ro-', markersize=10, alpha=0.4, label="actual class 0")
plt.plot(yb1[:, 0], 'bs-', markersize=10, alpha=0.4, label="predicted class 0")
plt.legend()
plt.xlim(0, len(iris.target)-1);
plt.ylim(-0.1, 1.1);

print(yb0[:, 1].sum(), yb1[:, 1].sum())
plt.plot(yb0[:, 1], 'ro-', markersize=10, alpha=0.6, label="actual class 1")
plt.plot(yb1[:, 1], 'bs-', markersize=10, alpha=0.6, label="predicted class 1")
plt.legend()
plt.xlim(45, 145);
plt.ylim(-0.1, 1.1);


I am working on the file "18. 분류의 기초/04. 분류(classification) 성능 평가.ipynb".
The first task is:
ROC 커브
ROC(Receiver Operator Characteristic) 커브는 클래스 판별 기준값의 변화에 따른 Fall-out과 Recall의 변화를 시각화한 것이다.
모든 이진 분류 모형은 판별 평면으로부터의 거리에 해당하는 판별 함수(discriminant function)를 가지며 판별 함수 값이 음수이면 0인 클래스, 양수이면 1인 클래스에 해당한다고 판별한다. 즉 0 이 클래스 판별 기준값이 된다. ROC 커브는 이 클래스 판별 기준값이 달라진다면 판별 결과가 어떻게 달라지는지는 표현한 것이다.
Scikit-Learn 의 Classification 클래스는 판별 함수 값을 계산하는 decision_function 메서드를 제공한다. ROC 커브는 이 판별 함수 값을 이용하여 다음과 같이 작성한다.

모든 표본 데이터에 대해 판별 함수 값을 계산한다.
계산된 판별 함수 값을 정렬한다.
만약 0이 아닌 가장 작은 판별 함수값을 클래스 구분 기준값으로 하면 모든 표본은 클래스 1(Positive)이 된다.
   이 때의 Fall-out과 Recall을 계산하면 Recall과 Fall-out이 모두 1이된다.
두번째로 작은 판별 함수값을 클래스 구분 기준값으로 하면 판별 함수 값이 가장 작은 표본 1개를 제외하고 나머지 표본은 클래스 1(Positive)이 된다. 마찬가지로 이 때의 Fall-out과 Recall을 계산하여 기록한다.
가장 큰 판별 함수값이 클래스 구분 기준값이 될 때까지 이를 반복한다. 이 때는 모든 표본이 클래스 0(Negative)으로 판별되며 Recall과 Fall-out이 모두 0이된다.

일반적으로 클래스 판별 기준이 변화함에 따라 Recall과 Fall-out은 같이 증가하거나 감소한다. Recall이 크고 Fall-out이 작은 모형은 좋은 모형으로 생각할 수 있다.
Can you write Python code for it?

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_features=1, n_redundant=0, n_informative=1, n_clusters_per_class=1, random_state=4)
model = LogisticRegression().fit(X, y)

print(confusion_matrix(y, model.predict(X)))

print(classification_report(y, model.predict(X)))

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y, model.decision_function(X))

plt.plot(fpr, tpr)
plt.plot([0, 1], [0, 1], 'k--', label="random guess")
plt.xlabel('False Positive Rate (Fall-Out)')
plt.ylabel('True Positive Rate (Recall)')
plt.title('Receiver operating characteristic example')
plt.show()


I am working on the file "18. 분류의 기초/04. 분류(classification) 성능 평가.ipynb".
The first task is:
이진 분류 결과표 Binary Confusion Matrix
클래스가 0과 1 두 종류 밖에 없는 경우에는 일반적으로 클래스 이름을 "Positive"와 "Negative"로 표시한다.
또, 분류 모형의 예측 결과가 맞은 경우, 즉 Positive를 Positive라고 예측하거나 Negative를 Negative라고 예측한 경우에는 "True"라고 하고 예측 결과가 틀린 경우, 즉 Positive를 Negative라고 예측하거나 Negative를 Positive라고 예측한 경우에는 "False"라고 한다.
이 경우의 이진 분류 결과의 명칭과 결과표는 다음과 같다.
| | Positive라고 예측  | Negative라고 예측 |
|-|-|-|
| 실제 Positive | True Positive   |  False Negative |
| 실제 Negative | False Positive  |  True Negative  |
FDS(Fraud Detection System)의 예
FDS(Fraud Detection System)는 금융 거래, 회계 장부 등에서 잘못된 거래, 사기 거래를 찾아내는 시스템을 말한다. FDS의 예측 결과가 Positive 이면  사기 거래라고 예측한 것이고 Negative 이면 정상 거래라고 예측한 것이다. 이 결과가 사실과 일치하는지 틀리는지에 따라 다음과 같이 말한다.

True Positive: 사기를 사기라고 정확하게 예측
True Negative: 정상을 정상이라고 정확하게 예측
False Positive: 정상을 사기라고 잘못 예측
False Negative: 사기를 정상이라고 잘못 예측

|                     | 사기 거래라고 예측                | 정상 거래라고 예측                         |
| --------------------| ------------------------ | --------------------------------- |
| 실제로 사기 거래  | True Positive            |  False Negative                   |
| 실제로 정상 거래  | False Positive            |  True Negative                    |
평가 스코어
Accuracy 정확도

전체 샘플 중 맞게 출력한 샘플 수의 비율

$$\text{accuracy} = \dfrac{TP + TN}{TP + TN + FP + FN}$$
Precision 정밀도

클래스에 속한다고 출력한 샘플 중 실제로 클래스에 속하는 샘플 수의 비율
FDS의 경우, 사기 거래라고 판단한 거래 중 실제 사기 거래의 비율. 유죄율

$$\text{precision} = \dfrac{TP}{TP + FP}$$
Recall 재현율

TPR: true positive rate
실제 클래스에 속한 샘플 중에 클래스에 속한다고 출력한 샘플의 수
FDS의 경우, 실제 사기 거래 중에서 실제 사기 거래라고 예측한 거래의 비율. 검거율
sensitivity(민감도)

$$\text{recall} = \dfrac{TP}{TP + FN}$$
Fall-Out

FPR: false positive rate
실제 클래스에 속하지 않는 샘플 중에 클래스에 속한다고 출력한 샘플의 수
FDS의 경우, 실제 정상 거래 중에서 FDS가 사기 거래라고 예측한 거래의 비율, 원죄(寃罪)율

$$\text{fallout} = \dfrac{FP}{FP + TN}$$
F (beta) score

정밀도(Precision)과 재현율(Recall)의 가중 조화 평균

$$
F_\beta = (1 + \beta^2) \, ({\text{precision} \times \text{recall}}) \, / \, ({\beta^2 \, \text{precision} + \text{recall}})
$$

F1 score
beta = 1

$$
F_1 = 2 \cdot \text{precision} \cdot \text{recall} \, /...
Can you write Python code for it?

from sklearn.metrics import classification_report

y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
target_names = ['class 0', 'class 1', 'class 2']
print(classification_report(y_true, y_pred, target_names=target_names))

y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
print(classification_report(y_true, y_pred, target_names=["ant", "bird", "cat"]))


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
As in the original vignette, cells can be easily separated by their cell cycle status when cell cycle genes are used.
Now we can regress out both S score and G2M score.
Can you write Python code for it?

sc.pp.regress_out(adata, ['S_score', 'G2M_score'])
sc.pp.scale(adata)


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
Finally, we reproject dataset using cell cycle genes again. Since we regressed the scores, no effect of cell cycle is now evident.
Can you write Python code for it?

adata_cc_genes = adata[:, cell_cycle_genes]
sc.tl.pca(adata_cc_genes)
sc.pl.pca_scatter(adata_cc_genes, color='phase')


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
Here comes another difference from Seurat. The R package stores raw data, scaled data and variable genes information in separate slots, Scanpy instead keeps only one snapshot of the data. This implies that PCA is always calculated on the entire dataset. In order to calculate PCA reduction using only a subset of genes (like cell_cycle_genes), a trick should be used.
Basically we create a dummy object to store information of PCA projection, which is then reincorporated into original dataset.
Can you write Python code for it?


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
Here we define two lists, genes associated to the S phase and genes associated to the G2M phase
Can you write Python code for it?

s_genes = cell_cycle_genes[:43]
g2m_genes = cell_cycle_genes[43:]
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
Load cell cycle genes defined in Tirosh et al, 2015. It is a list of 97 genes, represented by their gene symbol. The list here is for humans, in case of alternate organism, a list of ortologues should be compiled.
There are major differences in the way Scanpy and Seurat manage data, in particular we need to filter out cell cycle genes that are not present in our dataset to avoid errors.
Can you write Python code for it?

cell_cycle_genes = [x.strip() for x in open('./data/regev_lab_cell_cycle_genes.txt')]


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
Load data
Can you write Python code for it?

adata = sc.read_csv('./data/nestorawa_forcellcycle_expressionMatrix.txt', delimiter='\t').T


I am working on the file "180209_cell_cycle/cell_cycle.ipynb".
The first task is:
Log-transformation of data and scaling should always be performed before scoring
Can you write Python code for it?


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
(Sorry that this example really isn't complicated enough to truly show the power of cProfile)
3) numba
So you found a simple function, which is run a bunch of times, and it's making your code super slower. How do you make it faster?
numba is a great approach for keeping the simplicity + readability of pure-python, while still getting the speed-up that you might get with compiled C (either native C or cython).  It's uses "just-in-time" (JIT) compiler; if you use a function repeatedly, with predictable inputs, it'll compile a version of that function for those inputs.
Can you write Python code for it?

def simple_sum_sq(size):
    image = np.random.random((size, size))

    sum_sq = 0
    for i in range(size):
        for j in range(size):
            sum_sq += image[i, j]**2

%%timeit

simple_sum_sq(1000)

import numba


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
(all I've changed in the next cell is I've added the decorator, @numba.jit())
Can you write Python code for it?

@numba.jit()
def simple_sum_sq_new(size):
    image = np.random.random((size, size))

    sum_sq = 0
    for i in range(size):
        for j in range(size):
            sum_sq += image[i, j]**2

%%timeit

simple_sum_sq_new(1000)


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
Example output:
~~~~
Timer unit: 1e-06 s
Total time: 0.060595 s
File: <ipython-input-20-d9bf5ccb538f>
Function: simple_sum_sq at line 1
Line #      Hits         Time  Per Hit   % Time  Line Contents
 1                                           def simple_sum_sq(size):
 2         1        821.0    821.0      1.4      image = np.random.random((size, size))
 3
 4         1          1.0      1.0      0.0      sum_sq = 0
 5       201         95.0      0.5      0.2      for i in range(size):
 6     40200      16937.0      0.4     28.0          for j in range(size):
 7     40000      42741.0      1.1     70.5              sum_sq += image[i, j]**2

~~~~
That's a pretty nice overview of how long it takes to run a certain line and how often that line gets run. While the image = ... line takes the longest per-run, the sum_sq += ... line gets run way more, and ends up taking up ~70% of the runtime.
I show a more complex profiling example near the bottom of this notebook ("Real World Example")
2c) cProfile
This is great if you want to see which functions are slow, but don't care where they are located within your code.
This is good for a very complex project, with lots of nested functions.  This way you know that calculate_pressure is taking up a bunch of time, even though it's located at 20 locations in your code (and at each location it's not noticably slow).
(Note: I haven't used cProfile much, so I don't know too much about it...)
Can you write Python code for it?

import cProfile

def create_mesh_grid(size):
    size=20000

    xs = range(size)
    ys = range(size)
    Y, X = np.meshgrid(xs, ys)

cProfile.run("create_mesh_grid(20000)")


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
Sample output:
~~~~
Timer unit: 1e-06 s
Total time: 8.40759 s
File: <ipython-input-34-c7b04c4d293b>
Function: process_checkpoints at line 1
Line #      Hits         Time  Per Hit   % Time  Line Contents
 1                                           def process_checkpoints(num_checkpoints):
 2         1        800.0    800.0      0.0      df = pd.DataFrame()
 3       301        863.0      2.9      0.0      for k in range(num_checkpoints):
 4
 5                                                   # Create some fake data
 6       300      27055.0     90.2      0.3          array_tmp = np.random.random((1000,7))
 7       300        851.0      2.8      0.0          array_tmp = array_tmp[1:-1] # drop cells with bad data
 8
 9
10                                                   # transform the array into a dataframe
11       300       2051.0      6.8      0.0          index     = pd.MultiIndex.from_product([[k], np.arange(array_tmp.shape[0])],
12       300     496092.0   1653.6      5.9                                                 names=["k","i"])
13       300     102367.0    341.2      1.2          df_tmp    = pd.DataFrame(array_tmp, columns=cols_in, index = index)
14
15       300      47172.0    157.2      0.6          df_tmp["Mass"]        = calculate_mass(df_tmp.Density.values,
16       300     207859.0    692.9      2.5                                                 df_tmp.dV.values)
17       300     297848.0    992.8      3.5          df_tmp["M_int"]       = df_tmp.Mass.cumsum()
18       300      37834.0    126.1      0.4          df_tmp["Temperature"] = calculate_temperature(df_tmp.Pressure.values,
19       300     196617.0    655.4      2.3                                                        df_tmp.Density.values, .67)
20       300      37470.0    124.9      0.4          df_tmp["Energy"]      =...
Can you write Python code for it?

dataframe_columns = ["Radius",
                     "dR",
                     "dV",
                     "Density",
                     "Pressure",
                     "Velocity",
                     "Z",
                     "Mass",
                     "M_int",
                     "Temperature",
                     "Energy",
                     "C_ad",
                     "Crossing_time",
                    ]

num_checkpoints=10
num_zones=20
index     = pd.MultiIndex.from_product([np.arange(num_checkpoints),
                                        np.arange(num_zones)],
                                       names=["k","i"])
df = pd.DataFrame(index=index,
                  columns=dataframe_columns,
                  dtype=float)

df

def process_checkpoints_new(num_checkpoints):
    num_zones = 1000

    dataframe_columns = ["Radius",
                         "dR",
                         "dV",
                         "Density",
                         "Pressure",
                         "Velocity",
                         "Z",
                         "Mass",
                         "M_int",
                         "Temperature",
                         "Energy",
                         "C_ad",
                         "Crossing_time",
                        ]

    index     = pd.MultiIndex.from_product([np.arange(num_checkpoints),
                                            np.arange(num_zones-2)],
                                           names=["k","i"])
    df = pd.DataFrame(index=index,
                      columns=dataframe_columns,
                      dtype=float)

    for k in range(num_checkpoints):

        # Create some fake data
        array_tmp = np.random.random((num_zones,7))
        array_tmp = array_tmp[1:-1] # drop cells with bad data

        # adding the data into the dataframe is much more complicated now
        df_tmp = df.loc[k]
        df_tmp[cols_in] = array_tmp

        df_tmp["Mass"]        =...


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
Simple example
Can you write Python code for it?

image_size = 1000
image = np.random.random((image_size, image_size))

%%timeit
sum_sq = 0
for i in range(image_size):
    for j in range(image_size):
        sum_sq += image[i, j]**2

rms = (sum_sq / (image_size**2))**.5

%%timeit
rms = (image**2).mean()**.5


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
That's pretty good. We didn't change a single bit of our code; just added a decorator, and now we have a factor of 50x speed up. Not quite 100x from vectorizing with numpy, but still pretty good given that it required less work.
In general, this is great for simple, mathematical, repetitive tasks with predictable input types. You probably can't just @numba.jit your entire code. But, hey, I've never tried.
As an added bonus, it doesn't conflict with vectorizing your code. In some cases it can even have a minor bonus from doing both:
vectorized
Can you write Python code for it?

def calculate_kinetic_energy(mass, velocity):
    E_kin = (1/2) * mass * velocity**2
    return E_kin

%%timeit

masses = np.random.random(10000)
velocities = np.random.random(masses.shape)
calculate_kinetic_energy(masses, velocities)


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
When I was just processing a single simulation, 30sec wasn't too bad, so I didn't worry about it. But then I wanted to process batches of hundreds of simulations. In order to do that I had to speed things up.
But what's the major slow down? Let's try line profiling!
Can you write Python code for it?

%lprun -f process_checkpoints process_checkpoints(300)


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
Wow! A factor of a 100 speed up! (And it's a bit easier to read.)
Note: This works great with algebra, but doesn't tend to work for things like doing calculus/numerical integration for different starting points.
numpy.vectorize exists, but it's primarily to make code pretty, rather than making it fast. It makes things look vectorized, but is actually just a for loop.
More complex example
We want the coordinates of a 2D grid (1000 x 1000). At each grid point, I want the x and y coodinate. These coordinates will be stored as 2 arrays that are each 2D (e.g. X will be 1000x1000, and will contain the x coordinate at each location.)
Can you write Python code for it?

size=1000

xs = range(size)
ys = range(size)

%%timeit

X = np.empty((size, size))
Y = np.empty((size, size))

for i, x in enumerate(xs):
    for j, y in enumerate(ys):
        X[i, j] = x
        Y[i, j] = y

%%timeit
Y, X = np.meshgrid(xs, ys)


I am working on the file "18_01_26_optimizing_python/Speeding up python code.ipynb".
The first task is:
vectorized + jit
Can you write Python code for it?

@numba.jit(nopython=True)
def calculate_kinetic_energy_jit(mass, velocity):
    E_kin = (1/2) * mass * velocity**2
    return E_kin

%%timeit

masses = np.random.random(10000)
velocities = np.random.random(masses.shape)
calculate_kinetic_energy_jit(masses, velocities)


I am working on the file "18_335/18.335_PS1.ipynb".
The first task is:
## 4c
The main result for the random walk is that the expected numer of equalizations $g_{2m}$ is proportional to the length of the walk $2m$
$$ g_{2m} \sim \sqrt{\frac{2}{\pi}} \sqrt{2m} $$
In errors accumulating from sequencial summation, the number of additions n is analagous to the length traveled $2m$ during a random walk. This result explains the mean error bound for randomly distributed values of $\varepsilon_{k}$  because $\sqrt{n}$ can be used to represent the average error away from the 0 that accumulates during n additions. This yields the result
$$ | \tilde{f}(x) - f(x) | = O( \sqrt{n}  \varepsilon_{machine} \sum_{i=1}^{n} x_i)) $$
In randomly distributed values of $\varepsilon_{k}$ positive and negative can cancel out in some cases, yeilding an overall lower bound.
Can you write Python code for it?

function my_cumsum(x)
    y = similar(x) # allocate an array of the same type and size as x
    y[1] = x[1]
    for i = 2:length(x)
        y[i] = y[i-1] + x[i]
    end
    return y
end

x = rand(Float32, 10^7) # 10^7 single-precision values uniform in [0,1)
@time y = my_cumsum(x)
yexact = my_cumsum(float64(x)) # same thing in double precision
err = abs(y - yexact) ./ abs(yexact) # relative error in y

using PyPlot
n = 1:10:length(err) # downsample by 10 for plotting speed
loglog(n, err[n])
ylabel("relative error")
xlabel("# summands")
# plot a √n line for comparison
loglog([1,length(err)], sqrt([1,length(err)]) * 1e-7, "k--")
gca()[:annotate]("~ √n", xy=(1e3,1e-5), xytext=(1e3,1e-5))
title("naive cumsum implementation")


I am working on the file "18_335/18.335_PS1.ipynb".
The first task is:
Catastrophic cancelation
"Catastrophic cancellation occurs if two numbers are close and are the result of previous calculations, so have inaccuracies in the lower digits. Most of the accurate digits may cancel as a result of subtraction, leaving behind cruft in the lower order digits. On the other hand, if the operands are exactly known rather than calculated their difference is guaranteed to have a very small relative error, which is referred to as benign cancellation" http://accu.org/index.php/journals/1558
problem 3 - part a)
$$ \text{First order Taylor series expansion:}\
0 = f(x_n) + f'(x_n)(x_{n+1} - x_n) \
x_{n+1} = \frac{f'(x_n)(x_n) - f(x_n)}{f'(x_n)} \
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}\ $$
$$ \text{Second order Taylor series expansion:}\
0 = f(x_n) + f'(x_n)(x_{n+1} - x_n) + \frac{f''(x_n)(x_{n+1} - x_n)^2}{2!} \
0 = f(x_n) + (x_{n+1} - x_n) (f'(x_n) + \frac{f''(x_n)(x_{n+1} - x_n)}{2!}) \
(x_{n+1})(f'(x_n)) + (x_{n+1})\frac{f''(x_n)(x_{n+1} - x_n)}{2!} = - f(x_n) + (x_{n})(f'(x_n)) + (x_{n})\frac{f''(x_n)(x_{n+1} - x_n)}{2!}\
(x_{n+1})(f'(x_n) + \frac{1}{2} f''(x_n)(x_{n+1} - x_n)) = - f(x_n) + (x_{n})(f'(x_n) + \frac{1}{2} f''(x_n)(x_{n+1}- x_n))\
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n) + \frac{1}{2} f''(x_n)(x_{n+1} - x_n)} \
\
\text{replace embeded } x_{n+1} \text{ with first order newton estimate: }  x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)} \
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n) + \frac{1}{2} f''(x_n))(x_n - \frac{f(x_n)}{f'(x_n)} - x_n)} \
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n) + \frac{1}{2} f''(x_n))(-\frac{f(x_n)}{f'(x_n)})} \
\text{multiply numerator and denominator by } 2f'(x_n) \
x_{n+1} = x_n - \frac{2f(x_n) f'(x_n)}{2f'(x_n)^2 -  f''(x_n)(f(x_n))} \ $$
*The last equation represents an iteration for a Newton-like root estimation
part 3b)
Analyze its asymptotic convergence rate: if
x is an exact root, write $x_n = x(1 + δn)$
as in class, and solve for δn+1 in terms of δn
assuming you are close to the root (δn << 1).
$$ 0 = f(x) $$
If an iteration of our Newtonish method takes the...
Can you write Python code for it?

function iterate_newton_basic(n,x)
    # find root for x^3−1
    # n is number of iterations
    # x is initial guess
    for i = 1:n
        x = x - (x^3 - 1)/(3x^2)
    end
    return x
end

function iterate_newton_like(n,x)
    # find root for x^3−1
    # n is number of iterations
    # x is initial guess
    for i = 1:n
        x = x - ( (6*x^5) - (6*x^2) )/( (12*x^4) + 6*x)
        #x = x - ( 2*(x^3 -1) * (3x^2) )/( 2* (3*x^2)^2 - 6x*(x^3 - 1))
    end
    return x
end

x1 = 2
n = 2
xn_basic = iterate_newton_basic(n,x1)
xn_like = iterate_newton_like(n,x1)

err = xn_like-1
xn_like-xn_basic

### To do:
# represent with arbitrary percision
# aproximate number of digits with:
log10(xn_like - 1)
#make table

xn_basic,xn_basic^3

xn_like,xn_like


I am working on the file "18_335/18.335_PS1.ipynb".
The first task is:
The poor relative error occurs when the b^2 is subtracted from c. Because b >> c, many of the accurate digits are lost when this subtraction occurs.
Part 2b
To correct for this, multiply both the numerator and denominator by by $-b + \sqrt{b^2 - c}$
$$ x= -b - \sqrt{b^2 - c} * \frac{-b + \sqrt{b^2 - c}}{-b + \sqrt{b^2 - c}} \
= \frac{b^2 - (b^2 - c)}{b + \sqrt{b^2 - c}} \
= \frac{c}{b + \sqrt{b^2 - c}}$$
Can you write Python code for it?

b =1
c = logspace(-20,-1,50)
length(c)
#x = b - sqrt(b^2 - c)
x = c ./ (b + sqrt(b^2 - c))
#x = b - (b^2 - c) ./ sqrt(b^2 - c)
# multiply by b^2
set_bigfloat_precision(256)
xbig = b - sqrt(b^2 - big(c))
err = float64(abs(x-xbig) ./ abs(xbig))

using PyPlot
loglog(c, err, "bo-")
xlabel(L"c")
ylabel(L"relative error in $x$")
title(L"Alternative calculation of root: $\frac{c}{b + \sqrt{b^2 - c}}$")

# individual values

# when the value of c is very small (~10^-20), the difference (b^2 - c)
# is evaluated to (b^2 - c) = 1.0

# when subtracting 1.0 from a very small number, the gap is too big so the
# difference is just reported as 1.0

c = 1e-20
b =1
length(c)
#x = b - sqrt(b^2 - c)
x = c ./ (-b + sqrt(b^2 - c))
set_bigfloat_precision(256)
xbig = b - sqrt(b^2 - big(c))
err = float64(abs(x-xbig) ./ abs(xbig))
x,xbig,err

-b + (b^2 - c) ./ sqrt(b^2 - c)


I am working on the file "18_335/18.335_PS1.ipynb".
The first task is:
part b) to resolve the relative error - try making the gap based on e_machine
Can you write Python code for it?

c=1e-20

sqr_diff = b^2 - c
if c != b^2 && sqr_diff == 1.0
    sqr_diff = 1-eps()
end
x = b - sqrt(sqr_diff)
set_bigfloat_precision(256)
xbig = b - sqrt(b^2 - big(c))
err = float64(abs(x-xbig) ./ abs(xbig))
x,xbig,err

x^2 + 2x + c = 0
x(x+2) + c = 0
x(x+2) = -c

(x - rt(c)) (x - rt(c))


I am working on the file "18_335/hogstrom_PS1_18_335_2015Feb27.ipynb".
The first task is:
Problem 2a
Using the code from the pset1 Julia note- book posted on the web page, plot the accuracy of x_ = b−√(b2 −c) for b = 1 and for a range of c values from 10−1 to 10−20. Explain the observed inaccuracy.
Can you write Python code for it?

b =1
c = logspace(-20,-1,50)
length(c)
x = b - sqrt(b^2 - c)
# multiply by b^2
set_bigfloat_precision(256)
xbig = b - sqrt(b^2 - big(c))
err = float64(abs(x-xbig) ./ abs(xbig))

using PyPlot
loglog(c, err, "bo-")
xlabel(L"c")
ylabel(L"relative error in $x$")
title(L"standard quadratic formula $x = b - \sqrt{b^2 - c}$")


I am working on the file "18_335/hogstrom_PS1_18_335_2015Feb27.ipynb".
The first task is:
numbers in the form $2^{53} + 2i$ will be in $\mathbb{F}$ for small values of i
Can you write Python code for it?


I am working on the file "18_335/hogstrom_PS1_18_335_2015Feb27.ipynb".
The first task is:
show that div2sum_performance is faster than div2sum
Can you write Python code for it?

## compare speed
x = rand(Float32, 2^25)
@time div2sum(x)
@time div2sum_performance(x)
@time sum(x)

# show that they all give the same result
div2sum(x), div2sum_performance(x), sum(x)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Convert the test-set to a TFRecords-file:
Can you write Python code for it?

convert(image_paths=image_paths_test,
        labels=cls_test,
        out_path=path_tfrecords_test)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Create TFRecords
TFRecords is the binary file-format used internally in TensorFlow which allows for high-performance reading and processing of datasets.
For this small dataset we will just create one TFRecords file for the training-set and another for the test-set. But if your dataset is very large then you can split it into several TFRecords files called shards. This will also improve the random shuffling, because the Dataset API only shuffles from a smaller buffer of e.g. 1024 elements loaded into RAM. So if you have e.g. 100 TFRecords files, then the randomization will be much better than for a single TFRecords file.
File-path for the TFRecords file holding the training-set.
Can you write Python code for it?

path_tfrecords_train = os.path.join(knifey.data_dir, "train.tfrecords")
path_tfrecords_train


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Create an Instance of the Estimator
We can specify hyper-parameters e.g. for the learning-rate of the optimizer.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Evaluation
Once the model has been trained, we can evaluate its performance on the test-set.
Can you write Python code for it?

result = model.evaluate(input_fn=test_input_fn)

result

print("Classification accuracy: {0:.2%}".format(result["accuracy"]))


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
File-path for the TFRecords file holding the test-set.
Can you write Python code for it?

path_tfrecords_test = os.path.join(knifey.data_dir, "test.tfrecords")
path_tfrecords_test


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Get the class-names.
Can you write Python code for it?

class_names = dataset.class_names
class_names


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Get the test-set.
Can you write Python code for it?

image_paths_test, cls_test, labels_test = dataset.get_test_set()


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Helper-function for creating an input-function that reads from TFRecords files for use with the Estimator API.
Can you write Python code for it?

def input_fn(filenames, train, batch_size=32, buffer_size=2048):
    # Args:
    # filenames:   Filenames for the TFRecords files.
    # train:       Boolean whether training (True) or testing (False).
    # batch_size:  Return batches of this size.
    # buffer_size: Read buffers of this size. The random shuffling
    #              is done on the buffer, so it must be big enough.

    # Create a TensorFlow Dataset-object which has functionality
    # for reading and shuffling data from TFRecords files.
    dataset = tf.data.TFRecordDataset(filenames=filenames)

    # Parse the serialized data in the TFRecords files.
    # This returns TensorFlow tensors for the image and labels.
    dataset = dataset.map(parse)

    if train:
        # If training then read a buffer of the given size and
        # randomly shuffle it.
        dataset = dataset.shuffle(buffer_size=buffer_size)

        # Allow infinite reading of the data.
        num_repeat = None
    else:
        # If testing then don't shuffle the data.

        # Only go through the data once.
        num_repeat = 1

    # Repeat the dataset the given number of times.
    dataset = dataset.repeat(num_repeat)

    # Get a batch of data with the given size.
    dataset = dataset.batch(batch_size)

    # Create an iterator for the dataset and the above modifications.
    iterator = dataset.make_one_shot_iterator()

    # Get the next batch of images and labels.
    images_batch, labels_batch = iterator.get_next()

    # The input-function must return a dict wrapping the images.
    x = {'image': images_batch}
    y = labels_batch

    return x, y


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Helper-function for loading images
This dataset does not load the actual images, instead it has a list of the images in the training-set and another list for the images in the test-set. This helper-function loads some image-files.
Can you write Python code for it?

def load_images(image_paths):
    # Load the images from disk.
    images = [imread(path) for path in image_paths]

    # Convert to a numpy array and return it.
    return np.asarray(images)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None, smooth=True):

    assert len(images) == len(cls_true)

    # Create figure with sub-plots.
    fig, axes = plt.subplots(3, 3)

    # Adjust vertical spacing.
    if cls_pred is None:
        hspace = 0.3
    else:
        hspace = 0.6
    fig.subplots_adjust(hspace=hspace, wspace=0.3)

    # Interpolation type.
    if smooth:
        interpolation = 'spline16'
    else:
        interpolation = 'nearest'

    for i, ax in enumerate(axes.flat):
        # There may be less than 9 images, ensure it doesn't crash.
        if i < len(images):
            # Plot image.
            ax.imshow(images[i],
                      interpolation=interpolation)

            # Name of the true class.
            cls_true_name = class_names[cls_true[i]]

            # Show true and predicted classes.
            if cls_pred is None:
                xlabel = "True: {0}".format(cls_true_name)
            else:
                # Name of the predicted class.
                cls_pred_name = class_names[cls_pred[i]]

                xlabel = "True: {0}\nPred: {1}".format(cls_true_name,
                                                       cls_pred_name)

            # Show the classes as the label on the x-axis.
            ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Helper-function for printing the conversion progress.
Can you write Python code for it?

def print_progress(count, total):
    # Percentage completion.
    pct_complete = float(count) / total

    # Status-message.
    # Note the \r which means the line should overwrite itself.
    msg = "\r- Progress: {0:.1%}".format(pct_complete)

    # Print it.
    sys.stdout.write(msg)
    sys.stdout.flush()


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Helper-function for wrapping an integer so it can be saved to the TFRecords file.
Can you write Python code for it?

def wrap_int64(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Helper-function for wrapping raw bytes so they can be saved to the TFRecords file.
Can you write Python code for it?

def wrap_bytes(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
In this example we want to use a 3-layer DNN with 512, 256 and 128 units respectively.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Input Function for Predicting on New Images
An input-function is also needed for predicting the class of new data. As an example we just use a few images from the test-set.
You could load any images you want here. Make sure they are the same dimensions as expected by the TensorFlow model, otherwise you need to resize the images.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Input Functions for the Estimator
The TFRecords files contain the data in a serialized binary format which needs to be converted back to images and labels of the correct data-type. We use a helper-function for this parsing:
Can you write Python code for it?

def parse(serialized):
    # Define a dict with the data-names and types we expect to
    # find in the TFRecords file.
    # It is a bit awkward that this needs to be specified again,
    # because it could have been written in the header of the
    # TFRecords file instead.
    features = \
        {
            'image': tf.FixedLenFeature([], tf.string),
            'label': tf.FixedLenFeature([], tf.int64)
        }

    # Parse the serialized data so we get a dict with our data.
    parsed_example = tf.parse_single_example(serialized=serialized,
                                             features=features)

    # Get the image as raw bytes.
    image_raw = parsed_example['image']

    # Decode the raw bytes so it becomes a tensor with type.
    image = tf.decode_raw(image_raw, tf.uint8)

    # The type is now uint8 but we need it to be float.
    image = tf.cast(image, tf.float32)

    # Get the label associated with the image.
    label = parsed_example['label']

    # The image and label are now correct TensorFlow types.
    return image, label


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Load Data
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
New Estimator
If you cannot use one of the built-in Estimators, then you can create an arbitrary TensorFlow model yourself. To do this, you first need to create a function which defines the following:

The TensorFlow model, e.g. a Convolutional Neural Network.
The output of the model.
The loss-function used to improve the model during optimization.
The optimization method.
Performance metrics.

The Estimator can be run in three modes: Training, Evaluation, or Prediction. The code is mostly the same, but in Prediction-mode we do not need to setup the loss-function and optimizer.
This is another aspect of the Estimator API that is poorly designed and resembles how we did ANSI C programming using structs in the old days. It would probably have been more elegant to split this into several functions and sub-classed the Estimator-class.
Can you write Python code for it?

def model_fn(features, labels, mode, params):
    # Args:
    #
    # features: This is the x-arg from the input_fn.
    # labels:   This is the y-arg from the input_fn.
    # mode:     Either TRAIN, EVAL, or PREDICT
    # params:   User-defined hyper-parameters, e.g. learning-rate.

    # Reference to the tensor named "image" in the input-function.
    x = features["image"]

    # The convolutional layers expect 4-rank tensors
    # but x is a 2-rank tensor, so reshape it.
    net = tf.reshape(x, [-1, img_size, img_size, num_channels])

    # First convolutional layer.
    net = tf.layers.conv2d(inputs=net, name='layer_conv1',
                           filters=32, kernel_size=3,
                           padding='same', activation=tf.nn.relu)
    net = tf.layers.max_pooling2d(inputs=net, pool_size=2, strides=2)

    # Second convolutional layer.
    net = tf.layers.conv2d(inputs=net, name='layer_conv2',
                           filters=32, kernel_size=3,
                           padding='same', activation=tf.nn.relu)
    net = tf.layers.max_pooling2d(inputs=net, pool_size=2, strides=2)

    # Flatten to a 2-rank tensor.
    net = tf.contrib.layers.flatten(net)
    # Eventually this should be replaced with:
    # net = tf.layers.flatten(net)

    # First fully-connected / dense layer.
    # This uses the ReLU activation function.
    net = tf.layers.dense(inputs=net, name='layer_fc1',
                          units=128, activation=tf.nn.relu)

    # Second fully-connected / dense layer.
    # This is the last layer so it does not use an activation function.
    net = tf.layers.dense(inputs=net, name='layer_fc_2',
                          units=num_classes)

    # Logits output of the neural network.
    logits = net

    # Softmax output of the neural network.
    y_pred = tf.nn.softmax(logits=logits)

    # Classification output of the neural network.
    y_pred_cls = tf.argmax(y_pred, axis=1)

    if mode == tf.estimator.ModeKeys.PREDICT:
        # If the estimator is supposed to...


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Note the 4 function calls required to write the data-dict to the TFRecords file. In the original code-example from the Google Developers, these 4 function calls were actually nested. The design-philosophy for TensorFlow generally seems to be: If one function call is good, then 4 function calls are 4 times as good, and if they are nested then it is exponential goodness!
Of course, this is quite poor API design because the last function writer.write() should just be able to take the data-dict directly and then call the 3 other functions internally.
Convert the training-set to a TFRecords-file. Note how we use the integer class-numbers as the labels instead of the One-Hot encoded arrays.
Can you write Python code for it?

convert(image_paths=image_paths_train,
        labels=cls_train,
        out_path=path_tfrecords_train)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Load the first images from the test-set.
images = load_images(image_paths=image_paths_test[0:9])

# Get the true classes for those images.
cls_true = cls_test[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true, smooth=True)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Pre-Made / Canned Estimator
When using a pre-made Estimator, we need to specify the input features for the data. In this case we want to input images from our data-set which are numeric arrays of the given shape.
Can you write Python code for it?

feature_image = tf.feature_column.numeric_column("image",
                                                 shape=img_shape)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Predictions
The model can also be used to make predictions on new data.
Can you write Python code for it?

predictions = model.predict(input_fn=predict_input_fn)

cls_pred = np.array(list(predictions))
cls_pred

plot_images(images=some_images,
            cls_true=some_images_cls,
            cls_pred=cls_pred)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Predictions for the Entire Test-Set
It appears that the model maybe classifies all images as 'spoony'. So let us see the predictions for the entire test-set. We can do this simply by using its input-function:
Can you write Python code for it?

predictions = model.predict(input_fn=test_input_fn)

cls = [p['classes'] for p in predictions]

cls_pred = np.array(cls, dtype='int').squeeze()


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Predictions for the Entire Test-Set
To get the predicted classes for the entire test-set, we just use its input-function:
Can you write Python code for it?

predictions = model.predict(input_fn=test_input_fn)

cls_pred = np.array(list(predictions))
cls_pred


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Print the first image-path to see if it looks OK.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Set the directory for storing the data-set on your computer.
Can you write Python code for it?

# knifey.data_dir = "data/knifey-spoony/"


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The Convolutional Neural Network predicts different classes for the images, although most have just been classified as 0 (forky), so the accuracy is horrible.
Can you write Python code for it?

np.sum(cls_pred == 0)

np.sum(cls_pred == 1)

np.sum(cls_pred == 2)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The DNNClassifier then constructs the neural network for us. We can also specify the activation function and various other parameters (see the docs). Here we just specify the number of classes and the directory where the checkpoints will be saved.
Can you write Python code for it?

model = tf.estimator.DNNClassifier(feature_columns=feature_columns,
                                   hidden_units=num_hidden_units,
                                   activation_fn=tf.nn.relu,
                                   n_classes=num_classes,
                                   model_dir="./checkpoints_tutorial18-1/")


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The Knifey-Spoony data-set has now been loaded and consists of 4700 images and associated labels (i.e. classifications of the images). The data-set is split into 2 mutually exclusive sub-sets, the training-set and the test-set.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(image_paths_train)))
print("- Test-set:\t\t{}".format(len(image_paths_test)))


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The Knifey-Spoony data-set is about 22 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The class-numbers are actually not used in the input-function as it is not needed for prediction. However, the true class-number is needed when we plot the images further below.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The data dimensions have already been defined in the knifey module, so we just need to import the ones we need.
Can you write Python code for it?

from knifey import img_size, img_size_flat, img_shape, num_classes, num_channels


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
The test-set contains 530 images in total and they have all been predicted as class 2 (spoony). So this model does not work at all for classifying the Knifey-Spoony dataset.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
These images are now stored as numpy arrays in memory, so we can use the standard input-function for the Estimator API. Note that the images are loaded as uint8 data but it must be input to the TensorFlow graph as floats so we do a type-cast.
Can you write Python code for it?

predict_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"image": some_images.astype(np.float32)},
    num_epochs=1,
    shuffle=False)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
This is the function for reading images from disk and writing them along with the class-labels to a TFRecords file. This loads and decodes the images to numpy-arrays and then stores the raw bytes in the TFRecords file. If the original image-files are compressed e.g. as jpeg-files, then the TFRecords file may be many times larger than the original image-files.
It is also possible to save the compressed image files directly in the TFRecords file because it can hold any raw bytes. We would then have to decode the compressed images when the TFRecords file is being read later in the parse() function below.
Can you write Python code for it?

def convert(image_paths, labels, out_path):
    # Args:
    # image_paths   List of file-paths for the images.
    # labels        Class-labels for the images.
    # out_path      File-path for the TFRecords output file.

    print("Converting: " + out_path)

    # Number of images. Used when printing the progress.
    num_images = len(image_paths)

    # Open a TFRecordWriter for the output-file.
    with tf.python_io.TFRecordWriter(out_path) as writer:

        # Iterate over all the image-paths and class-labels.
        for i, (path, label) in enumerate(zip(image_paths, labels)):
            # Print the percentage-progress.
            print_progress(count=i, total=num_images-1)

            # Load the image-file using matplotlib's imread function.
            img = imread(path)

            # Convert the image to raw bytes.
            img_bytes = img.tostring()

            # Create a dict with the data we want to save in the
            # TFRecords file. You can add more relevant data here.
            data = \
                {
                    'image': wrap_bytes(img_bytes),
                    'label': wrap_int64(label)
                }

            # Wrap the data as TensorFlow Features.
            feature = tf.train.Features(feature=data)

            # Wrap again as a TensorFlow Example.
            example = tf.train.Example(features=feature)

            # Serialize the data.
            serialized = example.SerializeToString()

            # Write the serialized data to the TFRecords file.
            writer.write(serialized)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
This is the input-function for the test-set for use with the Estimator API:
Can you write Python code for it?

def test_input_fn():
    return input_fn(filenames=path_tfrecords_test, train=False)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
This is the input-function for the training-set for use with the Estimator API:
Can you write Python code for it?

def train_input_fn():
    return input_fn(filenames=path_tfrecords_train, train=True)


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
This was developed using Python 3.6 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Training
Now that our new Estimator has been created, we can train it.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Training
We can now train the model for a given number of iterations. This automatically loads and saves checkpoints so we can continue the training later.
Can you write Python code for it?


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
Training and Test-Sets
This function returns the file-paths for the images, the class-numbers as integers, and the class-numbers as One-Hot encoded arrays called labels.
In this tutorial we will actually use the integer class-numbers and call them labels. This may be a little confusing but you can always add print-statements to see what the data actually is.
Can you write Python code for it?

image_paths_train, cls_train, labels_train = dataset.get_training_set()


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
We can then create an instance of the new Estimator.
Note that we don't provide feature-columns here as it is inferred automatically from the data-functions when model_fn() is called.
It is unclear from the TensorFlow documentation why it is necessary to specify the feature-columns when using DNNClassifier in the example above, when it is not needed here.
Can you write Python code for it?

model = tf.estimator.Estimator(model_fn=model_fn,
                               params=params,
                               model_dir="./checkpoints_tutorial18-2/")


I am working on the file "18_TFRecords_Dataset_API.ipynb".
The first task is:
You can have several input features which would then be combined in a list:
Can you write Python code for it?


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
@TODO: Check error
Alternative formats: COO and CSR formats
Take a look at the slides that we just started in the last class, which cover the basics of sparse matrix storage formats: link
Although these are available as native formats in SciPy, let's create native Python versions first using lists. We can then compare the performance of, say, sparse matrix-vector multiply, against the ones we ran above.
Exercise. Create a COO-like data structure. You may use the edges and name2id raw data structures created above. Name your arrays, coo_rows, coo_cols, and coo_vals.
Can you write Python code for it?

coo_rows = [name2id[e] for e in edges['Source']]
coo_cols = [name2id[e] for e in edges['Target']]
coo_vals = [1.] * len (coo_rows)


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
Exercise. Implement a sparse matrix-vector multiply that works when the matrix is A_named_keys. How much faster or slower is it than cse6040.spmv()?

Hint: Feel free to take a look at cse6040.spmv().
Can you write Python code for it?

def spmv_named_keys (n, A, x, name2id):
    y = cse6040.dense_vector (n)

    # @YOUSE: Complete this routine
    pass

    return y

# Measures the speed of your implementation:
%timeit spmv_named_keys (n, A_named_keys, x, name2id)


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
Exercise. Now implement a CSR-based sparse matrix-vector multiply.
Can you write Python code for it?

def csr_spmv (n, ptr, ind, val, x):
    assert n > 0
    assert type (ptr) == list
    assert type (ind) == list
    assert type (val) == list
    assert type (x) == list
    assert len (ptr) >= (n+1)  # Why?
    assert len (ind) >= ptr[n]  # Why?
    assert len (val) >= ptr[n]  # Why?

    y = cse6040.dense_vector (n)

    # @YOUSE: Insert your implementation here
    pass

    return y

%timeit csr_spmv (n, csr_ptrs, csr_inds, csr_vals, x)


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
Exercise. What does the following code do?
Can you write Python code for it?

id2name = {}
name2id = {}
for (k, v) in enumerate (V_names):
    if k <= 5: print ("[%d] %s" % (k, v)) # for debugging
    id2name[k] = v
    name2id[v] = k

A_named_keys = cse6040.sparse_matrix ()
A_numbered_keys = cse6040.sparse_matrix ()

for (k, row) in edges.iterrows ():
    ni = row['Source']
    nj = row['Target']
    A_named_keys[ni][nj] = 1.
    A_named_keys[nj][ni] = 1.

    i = name2id[ni]
    j = name2id[nj]
    A_numbered_keys[i][j] = 1.
    A_numbered_keys[j][i] = 1.


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
Exercise. What does this code do?
Can you write Python code for it?

V_names = set (edges.Source)
V_names.update (set (edges.Target))

m = len (edges)
n = len (V_names)
print ("==> |V| == %d, |E| == %d" % (n, m))


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
Per the notes, here is how we can convert our COO representation from before into a SciPy implementation.
Can you write Python code for it?

A_coo = sp.coo_matrix ((vals, (rows, cols)))


I am working on the file "19--sparse-matrix.ipynb".
The first task is:
Sparse matrix storage: Baseline methods
Let's start by reminding ourselves how our previous method for storing sparse matrices, based on nested default dictionaries, works and performs.
Can you write Python code for it?


I am working on the file "19-04-10-ml-workcamp/credit_data_work-10.ipynb".
The first task is:

Schritt: Einlesen der Daten als Panda Dataframe

Die Daten liegen im directory /data

Zum Einlesen von csv Daten wird der Befehl pd.read_csv("") verwendet

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

Sollen Daten aus dem gleichen directory eingelesen werden dann:

pd.read_csv("credit_data.csv")

sonst muss der relative Pfad angegeben werden:
pd.read_csv("./data/credit_data.csv")

Achtung Linux: statt / bitte \ verwenden.
Geben Sie in die nächste Zelle den Code:

training = pd.read_csv("./data/credit_data.csv")

ein und führen Sie die Zelle aus
Can you write Python code for it?

# hier Ihren Code einfügen und ausführen

# Wenn Sie keine Fehlermeldung erhalten haben, wurde die Datei erfolgreich geladen
# die Daten wurden dann in die Variable training als panda Dataframe geladen


I am working on the file "19-04-10-ml-workcamp/credit_data_work-10.ipynb".
The first task is:

Schritt: Exploration der Daten

Zu allererst sollte man sich die Daten einmal anschauen

shape, head(), tail(), describe() sind gute Möglichkeiten, sich einen Überblick zu verschaffen

Geben Sie in die nächste Zelle den Code:

training.shape

ein und führen Sie die Zelle aus
Can you write Python code for it?

# hier Ihren Code einfügen und ausführen


I am working on the file "19-04-10-ml-workcamp/credit_data_work-10.ipynb".
The first task is:
Geben Sie in die nächste Zelle den Code:

training.tail()

ein und führen Sie die Zelle aus
Can you write Python code for it?

# hier Ihren Code einfügen und ausführen

# Erkennen Sie den Unterschied zu training.head() ? <br>
# Geben Sie einfach in training.head() oder training.tail() mal eine Zahl ein. zB. traing.tail(25)


I am working on the file "19-04-10-ml-workcamp/credit_data_work-10.ipynb".
The first task is:
Warum sollte warnings importiert werden ?

https://docs.python.org/3/library/warnings.html

Fügen Sie in die nächste Zelle den Code:

import warnings

warnings.filterwarnings("ignore")

ein.
Can you write Python code for it?


I am working on the file "19-04-10-ml-workcamp/credit_data_work-10.ipynb".
The first task is:
Zum Lesen eines directories benötigen wir die Bibliothek os

mit

import os

wird die Bibliothek geladen

os.listdir("./data) liest dann das directory data aus

Das data directory muss ein Unterordner des aktuellen directories sein
z.B.:

/aktuell

/aktuell/data

Geben Sie in die nächste Zelle den Code:

import os

print(os.listdir("./data"))

ein.
Can you write Python code for it?

# hier Ihren Code einfügen und ausführen
import os
print(os.listdir("./data"))


I am working on the file "19-04-10-ml-workcamp/credit_data_work-10.ipynb".
The first task is:
https://seaborn.pydata.org/tutorial/aesthetics.html

Es gibt fünf voreingestellte Seaborn-Themen: darkgrid, whitegrid, dark, white, and ticks

Fügen Sie in folgende Zelle den Code

sns.set_style("whitegrid")
ein
Can you write Python code for it?


I am working on the file "19. 문서 전처리/01. Python 문자열 인코딩.ipynb".
The first task is:
str에 encode 메소드를 적용:
내부적으로 유니코드로 변환 시도
unicode에 decode 메소드를 적용:
바이트열이 스트링이라고 가정해 버린다.

디폴트 인코딩
Can you write Python code for it?

u"가".encode("utf-8"), u"가".encode("cp949"), "가"

import sys
print(sys.getdefaultencoding())
print(sys.stdin.encoding)
print(sys.stdout.encoding)
import locale
print(locale.getpreferredencoding())


I am working on the file "19. 문서 전처리/01. Python 문자열 인코딩.ipynb".
The first task is:
str에 encode 메소드를 적용하면 또는 unicode에 decode 메소드를 적용하면?
Can you write Python code for it?

"가".encode("utf-8")

unicode("가", "ascii").encode("utf-8")

u"가".decode("utf-8")

u"가".encode("ascii").decode("utf-8")


I am working on the file "19. 문서 전처리/01. Python 문자열 인코딩.ipynb".
The first task is:
유니코드 리터럴(Literal)

따옴표 앞에 u자를 붙이면 unicode 문자열로 인식
내부적으로 유니코드 포인트로 저장
Can you write Python code for it?

y = u"가"
y

print(y)

y = u"가나다"
print(y[0], y[1], y[2])


I am working on the file "19. 문서 전처리/02. NLTK 자연어 처리 패키지 소개.ipynb".
The first task is:
POS tagging
POS(part-of-speech)는 품사를 말한다.

Part-of-Speech Tagset
https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.htm
http://www.ibm.com/support/knowledgecenter/ko/SS5RWK_3.5.0/com.ibm.discovery.es.ta.doc/iiysspostagset.htm
Can you write Python code for it?

from nltk.tag import pos_tag
tagged_list = pos_tag(word_tokenize(emma_raw[:100]))
tagged_list

from nltk.tag import untag
untag(tagged_list)


I am working on the file "19. 문서 전처리/02. NLTK 자연어 처리 패키지 소개.ipynb".
The first task is:
형태소 분석
형태소 분석이란 어근, 접두사/접미사, 품사(POS, part-of-speech) 등 다양한 언어적 속성의 구조를 파악하는 작업이다. 구체적으로는 다음과 같은 작업으로 나뉜다.

stemming (어근 추출)
lemmatizing (원형 복원)
POS tagging (품사 태깅)

### Stemming and lemmatizing
Can you write Python code for it?

from nltk.stem import PorterStemmer
st = PorterStemmer()
st.stem("eating")

from nltk.stem import LancasterStemmer
st = LancasterStemmer()
st.stem("shopping")

from nltk.stem import RegexpStemmer
st = RegexpStemmer("ing")
st.stem("cooking")

from nltk.stem import WordNetLemmatizer
lm = WordNetLemmatizer()
print(lm.lemmatize("cooking"))
print(lm.lemmatize("cooking", pos="v"))
print(lm.lemmatize("cookbooks"))

print(WordNetLemmatizer().lemmatize("believes"))
print(LancasterStemmer().stem("believes"))


I am working on the file "19. 문서 전처리/03. konlpy 한국어 처리 패키지 소개.ipynb".
The first task is:
한국어 처리 유틸리티
Can you write Python code for it?

x = [u"한글", {u"한글 키": [u"한글 밸류1", u"한글 밸류2"]}]
print(x)

from konlpy.utils import pprint
pprint(x)

from konlpy.utils import concordance
idx = concordance(u'대한민국', c, show=True)

idx


I am working on the file "19. 문서 전처리/03. konlpy 한국어 처리 패키지 소개.ipynb".
The first task is:
형태소 분석
konlpy는 tag 서브패키지에서 형태소 분석을 위한 5개의 클래스를 제공한다.

Kkma
Hannanum
Twitter
Komoran
Mecab

이 클래스는 다음과 같은 메서드를 대부분 제공한다.

morphs : 형태소 추출
nouns : 명사 추출
pos : pos 태깅
Can you write Python code for it?

from konlpy.tag import *
hannanum = Hannanum()
kkma = Kkma()
twitter = Twitter()


I am working on the file "19. 문서 전처리/03. konlpy 한국어 처리 패키지 소개.ipynb".
The first task is:
형태소 추출
Can you write Python code for it?

pprint(hannanum.morphs(c[:65]))

pprint(kkma.morphs(c[:65]))

pprint(twitter.morphs(c[:65]))


I am working on the file "19. 문서 전처리/04. 문서 전처리.ipynb".
The first task is:
Hashing Trick
CountVectorizer는 모든 작업을 in-memory 상에서 수행하므로 데이터 양이 커지면 속도가 느려지거나 실행이 불가능해진다. 이 때
HashingVectorizer를 사용하면 Hashing Trick을 사용하여 메모리 및 실행 시간을 줄일 수 있다.
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
twenty = fetch_20newsgroups()
len(twenty.data)

%time CountVectorizer().fit(twenty.data).transform(twenty.data)

from sklearn.feature_extraction.text import HashingVectorizer
hv = HashingVectorizer(n_features=10)

%time hv.transform(twenty.data)


I am working on the file "19. 문서 전처리/04. 문서 전처리.ipynb".
The first task is:
TF-IDF
TF-IDF(Term Frequency – Inverse Document Frequency) 인코딩은 단어를 갯수 그대로 카운트하지 않고 모든 문서에 공통적으로 들어있는 단어의 경우 문서 구별 능력이 떨어진다고 보아 가중치를 축소하는 방법이다.
구제적으로는 문서 $d$(document)와 단어 $t$ 에 대해 다음과 같이 계산한다.
$$ \text{tf-idf}(d, t) = \text{tf}(d, t) \cdot \text{idf}(d, t) $$
여기에서

$\text{tf}(d, t)$: 단어의 빈도수
$\text{idf}(d, t)$ : inverse document frequency

$$ \text{idf}(d, t) = \log \dfrac{n_d}{1 + \text{df}(t)} $$

$n_d$ : 전체 문서의 수
$\text{df}(t)$:  단어 $t$를 가진 문서의 수
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

tfidv = TfidfVectorizer().fit(corpus)
tfidv.transform(corpus).toarray()


I am working on the file "19. 문서 전처리/04. 문서 전처리.ipynb".
The first task is:
n-그램
n-그램은 단어장 생성에 사용할 토큰의 크기를 결정한다. 1-그램은 토큰 하나만 단어로 사용하며 2-그램은 두 개의 연결된 토큰을 하나의 단어로 사용한다.
Can you write Python code for it?

vect = CountVectorizer(ngram_range=(2,2)).fit(corpus)
vect.vocabulary_

vect = CountVectorizer(ngram_range=(1,2), token_pattern="t\w+").fit(corpus)
vect.vocabulary_


I am working on the file "19. 문서 전처리/04. 문서 전처리.ipynb".
The first task is:
빈도수
max_df, min_df 인수를 사용하여 문서에서 토큰이 나타난 횟수를 기준으로 단어장을 구성할 수도 있다. 토큰의 빈도가 max_df로 지정한 값을 초과 하거나 min_df로 지정한 값보다 작은 경우에는 무시한다. 인수 값은 정수인 경우 횟수, 부동소수점인 경우 비중을 뜻한다.
Can you write Python code for it?

vect = CountVectorizer(max_df=4, min_df=2).fit(corpus)
vect.vocabulary_, vect.stop_words_

vect.transform(corpus).toarray().sum(axis=0)


I am working on the file "1995 RDIA.ipynb".
The first task is:
Run the Agent! Start the exploration!
Can you write Python code for it?

MyAgent = RDIA(n_states = n_states, n_actions = n_actions)
MyAgent.learner_life(transition_M=P)

# compare if the algorithm was successgul
print '_______original______________'
print
print P
print '_____________________________'
print
print
print '_______reconstruction________'
print
print MyAgent.P_experimental
print '_____________________________'


I am working on the file "19_Hyper-Parameters.ipynb".
The first task is:
Examples of Mis-Classified Images
We can plot some examples of mis-classified images from the test-set.
First we get the predicted classes for all the images in the test-set:
Can you write Python code for it?


I am working on the file "19_Hyper-Parameters.ipynb".
The first task is:
Get the predicted classes as One-Hot encoded arrays.
Can you write Python code for it?


I am working on the file "19_Hyper-Parameters.ipynb".
The first task is:
Helper-function for log-dir-name
We will log the training-progress for all parameter-combinations so they can be viewed and compared using TensorBoard. This is done by setting a common parent-dir and then have a sub-dir for each parameter-combination with an appropriate name.
Can you write Python code for it?

def log_dir_name(learning_rate, num_dense_layers,
                 num_dense_nodes, activation):

    # The dir-name for the TensorBoard log-dir.
    s = "./19_logs/lr_{0:.0e}_layers_{1}_nodes_{2}_{3}/"

    # Insert all the hyper-parameters in the dir-name.
    log_dir = s.format(learning_rate,
                       num_dense_layers,
                       num_dense_nodes,
                       activation)

    return log_dir


I am working on the file "19_Hyper-Parameters.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "19_Hyper-Parameters.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred):
    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Boolean array whether the predicted class is incorrect.
    incorrect = (cls_pred != data.y_test_cls)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.x_test[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.y_test_cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])