I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Configure training
The trainer strives to reduce the loss function by different optimization approaches, Stochastic Gradient Descent (sgd) being a basic one. Typically, one would start with random initialization of the model parameters. The sgd optimizer would calculate the loss or error between the predicted label against the corresponding ground-truth label and using gradient-decent generate a new set model parameters in a single iteration.
The aforementioned model parameter update using a single observation at a time is attractive since it does not require the entire data set (all observation) to be loaded in memory and also requires gradient computation over fewer datapoints, thus allowing for training on large data sets. However, the updates generated using a single observation sample at a time can vary wildly between iterations. An intermediate ground is to load a small set of observations and use an average of the loss or error from that set to update the model parameters. This subset is called a minibatch.
With minibatches we often sample observation from the larger training dataset. We repeat the process of model parameters update using different combination of training samples and over a period of time minimize the loss (and the error). When the incremental error rates are no longer changing significantly or after a preset number of maximum minibatches to train, we claim that our model is trained.
One of the key parameter for optimization is called the learning_rate. For now, we can think of it as a scaling factor that modulates how much we change the parameters in any iteration. We will be covering more details in later tutorial.
With this information, we are ready to create our trainer.
Can you write Python code for it?

# Instantiate the trainer object to drive the model training
learning_rate = 0.2
lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
learner = C.sgd(z.parameters, lr_schedule)
trainer = C.Trainer(z, (loss, label_error), [learner])


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Evaluation
Below, we define the Evaluation (or metric) function that is used to report a measurement of how well our model is performing.
For this problem, we choose the classification_error() function as our metric, which returns the average error over the associated samples (treating a match as "1", where the model's prediction matches the "ground truth" label, and a non-match as "0").
Can you write Python code for it?

label_error = C.classification_error(z, label)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Evaluation / Testing
Now that we have trained the network, let us evaluate the trained network on the test data. This is done using trainer.test_minibatch.
Can you write Python code for it?

# Read the training data
reader_test = create_reader(test_file, False, input_dim, num_output_classes)

test_input_map = {
    label  : reader_test.streams.labels,
    input  : reader_test.streams.features,
}

# Test data for trained model
test_minibatch_size = 512
num_samples = 10000
num_minibatches_to_test = num_samples // test_minibatch_size
test_result = 0.0

for i in range(num_minibatches_to_test):

    # We are loading test data in batches specified by test_minibatch_size
    # Each data point in the minibatch is a MNIST digit image of 784 dimensions
    # with one pixel per dimension that we will encode / decode with the
    # trained model.
    data = reader_test.next_minibatch(test_minibatch_size,
                                      input_map = test_input_map)

    eval_error = trainer.test_minibatch(data)
    test_result = test_result + eval_error

# Average of evaluation errors of all test minibatches
print("Average test error: {0:.2f}%".format(test_result*100 / num_minibatches_to_test))


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Goal:
Our goal is to train a classifier that will identify the digits in the MNIST dataset. Additionally, we aspire to achieve lower error rate with Multi-layer perceptron compared to Multi-class logistic regression.
Approach:
There are 4 stages in this lab:
- Data reading: We will use the CNTK Text reader.
- Data preprocessing: Covered in part A (suggested extension section).
- Model creation: Multi-Layer Perceptron model.
- Train-Test-Predict: This is the same workflow introduced in the lectures
Can you write Python code for it?

from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import sys
import os

import cntk as C

%matplotlib inline


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Lab 3 - Multi Layer Perceptron with MNIST
This lab corresponds to Module 3 of the "Deep Learning Explained" course.  We assume that you have successfully completed Lab 1 (Downloading the MNIST data).
In this lab, we train a multi-layer perceptron on MNIST data. This notebook provides the recipe using Python APIs.
Introduction
Problem
We will continue to work on the same problem of recognizing digits in MNIST data. The MNIST data comprises of hand-written digits with little background noise.
Can you write Python code for it?

# Figure 1
Image(url= "http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png", width=200, height=200)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Let us plot the errors over the different training minibatches. Note that as we iterate the training loss decreases though we do see some intermediate bumps.
Can you write Python code for it?

# Compute the moving average loss to smooth out the noise in SGD
plotdata["avgloss"] = moving_average(plotdata["loss"])
plotdata["avgerror"] = moving_average(plotdata["error"])

# Plot the training loss and the training error
import matplotlib.pyplot as plt

plt.figure(1)
plt.subplot(211)
plt.plot(plotdata["batchsize"], plotdata["avgloss"], 'b--')
plt.xlabel('Minibatch number')
plt.ylabel('Loss')
plt.title('Minibatch run vs. Training loss')

plt.show()

plt.subplot(212)
plt.plot(plotdata["batchsize"], plotdata["avgerror"], 'r--')
plt.xlabel('Minibatch number')
plt.ylabel('Label Prediction Error')
plt.title('Minibatch run vs. Label Prediction Error')
plt.show()


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Let us test a small minibatch sample from the test data.
Can you write Python code for it?

# Read the data for evaluation
reader_eval = create_reader(test_file, False, input_dim, num_output_classes)

eval_minibatch_size = 25
eval_input_map = {input: reader_eval.streams.features}

data = reader_test.next_minibatch(eval_minibatch_size, input_map = test_input_map)

img_label = data[label].asarray()
img_data = data[input].asarray()
predicted_label_prob = [out.eval(img_data[i]) for i in range(len(img_data))]

# Find the index with the maximum value for both predicted as well as the ground truth
pred = [np.argmax(predicted_label_prob[i]) for i in range(len(predicted_label_prob))]
gtlabel = [np.argmax(img_label[i]) for i in range(len(img_label))]

print("Label    :", gtlabel[:25])
print("Predicted:", pred)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Multi-layer Perceptron setup
The code below is a direct translation of the model shown above.
Can you write Python code for it?

def create_model(features):
    with C.layers.default_options(init = C.layers.glorot_uniform(), activation = C.ops.relu):
    #with C.layers.default_options(init = C.layers.glorot_uniform(), activation = C.ops.sigmoid):
            h = features
            for _ in range(num_hidden_layers):
                h = C.layers.Dense(hidden_layers_dim)(h)
            r = C.layers.Dense(num_output_classes, activation = None)(h)
            #r = C.layers.Dense(num_output_classes, activation = C.ops.sigmoid)(h)
            return r

z = create_model(input)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Network input and output:
- input variable (a key CNTK concept):

An input variable is a container in which we fill different observations in this case image pixels during model learning (a.k.a.training) and model evaluation (a.k.a. testing). Thus, the shape of the input must match the shape of the data that will be provided.  For example, when data are images each of  height 10 pixels  and width 5 pixels, the input feature dimension will be 50 (representing the total number of image pixels). More on data and their dimensions to appear in separate tutorials.

Knowledge Check What is the input dimension of your chosen model? This is fundamental to our understanding of variables in a network or model representation in CNTK.
Can you write Python code for it?

input = C.input_variable(input_dim)
label = C.input_variable(num_output_classes)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Note, this error is very comparable to our training error indicating that our model has good "out of sample" error a.k.a. generalization error. This implies that our model can very effectively deal with previously unseen observations (during the training process). This is key to avoid the phenomenon of overfitting.
This is a huge reduction in error compared to multi-class LR (from Lab 02).
We have so far been dealing with aggregate measures of error. Let us now get the probabilities associated with individual data points. For each observation, the eval function returns the probability distribution across all the classes. The classifier is trained to recognize digits, hence has 10 classes. First let us route the network output through a softmax function. This maps the aggregated activations across the network to probabilities across the 10 classes.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
Training
Below, we define the Loss function, which is used to guide weight changes during training.
As explained in the lectures, we use the softmax function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the softmax function and other activation functions).
We minimize the cross-entropy between the label and predicted probability by the network.
Can you write Python code for it?

loss = C.cross_entropy_with_softmax(z, label)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:
z will be used to represent the output of a network.
We introduced sigmoid function in CNTK 102, in this tutorial you should try different activation functions in the hidden layer. You may choose to do this right away and take a peek into the performance later in the tutorial or run the preset tutorial and then choose to perform the suggested exploration.
 Suggested Exploration
- Record the training error you get with sigmoid as the activation function
- Now change to relu as the activation function and see if you can improve your training error
Knowledge Check: Name some of the different supported activation functions.  Which activation function gives the least training error?
Can you write Python code for it?

# Scale the input to 0-1 range by dividing each pixel by 255.
z = create_model(input/255.0)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
A Brief Look at Data and Data Reading
For reading text, this tutorial uses the CNTKTextFormatReader. It expects the input data to be
in the CTF format, as described here.
But how do you generate this format?  For this tutorial, we created the CTF file for you, but it might be helpful to explain how this was accomplished.  The data was created in two steps:
* convert the raw data into a plain text file that contains of TAB-separated columns of space-separated text. For example:
BOS show flights from burbank to st. louis on monday EOS (TAB) flight (TAB) O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-depart_date.day_name O
This is meant to be compatible with the output of the paste command.

convert it to CNTK Text Format (CTF) with the following command:

python [CNTK root]/Scripts/txt2ctf.py --map query.wl intent.wl slots.wl --annotated True --input atis.test.txt --output atis.test.ctf
  where the three .wl files give the vocabulary as plain text files, one word per line.
In these CTF files, our columns are labeled S0, S1, and S2.
These are connected to the actual network inputs by the corresponding lines in the reader definition:
Can you write Python code for it?

def create_reader(path, is_training):
    return C.io.MinibatchSource(C.io.CTFDeserializer(path, C.io.StreamDefs(
         query         = C.io.StreamDef(field='S0', shape=vocab_size,  is_sparse=True),
         intent_unused = C.io.StreamDef(field='S1', shape=num_intents, is_sparse=True),
         slot_labels   = C.io.StreamDef(field='S2', shape=num_labels,  is_sparse=True)
     )), randomize=is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)

# peek
reader = create_reader(data['train']['file'], is_training=True)
reader.streams.keys()


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Importing CNTK and other useful libraries
CNTK's Python module contains several submodules like io, learner, and layers. We also use NumPy in some cases since the results returned by CNTK work like NumPy arrays.
Can you write Python code for it?

import math
import numpy as np

import cntk as C


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Now we are ready to create a model and inspect it.
Once a model is constructed, its attributes are fully accessible from Python. The first layer named embed is an Embedding layer. Here we use the CNTK default, which is linear embedding. It is a simple matrix with dimension (input word encoding x output projected dimension). You can access its parameter E (where the embeddings are stored) like any other attribute of a Python object. Its shape contains a -1 which indicates that this parameter (with input dimension) is not fully specified yet, while the output dimension is set to emb_dim ( = 150 in this tutorial).
Additionally we also inspect the value of the bias vector in the Dense layer named classify. The Dense layer is a fundamental compositional unit of a Multi-Layer Perceptron (as introduced in Lab 3). Each Dense layer has both weight and bias parameters.  Bias terms are by default initialized to 0 (but there is a way to change that if you need). As you create the model, one can name the layer component and then access the parameters as shown here.
Suggested Exploration: What should be the expected dimension of the weight matrix from the layer named classify.  Try printing the weight matrix of the classify layer.  Does it match with your expected size?
Can you write Python code for it?

# peek
z = create_model()
print(z.embed.E.shape)
print(z.classify.b.value)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Our input text words will be encoded as one-hot vectors of length 943 and the output dimension of our model emb_dim is set to 150. In the code below we pass the input variable x to our model z. This binds the model with input data of known shape. In this case, the input shape will be the size of the input vocabulary. With this modification, the parameter returned by the embed layer is completely specified (943, 150).
Note: As an alternative to our approach here, you can initialize the Embedding matrix with pre-computed vectors using Word2Vec or GloVe.
Can you write Python code for it?

# Pass an input and check the dimension
z = create_model()
print(z(x).embed.E.shape)


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Task 2: Bidirectional Recurrent Model
Aha, knowledge of future words help. So instead of a one-word lookahead,
why not look ahead until all the way to the end of the sentence, through a backward recurrence?
Let us create a bidirectional model!
Your task is to implement a new layer that
performs both a forward and a backward recursion over the data, and
concatenates the output vectors.
Note, however, that this differs from the previous task in that
the bidirectional layer contains learnable model parameters.
In function-composition style,
the pattern to implement a layer with model parameters is to write a factory function
that creates a function object.
A function object, also known as functor, is an object that is both a function and an object.
Which means nothing else that it contains data yet still can be invoked as if it was a function.
For example, Dense(outDim) is a factory function that returns a function object that contains
a weight matrix W, a bias b, and another function to compute
input @ W + b. (This is using
Python 3.5 notation for matrix multiplication.
In Numpy syntax it is input.dot(W) + b).
E.g. saying Dense(1024) will create this function object, which can then be used
like any other function, also immediately: Dense(1024)(x).
Let's look at an example for further clarity: Let us implement a new layer that combines
a linear layer with a subsequent batch normalization.
To allow function composition, the layer needs to be realized as a factory function,
which could look like this:
python
def DenseLayerWithBN(dim):
    F = Dense(dim)
    G = BatchNormalization()
    x = placeholder()
    apply_x = G(F(x))
    return apply_x
Invoking this factory function will create F, G, x, and apply_x. In this example, F and G are function objects themselves, and apply_x is the function to be applied to the data.
Thus, e.g. calling DenseLayerWithBN(1024) will
create an object containing a linear-layer function object called F, a batch-normalization function object G,
and apply_x which is the function that...
Can you write Python code for it?

# Your task: Add bidirectional recurrence
def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels)
        ])

# Enable these when done:
#z = create_model()
#do_train_test()


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Task and Model Structure
The task we want to approach in this tutorial is slot tagging.
We use the ATIS corpus.
ATIS contains human-computer queries from the domain of Air Travel Information Services,
and our task will be to annotate (tag) each word of a query with the specific item of information (slot) it belongs to, if any.
The data in your working folder has already been converted into a CTF (CNTK Text Format) file.
Let us look at an example from the test-set file atis.test.ctf:
19  |S0 178:1 |# BOS      |S1 14:1 |# flight  |S2 128:1 |# O
19  |S0 770:1 |# show                         |S2 128:1 |# O
19  |S0 429:1 |# flights                      |S2 128:1 |# O
19  |S0 444:1 |# from                         |S2 128:1 |# O
19  |S0 272:1 |# burbank                      |S2 48:1  |# B-fromloc.city_name
19  |S0 851:1 |# to                           |S2 128:1 |# O
19  |S0 789:1 |# st.                          |S2 78:1  |# B-toloc.city_name
19  |S0 564:1 |# louis                        |S2 125:1 |# I-toloc.city_name
19  |S0 654:1 |# on                           |S2 128:1 |# O
19  |S0 601:1 |# monday                       |S2 26:1  |# B-depart_date.day_name
19  |S0 179:1 |# EOS                          |S2 128:1 |# O

This file has 5-7 columns per line (each separated by the "|" character):

a sequence id (19). There are 11 entries with this sequence id. This means that sequence 19 consists
of 11 tokens;
column S0, which contains numeric word indices; the input data is encoded in one-hot vectors.  There are 943 words in the vocabulary, so each word is a 943 element vector of all 0 with a 1 at a vector index chosen to represent that word.  For example the word "from" is represented with a 1 at index 444 and zero everywhere else in the vector. The word "monday" is represented with a 1 at index 601 and zero everywhere else in the vector.
a comment column denoted by #, to allow a human reader to know what the numeric word index stands for;
Comment columns are ignored by the system. BOS and EOS are special words
to...
Can you write Python code for it?

# setting seed
np.random.seed(0)
C.cntk_py.set_fixed_random_seed(1)
C.cntk_py.force_deterministic_algorithms()

# number of words in vocab, slot labels, and intent labels
vocab_size = 943 ; num_labels = 129 ; num_intents = 26

# model dimensions
input_dim  = vocab_size
label_dim  = num_labels

emb_dim    = 150 # error 2.25%
# emb_dim    = 50 # error  2.71%
#emb_dim = 300  # error 2.20%

#hidden_dim = 100 # error 2.44%
hidden_dim = 300 # error 2.25%
#hidden_dim = 500 # error 2.09%

# Create the containers for input feature (x) and the label (y)
x = C.sequence.input_variable(vocab_size)
y = C.sequence.input_variable(num_labels)

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim, name='embed'),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels, name='classify')
        ])


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
The following block of code illustrates how to evaluate a single sequence. Additionally we show how one can pass in the information using NumPy arrays.
Can you write Python code for it?

# load dictionaries
query_wl = [line.rstrip('\n') for line in open(data['query']['file'])]
slots_wl = [line.rstrip('\n') for line in open(data['slots']['file'])]
query_dict = {query_wl[i]:i for i in range(len(query_wl))}
slots_dict = {slots_wl[i]:i for i in range(len(slots_wl))}

# let's run a sequence through
seq = 'BOS flights from new york to seattle EOS'
w = [query_dict[w] for w in seq.split()] # convert to word indices
print(w)
onehot = np.zeros([len(w),len(query_dict)], np.float32)
for t in range(len(w)):
    onehot[t,w[t]] = 1

#x = C.sequence.input_variable(vocab_size)
pred = z(x).eval({x:[onehot]})[0]
print(pred.shape)
best = np.argmax(pred,axis=1)
print(best)
list(zip(seq.split(),[slots_wl[s] for s in best]))


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
This shows how learning proceeds over epochs (passes through the data).
For example, after four epochs, the loss, which is the cross-entropy criterion, has reached 0.11 as measured on the ~18000 samples of this epoch,
and that the error rate is 2.6% on those same 18000 training samples.
The epoch size is the number of samples--counted as word tokens, not sentences--to
process between model checkpoints.
Once the training has completed (a little less than 2 minutes on a Titan-X or a Surface Book),
you will see an output like this
Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039
which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.
On a CPU-only machine, it can be 4 or more times slower. You can try setting
python
emb_dim    = 50
hidden_dim = 100
to reduce the time it takes to run on a CPU, but the model will not fit as well as when the
hidden and embedding dimension are larger.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Trainer
We also must define the training criterion (loss function), and also an error metric to track the progress of our model's performance. In most tutorials, we know the input dimensions and the corresponding labels. We directly create the loss and the error functions. In this tutorial we will do the same. However, we take a brief detour and learn about placeholders. This concept would be useful for Task 3.
Learning note: Introduction to placeholder: Remember that the code we have been writing is not actually executing any heavy computation it is just specifying the function we want to compute on data during training/testing. And in the same way that it is convenient to have names for arguments when you write a regular function in a programming language, it is convenient to have placeholders that refer to arguments (or local computations that need to be reused). Eventually, some other code will replace these placeholders with other known quantities in the same way that in a programming language the function will be called with concrete values bound to its arguments.
Specifically, the input variables you have created above x = C.sequence.input_variable(vocab_size) holds data pre-defined by vocab_size. In the case where such instantiations are challenging or not possible, using placeholder is a logical choice. Having the placeholder only allows you to defer the specification of the argument at a later time when you may have the data.
Here is an example below that illustrates the use of placeholder.
Can you write Python code for it?

def create_criterion_function(model):
    labels = C.placeholder(name='labels')
    ce   = C.cross_entropy_with_softmax(model, labels)
    errs = C.classification_error      (model, labels)
    return C.combine ([ce, errs]) # (features, labels) -> (loss, metric)

criterion = create_criterion_function(create_model())
criterion.replace_placeholders({criterion.placeholders[0]: C.sequence.input_variable(num_labels)})


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab6_TextClassification_with_LSTM-checkpoint.ipynb".
The first task is:
Training the model
We are using the Progress Printer to display the training loss and classification error throughout training epochs.
The training should take less than 2 minutes on a Titan-X or a Surface Book. Once the training completed, you will see an output like this
Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039
which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.
On a CPU-only machine, it can be 4 or more times slower. You can try setting
python
emb_dim    = 50
hidden_dim = 100
to reduce the time it takes to run on a CPU, but the model will not fit as well as when the
hidden and embedding dimension are larger.
Testing the model
We also use the Progress Printer to display the accuracy on a test set by computing the error over multiple minibatches of test data. For evaluating on a small sample read from a file, you can set a minibatch size reflecting the sample size and run the test_minibatch on that instance of data. To see how to evaluate a single sequence, we provide an instance later in the tutorial.
Can you write Python code for it?

def train_test(train_reader, test_reader, model_func, max_epochs=10):

    # Instantiate the model function; x is the input (feature) variable
    model = model_func(x)

    # Instantiate the loss and error function
    loss, label_error = create_criterion_function_preferred(model, y)

    # training config
    epoch_size = 18000        # 18000 samples is half the dataset size
    minibatch_size = 70

    # LR schedule over epochs
    # In CNTK, an epoch is how often we get out of the minibatch loop to
    # do other stuff (e.g. checkpointing, adjust learning rate, etc.)
    # (we don't run this many epochs, but if we did, these are good values)
    lr_per_sample = [0.003]*4+[0.0015]*24+[0.0003]
    lr_per_minibatch = [lr * minibatch_size for lr in lr_per_sample]
    lr_schedule = C.learning_rate_schedule(lr_per_minibatch, C.UnitType.minibatch, epoch_size)

    # Momentum schedule
    momentum_as_time_constant = C.momentum_as_time_constant_schedule(700)

    # adam error 2.25%
    # sgd error 10%
    # fsadagrad 3.16%

    # We use a the Adam optimizer which is known to work well on this dataset
    # Feel free to try other optimizers from
    # https://www.cntk.ai/pythondocs/cntk.learner.html#module-cntk.learner
    learner = C.adam(parameters=model.parameters,
                     lr=lr_schedule,
                     momentum=momentum_as_time_constant,
                     gradient_clipping_threshold_per_sample=15,
                     gradient_clipping_with_truncation=True)

#     learning_rate = 0.2
#     lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
#     learner = C.sgd(z.parameters, lr_schedule)

#     learning_rate = 0.2
#     lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
#     TIMESTEPS = 14
#     BATCH_SIZE = TIMESTEPS * 10
#     momentum_time_constant = C.momentum_as_time_constant_schedule(BATCH_SIZE / -math.log(0.9))
#     learner = C.fsadagrad(z.parameters,
#                       lr = lr_schedule,
#      ...


I am working on the file "DAT236x Deep Learning Explained/Lab1_MNIST_DataLoader.ipynb".
The first task is:
Data download
We will download the data onto the local machine. The MNIST database is a standard set of handwritten digits that has been widely used for training and testing of machine learning algorithms. It has a training set of 60,000 images and a test set of 10,000 images with each image being 28 x 28 grayscale pixels. This set is easy to use visualize and train on any computer.
Can you write Python code for it?

# Functions to load MNIST images and unpack into train and test set.
# - loadData reads image data and formats into a 28x28 long array
# - loadLabels reads the corresponding labels data, 1 for each image
# - load packs the downloaded image and labels data into a combined format to be read later by
#   CNTK text reader

def loadData(src, cimg):
    print ('Downloading ' + src)
    gzfname, h = urlretrieve(src, './delete.me')
    print ('Done.')
    try:
        with gzip.open(gzfname) as gz:
            n = struct.unpack('I', gz.read(4))
            # Read magic number.
            if n[0] != 0x3080000:
                raise Exception('Invalid file: unexpected magic number.')
            # Read number of entries.
            n = struct.unpack('>I', gz.read(4))[0]
            if n != cimg:
                raise Exception('Invalid file: expected {0} entries.'.format(cimg))
            crow = struct.unpack('>I', gz.read(4))[0]
            ccol = struct.unpack('>I', gz.read(4))[0]
            if crow != 28 or ccol != 28:
                raise Exception('Invalid file: expected 28 rows/cols per image.')
            # Read data.
            res = np.fromstring(gz.read(cimg * crow * ccol), dtype = np.uint8)
    finally:
        os.remove(gzfname)
    return res.reshape((cimg, crow * ccol))

def loadLabels(src, cimg):
    print ('Downloading ' + src)
    gzfname, h = urlretrieve(src, './delete.me')
    print ('Done.')
    try:
        with gzip.open(gzfname) as gz:
            n = struct.unpack('I', gz.read(4))
            # Read magic number.
            if n[0] != 0x1080000:
                raise Exception('Invalid file: unexpected magic number.')
            # Read number of entries.
            n = struct.unpack('>I', gz.read(4))
            if n[0] != cimg:
                raise Exception('Invalid file: expected {0} rows.'.format(cimg))
            # Read labels.
            res = np.fromstring(gz.read(cimg), dtype = np.uint8)
    finally:
        os.remove(gzfname)
    return res.reshape((cimg, 1))

def...


I am working on the file "DAT236x Deep Learning Explained/Lab1_MNIST_DataLoader.ipynb".
The first task is:
Save the images
Save the images in a local directory. While saving the data we flatten the images to a vector (28x28 image pixels becomes an array of length 784 data points).

The labels are encoded as 1-hot encoding (label of 3 with 10 digits becomes 0001000000, where the first index corresponds to digit 0 and the last one corresponds to digit 9.
Can you write Python code for it?

# Save the data files into a format compatible with CNTK text reader
def savetxt(filename, ndarray):
    dir = os.path.dirname(filename)

    if not os.path.exists(dir):
        os.makedirs(dir)

    if not os.path.isfile(filename):
        print("Saving", filename )
        with open(filename, 'w') as f:
            labels = list(map(' '.join, np.eye(10, dtype=np.uint).astype(str)))
            for row in ndarray:
                row_str = row.astype(str)
                label_str = labels[row[-1]]
                feature_str = ' '.join(row_str[:-1])
                f.write('|labels {} |features {}\n'.format(label_str, feature_str))
    else:
        print("File already exists", filename)

# Save the train and test files (prefer our default path for the data)
data_dir = os.path.join("..", "Examples", "Image", "DataSets", "MNIST")
if not os.path.exists(data_dir):
    data_dir = os.path.join("data", "MNIST")

print ('Writing train text file...')
savetxt(os.path.join(data_dir, "Train-28x28_cntk_text.txt"), train)

print ('Writing test text file...')
savetxt(os.path.join(data_dir, "Test-28x28_cntk_text.txt"), test)

print('Done')


I am working on the file "DAT236x Deep Learning Explained/Lab1_MNIST_DataLoader.ipynb".
The first task is:
Visualize the data
Here, we use matplotlib to display one of the training images and it's associated label.
Can you write Python code for it?

# Plot a random image
sample_number = 5001
plt.imshow(train[sample_number,:-1].reshape(28,28), cmap="gray_r")
plt.axis('off')
print("Image Label: ", train[sample_number,-1])


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:

Run the trainer
We are now ready to train our fully connected neural net. We want to decide what data we need to feed into the training engine.
In this example, each iteration of the optimizer will work on minibatch_size sized samples. We would like to train on all 60000 observations. Additionally we will make multiple passes through the data specified by the variable num_sweeps_to_train_with. With these parameters we can proceed with training our simple feed forward network.
Can you write Python code for it?

# Initialize the parameters for the trainer
minibatch_size = 64
num_samples_per_sweep = 60000
num_sweeps_to_train_with =  10
num_minibatches_to_train = (num_samples_per_sweep * num_sweeps_to_train_with) / minibatch_size

num_minibatches_to_train

# Create the reader to training data set
reader_train = create_reader(train_file, True, input_dim, num_output_classes)

# Map the data streams to the input and labels.
input_map = {
    label  : reader_train.streams.labels,
    input  : reader_train.streams.features
}

# Run the trainer on and perform model training
training_progress_output_freq = 500

plotdata = {"batchsize":[], "loss":[], "error":[]}

import time
start = time.clock()
for i in range(0, int(num_minibatches_to_train)):

    # Read a mini batch from the training data file
    data = reader_train.next_minibatch(minibatch_size, input_map = input_map)

    trainer.train_minibatch(data)
    batchsize, loss, error = print_training_progress(trainer, i, training_progress_output_freq, verbose=1)

    if not (loss == "NA" or error =="NA"):
        plotdata["batchsize"].append(batchsize)
        plotdata["loss"].append(loss)
        plotdata["error"].append(error)

elapsed = (time.clock() - start)
print("Time used:",elapsed)


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Configure training
The trainer strives to reduce the loss function by different optimization approaches, Stochastic Gradient Descent (sgd) being one of the most popular. Typically, one would start with random initialization of the model parameters. The sgd optimizer would calculate the loss or error between the predicted label against the corresponding ground-truth label and using gradient-decent generate a new set model parameters in a single iteration.
The aforementioned model parameter update using a single observation at a time is attractive since it does not require the entire data set (all observation) to be loaded in memory and also requires gradient computation over fewer datapoints, thus allowing for training on large data sets. However, the updates generated using a single observation sample at a time can vary wildly between iterations. An intermediate ground is to load a small set of observations and use an average of the loss or error from that set to update the model parameters. This subset is called a minibatch.
With minibatches, we sample observations from the larger training dataset. We repeat the process of model parameters update using different combination of training samples and over a period of time minimize the loss (and the error metric). When the incremental error rates are no longer changing significantly or after a preset number of maximum minibatches to train, we claim that our model is trained.
One of the key optimization parameters is called the learning_rate. For now, we can think of it as a scaling factor that modulates how much we change the parameters in any iteration.
With this information, we are ready to create our trainer.
Can you write Python code for it?

# Instantiate the trainer object to drive the model training
learning_rate = 0.1 # 0.2
lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
learner = C.sgd(z.parameters, lr_schedule)
trainer = C.Trainer(z, (loss, label_error), [learner])


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Evaluation
Below, we define the Evaluation (or metric) function that is used to report a measurement of how well our model is performing.
For this problem, we choose the classification_error() function as our metric, which returns the average error over the associated samples (treating a match as "1", where the model's prediction matches the "ground truth" label, and a non-match as "0").
Can you write Python code for it?

label_error = C.classification_error(z, label)


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Goal:
Our goal is to train a classifier that will identify the digits in the MNIST dataset.
Approach:
There are 4 stages in this lab:
- Data reading: We will use the CNTK Text reader.
- Data preprocessing: Covered in part A (suggested extension section).
- Model creation: Multiclass Logistic Regression model.
- Train-Test-Predict: This is the same workflow introduced in the lectures
Logistic Regression
Logistic Regression (LR) is a fundamental machine learning technique that uses a linear weighted combination of features and generates probability-based predictions of different classes.
There are two basic forms of LR: Binary LR (with a single output that can predict two classes) and multiclass LR (with multiple outputs, each of which is used to predict a single class).

In Binary Logistic Regression (see top of figure above), the input features are each scaled by an associated weight and summed together.  The sum is passed through a squashing (aka activation) function and generates an output in [0,1].  This output value is then compared with a threshold (such as 0.5) to produce a binary label (0 or 1), predicting 1 of 2 classes.  This technique supports only classification problems with two output classes, hence the name binary LR.  In the binary LR example shown above, the sigmoid function is used as the squashing function.
In Multiclass Linear Regression (see bottom of figure above), 2 or more output nodes are used, one for each output class to be predicted.  Each summation node uses its own set of weights to scale the input features and sum them together. Instead of passing the summed output of the weighted input features through a sigmoid squashing function, the output is often passed through a softmax function (which in addition to squashing, like the sigmoid, the softmax normalizes each nodes' output value using the sum of all unnormalized nodes). (Details in the context of MNIST image to follow)
We will use multiclass LR for classifying the MNIST digits (0-9) using 10 output nodes (1 for each of...
Can you write Python code for it?

# Import the relevant components
from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import sys
import os

import cntk as C

%matplotlib inline


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU).
Can you write Python code for it?

# Select the right target device when this notebook is being tested:
if 'TEST_DEVICE' in os.environ:
    if os.environ['TEST_DEVICE'] == 'cpu':
        C.device.try_set_default_device(C.device.cpu())
    else:
        C.device.try_set_default_device(C.device.gpu(0))

# Test for CNTK version
if not C.__version__ == "2.0":
    raise Exception("this lab is designed to work with 2.0. Current Version: " + C.__version__)


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Initialization
Can you write Python code for it?

# Ensure we always get the same amount of randomness
np.random.seed(0)
C.cntk_py.set_fixed_random_seed(1)
C.cntk_py.force_deterministic_algorithms()

# Define the data dimensions
input_dim = 784
num_output_classes = 10


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Lab 2 - Logistic Regression (LR) with MNIST
This lab corresponds to Module 2 of the "Deep Learning Explained" course. We assume that you have successfully completed Lab 1 (Downloading the MNIST data).
In this lab we will build and train a Multiclass Logistic Regression model using the MNIST data.
Introduction
Problem:
Optical Character Recognition (OCR) is a hot research area and there is a great demand for automation. The MNIST data is comprised of hand-written digits with little background noise making it a nice dataset to create, experiment and learn deep learning models with reasonably small comptuing resources.
Can you write Python code for it?

# Figure 1
Image(url= "http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png", width=200, height=200)


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Let us plot the errors over the different training minibatches. Note that as we progress in our training, the loss decreases though we do see some intermediate bumps.
Can you write Python code for it?

# Compute the moving average loss to smooth out the noise in SGD
plotdata["avgloss"] = moving_average(plotdata["loss"])
plotdata["avgerror"] = moving_average(plotdata["error"])

# Plot the training loss and the training error
import matplotlib.pyplot as plt

plt.figure(1)
plt.subplot(211)
plt.plot(plotdata["batchsize"], plotdata["avgloss"], 'b--')
plt.xlabel('Minibatch number')
plt.ylabel('Loss')
plt.title('Minibatch run vs. Training loss')

plt.show()

plt.subplot(212)
plt.plot(plotdata["batchsize"], plotdata["avgerror"], 'r--')
plt.xlabel('Minibatch number')
plt.ylabel('Label Prediction Error')
plt.title('Minibatch run vs. Label Prediction Error')
plt.show()


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Let us test a small minibatch sample from the test data.
Can you write Python code for it?

# Read the data for evaluation
reader_eval = create_reader(test_file, False, input_dim, num_output_classes)

eval_minibatch_size = 25
eval_input_map = {input: reader_eval.streams.features}

data = reader_test.next_minibatch(eval_minibatch_size, input_map = test_input_map)

img_label = data[label].asarray()
img_data = data[input].asarray()
predicted_label_prob = [out.eval(img_data[i]) for i in range(len(img_data))]

# Find the index with the maximum value for both predicted as well as the ground truth
pred = [np.argmax(predicted_label_prob[i]) for i in range(len(predicted_label_prob))]
gtlabel = [np.argmax(img_label[i]) for i in range(len(img_label))]

print("Label    :", gtlabel[:25])
print("Predicted:", pred)


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Logistic Regression network setup
The CNTK Layers module provides a Dense function that creates a fully connected layer which performs the above operations of weighted input summing and bias addition.
Can you write Python code for it?

def create_model(features):
    with C.layers.default_options(init = C.glorot_uniform()):
        r = C.layers.Dense(num_output_classes, activation = None)(features)
        #r = C.layers.Dense(num_output_classes, activation = None)(C.ops.splice(C.ops.sqrt(features), features, C.ops.square(features)))
        return r


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Model Creation
A multiclass logistic regression (LR) network is a simple building block that has been effectively powering many ML
applications in the past decade. The figure below summarizes the model in the context of the MNIST data.

LR is a simple linear model that takes as input, a vector of numbers describing the properties of what we are classifying (also known as a feature vector, $\bf \vec{x}$, the pixels in the input MNIST digit image) and emits the evidence ($z$). For each of the 10 digits, there is a vector of weights corresponding to the input pixels as show in the figure. These 10 weight vectors define the weight matrix ($\bf {W}$) with dimension of 10 x 784.  Each feature in the input layer is connected with a summation node by a corresponding weight $w$ (individual weight values from the $\bf{W}$ matrix). Note there are 10 such nodes, 1 corresponding to each digit to be classified.
The first step is to compute the evidence for an observation.
$$\vec{z} = \textbf{W} \bf \vec{x}^T + \vec{b}$$
where $\bf{W}$ is the weight matrix of dimension 10 x 784 and $\vec{b}$ is known as the bias vector with lenght 10, one for each digit.
The evidence ($\vec{z}$) is not squashed (hence no activation). Instead the output is normalized using a softmax function such that all the outputs add up to a value of 1, thus lending a probabilistic iterpretation to the prediction. In CNTK, we use the softmax operation combined with the cross entropy error as our Loss Function for training.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
Training
Below, we define the Loss function, which is used to guide weight changes during training.
As explained in the lectures, we use the softmax function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the softmax function and other activation functions).
We minimize the cross-entropy between the label and predicted probability by the network.
Can you write Python code for it?

loss = C.cross_entropy_with_softmax(z, label)

loss


I am working on the file "DAT236x Deep Learning Explained/Lab2_LogisticRegression.ipynb".
The first task is:
We have so far been dealing with aggregate measures of error. Let us now get the probabilities associated with individual data points. For each observation, the eval function returns the probability distribution across all the classes. The classifier is trained to recognize digits, hence has 10 classes. First let us route the network output through a softmax function. This maps the aggregated activations across the network to probabilities across the 10 classes.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:

Run the trainer
We are now ready to train our fully connected neural net. We want to decide what data we need to feed into the training engine.
In this example, each iteration of the optimizer will work on minibatch_size sized samples. We would like to train on all 60000 observations. Additionally we will make multiple passes through the data specified by the variable num_sweeps_to_train_with. With these parameters we can proceed with training our simple multi-layer perceptron network.
Can you write Python code for it?

# Create the reader to training data set
reader_train = create_reader(train_file, True, input_dim, num_output_classes)

# Map the data streams to the input and labels.
input_map = {
    label  : reader_train.streams.labels,
    input  : reader_train.streams.features
}

# Run the trainer on and perform model training
training_progress_output_freq = 500

plotdata = {"batchsize":[], "loss":[], "error":[]}

for i in range(0, int(num_minibatches_to_train)):

    # Read a mini batch from the training data file
    data = reader_train.next_minibatch(minibatch_size, input_map = input_map)

    trainer.train_minibatch(data)
    batchsize, loss, error = print_training_progress(trainer, i, training_progress_output_freq, verbose=1)

    if not (loss == "NA" or error =="NA"):
        plotdata["batchsize"].append(batchsize)
        plotdata["loss"].append(loss)
        plotdata["error"].append(error)


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
As you can see above, our model is much better.  Do you see any mismatches?
Let us visualize one of the test images and its associated label.  Do they match?
Can you write Python code for it?

# Plot a random image
sample_number = 5
plt.imshow(img_data[sample_number].reshape(28,28), cmap="gray_r")
plt.axis('off')

img_gt, img_pred = gtlabel[sample_number], pred[sample_number]
print("Image Label: ", img_pred)


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Configure training
The trainer strives to reduce the loss function by different optimization approaches, Stochastic Gradient Descent (sgd) being a basic one. Typically, one would start with random initialization of the model parameters. The sgd optimizer would calculate the loss or error between the predicted label against the corresponding ground-truth label and using gradient-decent generate a new set model parameters in a single iteration.
The aforementioned model parameter update using a single observation at a time is attractive since it does not require the entire data set (all observation) to be loaded in memory and also requires gradient computation over fewer datapoints, thus allowing for training on large data sets. However, the updates generated using a single observation sample at a time can vary wildly between iterations. An intermediate ground is to load a small set of observations and use an average of the loss or error from that set to update the model parameters. This subset is called a minibatch.
With minibatches we often sample observation from the larger training dataset. We repeat the process of model parameters update using different combination of training samples and over a period of time minimize the loss (and the error). When the incremental error rates are no longer changing significantly or after a preset number of maximum minibatches to train, we claim that our model is trained.
One of the key parameter for optimization is called the learning_rate. For now, we can think of it as a scaling factor that modulates how much we change the parameters in any iteration. We will be covering more details in later tutorial.
With this information, we are ready to create our trainer.
Can you write Python code for it?

# Instantiate the trainer object to drive the model training
learning_rate = 0.2
lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
learner = C.sgd(z.parameters, lr_schedule)
trainer = C.Trainer(z, (loss, label_error), [learner])


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Data reading
There are different ways one can read data into CNTK. The easiest way is to load the data in memory using NumPy / SciPy / Pandas readers. However, this can be done only for small data sets. Since deep learning requires large amount of data we have chosen in this course to show how to leverage built-in distributed readers that can scale to terrabytes of data with little extra effort.
We are using the MNIST data you have downloaded using Lab 1 DataLoader notebook. The dataset has 60,000 training images and 10,000 test images with each image being 28 x 28 pixels. Thus the number of features is equal to 784 (= 28 x 28 pixels), 1 per pixel. The variable num_output_classes is set to 10 corresponding to the number of digits (0-9) in the dataset.
In Lab 1, the data was downloaded and written to 2 CTF (CNTK Text Format) files, 1 for training, and 1 for testing. Each line of these text files takes the form:
|labels 0 0 0 1 0 0 0 0 0 0 |features 0 0 0 0 ...
                                              (784 integers each representing a pixel)

We are going to use the image pixels corresponding the integer stream named "features". We define a create_reader function to read the training and test data using the CTF deserializer. The labels are 1-hot encoded. Refer to Lab 1 for data format visualizations.
Can you write Python code for it?

# Read a CTF formatted text (as mentioned above) using the CTF deserializer from a file
def create_reader(path, is_training, input_dim, num_label_classes):
    return C.io.MinibatchSource(C.io.CTFDeserializer(path, C.io.StreamDefs(
        labels = C.io.StreamDef(field='labels', shape=num_label_classes, is_sparse=False),
        features   = C.io.StreamDef(field='features', shape=input_dim, is_sparse=False)
    )), randomize = is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)

# Ensure the training and test data is generated and available for this tutorial.
# We search in two locations in the toolkit for the cached MNIST data set.
data_found = False
for data_dir in [os.path.join("..", "Examples", "Image", "DataSets", "MNIST"),
                 os.path.join("data", "MNIST")]:
    train_file = os.path.join(data_dir, "Train-28x28_cntk_text.txt")
    test_file = os.path.join(data_dir, "Test-28x28_cntk_text.txt")
    if os.path.isfile(train_file) and os.path.isfile(test_file):
        data_found = True
        break
if not data_found:
    raise ValueError("Please generate the data by completing Lab1_MNIST_DataLoader")
print("Data directory is {0}".format(data_dir))


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
First let us create some helper functions that will be needed to visualize different functions associated with training.
Can you write Python code for it?

# Define a utility function to compute the moving average sum.
# A more efficient implementation is possible with np.cumsum() function
def moving_average(a, w=5):
    if len(a) < w:
        return a[:]    # Need to send a copy of the array
    return [val if idx < w else sum(a[(idx-w):idx])/w for idx, val in enumerate(a)]

# Defines a utility that prints the training progress
def print_training_progress(trainer, mb, frequency, verbose=1):
    training_loss = "NA"
    eval_error = "NA"

    if mb%frequency == 0:
        training_loss = trainer.previous_minibatch_loss_average
        eval_error = trainer.previous_minibatch_evaluation_average
        if verbose:
            print ("Minibatch: {0}, Loss: {1:.4f}, Error: {2:.2f}%".format(mb, training_loss, eval_error*100))

    return mb, training_loss, eval_error

# Initialize the parameters for the trainer
minibatch_size = 64
#minibatch_size = 512
num_samples_per_sweep = 60000
num_sweeps_to_train_with = 10
num_minibatches_to_train = (num_samples_per_sweep * num_sweeps_to_train_with) / minibatch_size


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Goal:
Our goal is to train a classifier that will identify the digits in the MNIST dataset. Additionally, we aspire to achieve lower error rate with Multi-layer perceptron compared to Multi-class logistic regression.
Approach:
There are 4 stages in this lab:
- Data reading: We will use the CNTK Text reader.
- Data preprocessing: Covered in part A (suggested extension section).
- Model creation: Multi-Layer Perceptron model.
- Train-Test-Predict: This is the same workflow introduced in the lectures
Can you write Python code for it?

from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import sys
import os

import cntk as C

%matplotlib inline


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Lab 3 - Multi Layer Perceptron with MNIST
This lab corresponds to Module 3 of the "Deep Learning Explained" course.  We assume that you have successfully completed Lab 1 (Downloading the MNIST data).
In this lab, we train a multi-layer perceptron on MNIST data. This notebook provides the recipe using Python APIs.
Introduction
Problem
We will continue to work on the same problem of recognizing digits in MNIST data. The MNIST data comprises of hand-written digits with little background noise.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Let us plot the errors over the different training minibatches. Note that as we iterate the training loss decreases though we do see some intermediate bumps.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Let us test a small minibatch sample from the test data.
Can you write Python code for it?

# Read the data for evaluation
reader_eval = create_reader(test_file, False, input_dim, num_output_classes)

eval_minibatch_size = 25
eval_input_map = {input: reader_eval.streams.features}

data = reader_test.next_minibatch(eval_minibatch_size, input_map = test_input_map)

img_label = data[label].asarray()
img_data = data[input].asarray()
predicted_label_prob = [out.eval(img_data[i]) for i in range(len(img_data))]

predicted_label_prob[0], np.argmax(predicted_label_prob[0])

# Find the index with the maximum value for both predicted as well as the ground truth
pred = [np.argmax(predicted_label_prob[i]) for i in range(len(predicted_label_prob))]
gtlabel = [np.argmax(img_label[i]) for i in range(len(img_label))]

print("Label    :", gtlabel[:25])
print("Predicted:", pred)


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Multi-layer Perceptron setup
The code below is a direct translation of the model shown above.
Can you write Python code for it?

def create_model(features):
    with C.layers.default_options(init = C.layers.glorot_uniform(), activation = C.ops.relu):
    #with C.layers.default_options(init = C.layers.glorot_uniform(), activation = C.ops.sigmoid):
            h = features
            for _ in range(num_hidden_layers):
                h = C.layers.Dense(hidden_layers_dim)(h)
            r = C.layers.Dense(num_output_classes, activation = None)(h)
            #r = C.layers.Dense(num_output_classes, activation = C.ops.sigmoid)(h)
            return r

z = create_model(input)


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Network input and output:
- input variable (a key CNTK concept):

An input variable is a container in which we fill different observations in this case image pixels during model learning (a.k.a.training) and model evaluation (a.k.a. testing). Thus, the shape of the input must match the shape of the data that will be provided.  For example, when data are images each of  height 10 pixels  and width 5 pixels, the input feature dimension will be 50 (representing the total number of image pixels). More on data and their dimensions to appear in separate tutorials.

Knowledge Check What is the input dimension of your chosen model? This is fundamental to our understanding of variables in a network or model representation in CNTK.
Can you write Python code for it?

input = C.input_variable(input_dim)
label = C.input_variable(num_output_classes)


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Note, this error is very comparable to our training error indicating that our model has good "out of sample" error a.k.a. generalization error. This implies that our model can very effectively deal with previously unseen observations (during the training process). This is key to avoid the phenomenon of overfitting.
This is a huge reduction in error compared to multi-class LR (from Lab 02).
We have so far been dealing with aggregate measures of error. Let us now get the probabilities associated with individual data points. For each observation, the eval function returns the probability distribution across all the classes. The classifier is trained to recognize digits, hence has 10 classes. First let us route the network output through a softmax function. This maps the aggregated activations across the network to probabilities across the 10 classes.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
Training
Below, we define the Loss function, which is used to guide weight changes during training.
As explained in the lectures, we use the softmax function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the softmax function and other activation functions).
We minimize the cross-entropy between the label and predicted probability by the network.
Can you write Python code for it?

loss = C.cross_entropy_with_softmax(z, label)


I am working on the file "DAT236x Deep Learning Explained/Lab3_MultiLayerPerceptron.ipynb".
The first task is:
z will be used to represent the output of a network.
We introduced sigmoid function in CNTK 102, in this tutorial you should try different activation functions in the hidden layer. You may choose to do this right away and take a peek into the performance later in the tutorial or run the preset tutorial and then choose to perform the suggested exploration.
 Suggested Exploration
- Record the training error you get with sigmoid as the activation function
- Now change to relu as the activation function and see if you can improve your training error
Knowledge Check: Name some of the different supported activation functions.  Which activation function gives the least training error?
Can you write Python code for it?

# Scale the input to 0-1 range by dividing each pixel by 255.
z = create_model(input/255.0)


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:

CNN Model Creation
CNN is a feedforward network made up of bunch of layers in such a way that the output of one layer becomes the input to the next layer (similar to MLP). In MLP, all possible pairs of input pixels are connected to the output nodes with each pair having a weight, thus leading to a combinatorial explosion of parameters to be learnt and also increasing the possibility of overfitting (details). Convolution layers take advantage of the spatial arrangement of the pixels and learn multiple filters that significantly reduce the amount of parameters in the network (details). The size of the filter is a parameter of the convolution layer.
In this section, we introduce the basics of convolution operations. We show the illustrations in the context of RGB images (3 channels), eventhough the MNIST data we are using a grayscale image (single channel).

Convolution Layer
A convolution layer is a set of filters. Each filter is defined by a weight (W) matrix, and  bias ($b$).

These filters are scanned across the image performing the dot product between the weights and corresponding input value (${x}$). The bias value is added to the output of the dot product and the resulting sum is optionally mapped through an activation function. This process is illustrated in the following animation.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Configure training
Previously we have described the concepts of loss function, the optimizers or learners and the associated machinery needed to train a model. Please refer to earlier labs for gaining familiarility with these concepts. Here we combine model training and testing in a helper function below.
Can you write Python code for it?

def train_test(train_reader, test_reader, model_func, num_sweeps_to_train_with=10):

    # Instantiate the model function; x is the input (feature) variable
    # We will scale the input image pixels within 0-1 range by dividing all input value by 255.
    model = model_func(x/255)

    # Instantiate the loss and error function
    loss, label_error = create_criterion_function(model, y)

    # Instantiate the trainer object to drive the model training
    learning_rate = 0.2
    lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
    learner = C.sgd(z.parameters, lr_schedule)
    trainer = C.Trainer(z, (loss, label_error), [learner])

    # Initialize the parameters for the trainer
    minibatch_size = 64
    num_samples_per_sweep = 60000
    num_minibatches_to_train = (num_samples_per_sweep * num_sweeps_to_train_with) / minibatch_size

    # Map the data streams to the input and labels.
    input_map={
        y  : train_reader.streams.labels,
        x  : train_reader.streams.features
    }

    # Uncomment below for more detailed logging
    training_progress_output_freq = 500

    # Start a timer
    start = time.time()

    for i in range(0, int(num_minibatches_to_train)):
        # Read a mini batch from the training data file
        data=train_reader.next_minibatch(minibatch_size, input_map=input_map)
        trainer.train_minibatch(data)
        print_training_progress(trainer, i, training_progress_output_freq, verbose=1)

    # Print training time
    print("Training took {:.1f} sec".format(time.time() - start))

    # Test the model
    test_input_map = {
        y  : test_reader.streams.labels,
        x  : test_reader.streams.features
    }

    # Test data for trained model
    test_minibatch_size = 512
    num_samples = 10000
    num_minibatches_to_test = num_samples // test_minibatch_size

    test_result = 0.0

    for i in range(num_minibatches_to_test):

        # We are loading test data in batches specified by...


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Data reading
There are different ways one can read data into CNTK. The easiest way is to load the data in memory using NumPy / SciPy / Pandas readers. However, this can be done only for small data sets. Since deep learning requires large amount of data we have chosen in this course to show how to leverage built-in distributed readers that can scale to terrabytes of data with little extra effort.
We are using the MNIST data you have downloaded using Lab 1 DataLoader notebook. The dataset has 60,000 training images and 10,000 test images with each image being 28 x 28 pixels. Thus the number of features is equal to 784 (= 28 x 28 pixels), 1 per pixel. The variable num_output_classes is set to 10 corresponding to the number of digits (0-9) in the dataset.
In Lab 1, the data was downloaded and written to 2 CTF (CNTK Text Format) files, 1 for training, and 1 for testing. Each line of these text files takes the form:
|labels 0 0 0 1 0 0 0 0 0 0 |features 0 0 0 0 ...
                                              (784 integers each representing a pixel)

We are going to use the image pixels corresponding the integer stream named "features". We define a create_reader function to read the training and test data using the CTF deserializer. The labels are 1-hot encoded. Refer to Lab 1 for data format visualizations.
Can you write Python code for it?

# Read a CTF formatted text (as mentioned above) using the CTF deserializer from a file
def create_reader(path, is_training, input_dim, num_label_classes):

    ctf = C.io.CTFDeserializer(path, C.io.StreamDefs(
          labels=C.io.StreamDef(field='labels', shape=num_label_classes, is_sparse=False),
          features=C.io.StreamDef(field='features', shape=input_dim, is_sparse=False)))

    return C.io.MinibatchSource(ctf,
        randomize = is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)

# Ensure the training and test data is available
# We search in two locations in the toolkit for the cached MNIST data set.

data_found=False # A flag to indicate if train/test data found in local cache
for data_dir in [os.path.join("..", "Examples", "Image", "DataSets", "MNIST"),
                 os.path.join("data", "MNIST")]:

    train_file=os.path.join(data_dir, "Train-28x28_cntk_text.txt")
    test_file=os.path.join(data_dir, "Test-28x28_cntk_text.txt")

    if os.path.isfile(train_file) and os.path.isfile(test_file):
        data_found=True
        break

if not data_found:
    raise ValueError("Please generate the data by completing Lab1_MNIST_DataLoader")

print("Data directory is {0}".format(data_dir))


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Evaluation / Prediction
We have so far been dealing with aggregate measures of error. Let us now get the probabilities associated with individual data points. For each observation, the eval function returns the probability distribution across all the classes. The classifier is trained to recognize digits, hence has 10 classes. First let us route the network output through a softmax function. This maps the aggregated activations across the network to probabilities across the 10 classes.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Goal:
Our goal is to train a classifier that will identify the digits in the MNIST dataset.
Approach:
The same 5 stages we have used in the previous labs are applicable: Data reading, Data preprocessing, Creating a model, Learning the model parameters and Evaluating (a.k.a. testing/prediction) the model.
We will experiment with two models with different architechtural components.
Can you write Python code for it?

from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
import time

import cntk as C

%matplotlib inline


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU).
Can you write Python code for it?

# Select the right target device when this notebook is being tested:
if 'TEST_DEVICE' in os.environ:
    if os.environ['TEST_DEVICE'] == 'cpu':
        C.device.try_set_default_device(C.device.cpu())
    else:
        C.device.try_set_default_device(C.device.gpu(0))

# Test for CNTK version
if not C.__version__ == "2.0":
    raise Exception("this notebook is designed to work with 2.0. Current Version: " + C.__version__)


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Lab 4 - Convolutional Neural Network with MNIST
This lab corresponds to Module 4 of the "Deep Learning Explained" course. We assume that you have successfully Lab 1 to download the MNIST data.
We will train a Convolutional Neural Network (CNN) on MNIST data.
Introduction
A convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network made up of neurons that have learnable weights and biases, very similar to ordinary multi-layer perceptron (MLP) networks introduced in Module 3. The CNNs take advantage of the spatial nature of the data.
In nature, we perceive different objects by their shapes, size and colors. For example, objects in a natural scene are typically composed of edges, corners/vertices (defined by two of more edges), color patches etc. These primitives are often identified using different detectors (e.g., edge detection, color detector) or combination of detectors interacting to facilitate image interpretation (object classification, region of interest detection, scene description etc.) in real world vision related tasks. These detectors are also known as filters.
Convolution is a mathematical operator that takes an image and a filter as input and produces a filtered output (representing say egdges, corners, colors etc in the input image).  Historically, these filters are a set of weights that were often hand crafted or modeled with mathematical functions (e.g., Gaussian / Laplacian / Canny filter).  The filter outputs are mapped through non-linear activation functions mimicking human brain cells called neurons.
Convolutional networks provide a machinery to learn these filters from the data directly instead of explicit mathematical models and have been found to be superior (in real world tasks) compared to historically crafted filters.  With convolutional networks, the focus is on learning the filter weights instead of learning individually fully connected pair-wise (between inputs and outputs) weights. In this way, the number of weights to learn is reduced...
Can you write Python code for it?

# Figure 1
Image(url= "http://3.bp.blogspot.com/_UpN7DfJA0j4/TJtUBWPk0SI/AAAAAAAAABY/oWPMtmqJn3k/s1600/mnist_originals.png", width=200, height=200)


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Let us create an instance of the model and inspect the different components of the model. z will be used to represent the output of a network. In this model, we use the relu activation function. Note: using the C.layers.default_options is an elegant and concise way to build models. This is key to minimizing modeling errors, saving precious debugging time.
Can you write Python code for it?

# Create the model
z = create_model(x)

# Print the output shapes / parameters of different components
print("Output Shape of the first convolution layer:", z.first_conv.shape)
print("Bias value of the last dense layer:", z.classify.b.value)

print("Output Shape of the second convolution layer:", z.second_conv.shape)


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Let us visualize some of the results
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
The first model we build is a simple convolution only network. Here we have two convolutional layers. Since, our task is to detect the 10 digits in the MNIST database, the output of the network should be a vector of length 10, 1 element corresponding to each output class. This is achieved by projecting the output of the last convolutional layer using a dense layer with the output being num_output_classes. We have seen this before with Logistic Regression and MLP where features were mapped to the number of classes in the final layer. Also, note that since we will be using the softmax operation that is combined with the cross entropy loss function during training (see a few cells below), the final dense layer has no activation function associated with it.
The following figure illustrates the model we are going to build. Note the parameters in the model below are to be experimented with. These are often called network hyperparameters. Increasing the filter shape leads to an increase in the number of model parameters, increases the compute time and helps the model better fit to the data. However, one runs the risk of overfitting. Typically, the number of filters in the deeper layers are more than the number of filters in the layers before them. We have chosen 8, 16 for the first and second layers, respectively. These hyperparameters should be experimented with during model building.
Can you write Python code for it?

# function to build model
# relu error:1.37
# sigmoid error: 2.69
# leaky relu error: 1.26
# tanh error: 1.4
# def create_model(features):
#     with C.layers.default_options(init=C.glorot_uniform(), activation=C.relu):
#     #with C.layers.default_options(init=C.glorot_uniform(), activation=C.sigmoid):
#     #with C.layers.default_options(init=C.glorot_uniform(), activation=C.leaky_relu):
#     #with C.layers.default_options(init=C.glorot_uniform(), activation=C.tanh):
#             h = features
#             h = C.layers.Convolution2D(filter_shape=(5,5),
#                                        num_filters=8,
#                                        strides=(2,2),
#                                        pad=True, name='first_conv')(h)
#             h = C.layers.Convolution2D(filter_shape=(5,5),
#                                        num_filters=16,
#                                        strides=(2,2),
#                                        pad=True, name='second_conv')(h)
#             r = C.layers.Dense(num_output_classes, activation=None, name='classify')(h)
#             return r

# average pool
# relu error:1.25
# leak relu error: 1.17
# sigmoid error:3.89
# tanh erro:1.47
# def create_model(features):
#     with C.layers.default_options(init=C.glorot_uniform(), activation=C.leaky_relu):
#             h = features
#             h = C.layers.Convolution2D(filter_shape=(5,5),
#                                        num_filters=8,
#                                        strides=(1,1),
#                                        pad=True, name='first_conv')(h)
#             h = C.layers.AveragePooling(filter_shape=(5,5),
#                                         strides=(2,2), name = 'first_avg_pool')(h)
#             h = C.layers.Convolution2D(filter_shape=(5,5),
#                                        num_filters=16,
#                                        strides=(1,1),
#                                        pad=True, name='second_conv')(h)
#             h =...


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Understanding Parameters:
Our model has 2 convolution layers each having a weight and bias. This adds up to 4 parameter tensors. Additionally the dense layer has weight and bias tensors. Thus, the 6 parameter tensors.
Remember that in a convolutional layer, the number of parameters is not dependent on the number of nodes, only on the shared weights and bias of each filter.
Let us now count the number of parameters:
- First convolution layer: There are 8 filters each of size (1 x 5 x 5) where 1 is the number of channels in the input image. This adds up to 200 values in the weight matrix and 8 bias values.

Second convolution layer: There are 16 filters each of size (8 x 5 x 5) where 8 is the number of channels in the input to the second layer (= output of the first layer). This adds up to 3200 values in the weight matrix and 16 bias values.

Last dense layer: There are 16 x 7 x 7 input values and it produces 10 output values corresponding to the 10 digits in the MNIST dataset. This corresponds to (16 x 7 x 7) x 10 weight values and 10 bias values.

Adding these up gives the 11274 parameters in the model.
Knowledge Check: Does the dense layer shape align with the task (MNIST digit classification)?
 Suggested Explorations
- Try printing shapes and parameters of different network layers,
- Record the training error you get with relu as the activation function,
- Now change to sigmoid as the activation function and see if you can improve your training error.
- Different supported activation functions can be found here. Which activation function gives the least training error?
Learning model parameters
We use the softmax function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the softmax function and other activation functions).
Training
We minimize the cross-entropy between the label and predicted probability by the network. Since we are going to build more than one model, we will create a few helper functions.
Can you write Python code for it?

def create_criterion_function(model, labels):
    loss = C.cross_entropy_with_softmax(model, labels)
    errs = C.classification_error(model, labels)
    return loss, errs # (model, labels) -> (loss, error metric)


I am working on the file "DAT236x Deep Learning Explained/Lab4_ConvolutionalNeuralNetwork.ipynb".
The first task is:
Understanding the number of learnable parameters in a model is key to deep learning since there is a  dependency between the number of parameters and the amount of data one needs to have to train the model.
You need more data for a model that has a larger number of parameters to prevent overfitting. In other words, with a fixed amount of data, one has to constrain the number of parameters. There is no golden rule between the amount of data one needs for a model. However, there are ways one can boost performance of model training with data augmentation.
Can you write Python code for it?

# Number of parameters in the network
C.logging.log_number_of_parameters(z)


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
A look how the loss function shows how the model is converging:
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
Data generation
Our solar panel emits two measures at 30 min interval:
- solar.current is the current production in Watt
- solar.total is the total produced for the day so far in Watt/hour
Our prediction approach involves starting with the first 2 initial readings of the day. Based on these readings we start predicting and adjust the prediction with each new reading. The training data we are going to use comes as a CSV file and has the following format:

time,solar.current,solar.total
7am,6.3,1.7
7:30am,44.3,11.4
...

Our training dataset contains 3 years of captured data and can be found here.
The dataset is not pre-processed: it is raw data and contains smaller gaps and errors (like a panel failed to report).
Pre-processing
Most of the code in this example is related to data preparation. Thankfully the pandas library make this easy.
Below, we define a generate_solar_data() function that performs the following tasks:
- read raw data into a pandas dataframe
- normalize the data
- group the data by day
- append the columns "solar.current.max" and "solar.total.max"
- generate the sequences for each day
 Sequence Generation  The measurements for a day are grouped into a set of sequences, corresponding to each measurement for the day.  Each measurement sequence will contain all of the raw measurements (solar.current, solar.total) seen for the current day, up until and including the current measurement.
Here is an example of some sequehnces for a day:
1.7,11.4 -> 10300
1.7,11.4,67.5 -> 10300
1.7,11.4,67.5,250.5 ... -> 10300
1.7,11.4,67.5,250.5,573.5 -> 10300
All sequences for all days are flattened into a single list of sequences. The day and timestamp information have been removed; only the sequences matter.
Note if we have less than 8 datapoints for a day we skip over the day assuming something is missing in the raw data. If we get more than 14 data points in a day we truncate the readings.
Training / Testing / Validation data preparation
We start by reading the CSV file. The raw data is sorted by...
Can you write Python code for it?

def generate_solar_data(input_url, time_steps, normalize=1, val_size=0.1, test_size=0.1):
    """
    generate sequences to feed to rnn based on data frame with solar panel data
    the csv has the format: time ,solar.current, solar.total
     (solar.current is the current output in Watt, solar.total is the total production
      for the day so far in Watt hours)
    """
    # try to find the data file local. If it doesn't exists download it.
    cache_path = os.path.join("data", "iot")
    cache_file = os.path.join(cache_path, "solar.csv")
    if not os.path.exists(cache_path):
        os.makedirs(cache_path)
    if not os.path.exists(cache_file):
        urlretrieve(input_url, cache_file)
        print("downloaded data successfully from ", input_url)
    else:
        print("using cache for ", input_url)

    df = pd.read_csv(cache_file, index_col="time", parse_dates=['time'], dtype=np.float32)

    df["date"] = df.index.date

    # normalize data
    df['solar.current'] /= normalize
    df['solar.total'] /= normalize

    # group by day, find the max for a day and add a new column .max
    grouped = df.groupby(df.index.date).max()
    grouped.columns = ["solar.current.max", "solar.total.max", "date"]

    # merge continuous readings and daily max values into a single frame
    df_merged = pd.merge(df, grouped, right_index=True, on="date")
    df_merged = df_merged[["solar.current", "solar.total",
                           "solar.current.max", "solar.total.max"]]
    # we group by day so we can process a day at a time.
    grouped = df_merged.groupby(df_merged.index.date)
    per_day = []
    for _, group in grouped:
        per_day.append(group)

    # split the dataset into train, validatation and test sets on day boundaries
    val_size = int(len(per_day) * val_size)
    test_size = int(len(per_day) * test_size)
    next_val = 0
    next_test = 0

    result_x = {"train": [], "val": [], "test": []}
    result_y = {"train": [], "val": [], "test": []}

    # generate sequences a day at...


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
Let us evaluate our performance for the training, validation, and test datasets. We use mean squared error as our metric which might be a little simplistic. A method that would define a ratio of how many predictions have been inside a given tolerance would make a better measure.
 Suggested Exploration:  Try out absolute error (C.abs(z-l)) instead of a squared error.
Can you write Python code for it?

# evaluate the specified X and Y data on our model
def get_error(X,Y,labeltxt):
    result = 0.0
    numbatches = 0
    for x1, y1 in next_batch(X, Y, labeltxt):
        eval_error = trainer.test_minibatch({x : x1, l : y1})
        result += eval_error
        numbatches += 1

    return float(result)/numbatches

# Print the training and validation errors
for labeltxt in ["train", "val"]:
    print("Error for {}: {:.8f}".format(labeltxt, get_error(X, Y, labeltxt)))

# Print the test error
labeltxt = "test"
print("Error for {}: {:.8f}".format(labeltxt, get_error(X, Y, labeltxt)))


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
There are two training modes that we can choose from for this lab:
- Fast mode: isFast is set to True. This is the default mode for the notebooks, which means we train for fewer iterations or train / test on limited data. This ensures functional correctness of the notebook though the models produced are far from what a completed training would produce.

Slow mode: We recommend the user to set this flag to False once the user has gained familiarity with the notebook content and wants to gain insight from running the notebooks for a longer period with different parameters for training.

For Fast mode we train the model for 100 epochs and results have low accuracy but is good enough for development. The model yields good accuracy after 1000-2000 epochs.
Can you write Python code for it?

isFast = True

# we need around 2000 epochs to see good accuracy. For testing 100 epochs will do.
EPOCHS = 200 if isFast else 2000


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
Training
Before we can start training we need to bind our input variables for the model and define which optimizer we want to use. For this example we choose the adam optimizer. We choose squared_error as our loss function.
Can you write Python code for it?

# input sequences
x = C.sequence.input_variable(1)

# create the model
z = create_model(x)

# expected output (label), also the dynamic axes of the model output
# is specified as the model of the label input
l = C.input_variable(1, dynamic_axes=z.dynamic_axes, name="y")

# the learning rate
learning_rate = 0.005
lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)

# loss and error function
loss = C.squared_error(z, l)
error = C.squared_error(z, l)
# loss = C.abs(z-l)
# error = C.abs(z-l)

# use adam optimizer
momentum_time_constant = C.momentum_as_time_constant_schedule(BATCH_SIZE / -math.log(0.9))
learner = C.fsadagrad(z.parameters,
                      lr = lr_schedule,
                      momentum = momentum_time_constant)

trainer = C.Trainer(z, (loss, error), [learner])


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
Understand the data format
You can now see the sequence we are going to feed to the LSTM. Note if we have less than 8 datapoints for a day we skip over the day assuming something is missing in the raw data. If we get more than 14 data points in a day we truncate the readings.
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
Utility for data fetching
next_batch() yields the next batch for training. We use variable size sequences supported by CNTK and batches are a list of numpy arrays where the numpy arrays have variable length.
A standard practice is to shuffle batches with each epoch. We don't do this here because we want to be able to graph the data that is easily interpretable.
Can you write Python code for it?

# process batches of 10 days
BATCH_SIZE = TIMESTEPS * 10

def next_batch(x, y, ds):
    """get the next batch for training"""

    def as_batch(data, start, count):
        return data[start:start + count]

    for i in range(0, len(x[ds]), BATCH_SIZE):
        yield as_batch(X[ds], i, BATCH_SIZE), as_batch(Y[ds], i, BATCH_SIZE)


I am working on the file "DAT236x Deep Learning Explained/Lab5_RecurrentNetwork.ipynb".
The first task is:
Visualize the prediction
Our model has been trained well, given that the training, validation and test errors are in the same ballpark. To better understand our predictions, let's visualize the results.  We will take our newly created model, make predictions and plot them against the actual readings.
Can you write Python code for it?

# predict
f, a = plt.subplots(2, 1, figsize=(12, 8))
for j, ds in enumerate(["val", "test"]):
    results = []
    for x_batch, _ in next_batch(X, Y, ds):
        pred = z.eval({x: x_batch})
        results.extend(pred[:, 0])
    # because we normalized the input data we need to multiply the prediction
    # with SCALER to get the real values.
    a[j].plot((Y[ds] * NORMALIZE).flatten(), label=ds + ' raw');
    a[j].plot(np.array(results) * NORMALIZE, label=ds + ' pred');
    a[j].legend();


I am working on the file "DAT236x Deep Learning Explained/Lab6_TextClassification_with_LSTM.ipynb".
The first task is:
A Word About Sequential()
Before jumping to the tasks, let's have a look again at the model we just ran.
The model is described in what we call function-composition style.
python
        Sequential([
            Embedding(emb_dim),
            Recurrence(LSTM(hidden_dim), go_backwards=False),
            Dense(num_labels)
        ])
You may be familiar with the "sequential" notation from other neural-network toolkits.
If not, Sequential() is a powerful operation that,
in a nutshell, allows to compactly express a very common situation in neural networks
where an input is processed by propagating it through a progression of layers.
Sequential() takes an list of functions as its argument,
and returns a new function that invokes these functions in order,
each time passing the output of one to the next.
For example,
python
    FGH = Sequential ([F,G,H])
    y = FGH (x)
means the same as
y = H(G(F(x)))
This is known as "function composition",
and is especially convenient for expressing neural networks, which often have this form:
     +-------+   +-------+   +-------+
x -->|   F   |-->|   G   |-->|   H   |--> y
     +-------+   +-------+   +-------+

Coming back to our model at hand, the Sequential expression simply
says that our model has this form:
     +-----------+   +----------------+   +------------+
x -->| Embedding |-->| Recurrent LSTM |-->| DenseLayer |--> y
     +-----------+   +----------------+   +------------+

Task 1: Add a Lookahead
Our recurrent model suffers from a structural deficit:
Since the recurrence runs from left to right, the decision for a slot label
has no information about upcoming words. The model is a bit lopsided.
Your task will be to modify the model such that
the input to the recurrence consists not only of the current word, but also of the next one
(lookahead).
Your solution should be in function-composition style.
Hence, you will need to write a Python function that does the following:

takes no input arguments
creates a placeholder (sequence)...
Can you write Python code for it?

# Your task: Add lookahead
def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels)
        ])

# Enable these when done:
#z = create_model()
#do_train_test()


I am working on the file "DAT236x Deep Learning Explained/Lab6_TextClassification_with_LSTM.ipynb".
The first task is:
Importing CNTK and other useful libraries
CNTK's Python module contains several submodules like io, learner, and layers. We also use NumPy in some cases since the results returned by CNTK work like NumPy arrays.
Can you write Python code for it?

import math
import numpy as np

import cntk as C


I am working on the file "DAT236x Deep Learning Explained/Lab6_TextClassification_with_LSTM.ipynb".
The first task is:
Our input text words will be encoded as one-hot vectors of length 943 and the output dimension of our model emb_dim is set to 150. In the code below we pass the input variable x to our model z. This binds the model with input data of known shape. In this case, the input shape will be the size of the input vocabulary. With this modification, the parameter returned by the embed layer is completely specified (943, 150).
Note: As an alternative to our approach here, you can initialize the Embedding matrix with pre-computed vectors using Word2Vec or GloVe.
Can you write Python code for it?

# Pass an input and check the dimension
z = create_model()
print(z(x).embed.E.shape)