I am working on the file "RNN with Keras - Text Generation.ipynb".
The first task is:
Data loading and preprocessing
First we are going to load our dataset and preprocess it such it can be fed to the model. Notice we are working at word level. Moving to character level would require some adjustment to the process and overall model.
These are the common steps you should follow to create your training data from your original dataset:
* sentence segmentation (if you don't have already separate individual sentences)
* sentence tokenization (from sentence to list of words)
* add start and end tokens
* generate words indexing
* pad sequences (pad or truncate sentences to fixed length)
* one-hot encode (if you are not going to use an embedding layer in the keras model)
Can you write Python code for it?

# load dataset with pickle
corpus_name = "short_oneliners"
with open("resources/short_oneliners.pickle", 'rb') as f: #binary mode (b) is required for pickle
    dataset = pickle.load(f, encoding='utf-8') #our dataset is simply a list of string
print('Loaded {} sentences\nExample: "{}"'.format(len(dataset), dataset[0]))

# constant token and params for our models
START_TOKEN = "SENTENCE_START"
END_TOKEN = "SENTENCE_END"
UNKNOWN_TOKEN = "UNKNOWN_TOKEN"
PADDING_TOKEN = "PADDING"

vocabulary_size = 5000
sent_max_len = 20

# work tokenization for each sentence, while adding start and end tokens
sentences = [[START_TOKEN] + nltk.word_tokenize(entry.lower()) + [END_TOKEN] for entry in dataset]
print('Example: {}'.format(sentences[0]))

# creates index_to_word and word_to_index mappings, given the data and a max vocabulary size
def get_words_mappings(tokenized_sentences, vocabulary_size):
    # we can rely on nltk to quickly get the most common words, and then limit our vocabulary to the specified size
    frequence = nltk.FreqDist(itertools.chain(*tokenized_sentences))
    vocab = frequence.most_common(vocabulary_size)
    index_to_word = [x[0] for x in vocab]
    # Add padding for index 0
    index_to_word.insert(0, PADDING_TOKEN)
    # Append unknown token (with index = vocabulary size + 1)
    index_to_word.append(UNKNOWN_TOKEN)
    word_to_index = dict([(w,i) for i,w in enumerate(index_to_word)])
    return index_to_word, word_to_index

# get mappings and update vocabulary size
index_to_word, word_to_index = get_words_mappings(sentences, vocabulary_size)
vocabulary_size = len(index_to_word)
print("Vocabulary size = " + str(vocabulary_size))

# Generate training data by converting tokenized sentenced to indexes (and replacing unknown words)
train_size = min(len(sentences), 100000)
train_data = [[word_to_index.get(w,word_to_index[UNKNOWN_TOKEN])  for w in sent] for sent in sentences[:train_size]]

# pad sentences to fixed lenght (pad with 0s if shorter, truncate if longer)
train_data =...


I am working on the file "RNN with Keras - Text Generation.ipynb".
The first task is:
Model training and evaluation
We are going to define a RNN model architecture, train it on our data, and eventually save the results for future usage.
Can you write Python code for it?

from keras.models import Sequential
from keras.layers import Dense, Dropout, BatchNormalization
from keras.layers.core import Activation, Flatten
from keras.layers.embeddings import Embedding
from keras.layers import LSTM, TimeDistributed

# Define model and parameters
hidden_size = 512
embedding_size = 128

# model with embedding
model = Sequential()
model.add(Embedding(vocabulary_size, embedding_size, mask_zero=True))
# add batch norm
model.add(TimeDistributed(Flatten()))
model.add(LSTM(hidden_size, return_sequences=True, activation='relu'))
model.add(TimeDistributed(Dense(vocabulary_size, activation='softmax')))

# basic single-layer model
#model = Sequential()
#model.add(LSTM(hidden_size, input_shape=(None, vocabulary_size), return_sequences=True))
#model.add(TimeDistributed(Dense(vocabulary_size, activation='softmax')))

# need time distributes for embedding?
#model.add(TimeDistributed(
#        Embedding(vocabulary_size, output_dim=hidden_size, mask_zero=True, input_length=sent_max_len-1),
#          input_shape=(sent_max_len-1, 1), input_dtype='int32'))

model.summary()

# recompile also if you just want to keep training a model just loaded from memory
loss = 'sparse_categorical_crossentropy'
optimizer = 'adam'
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

# Train model
# you might want to train several times on few epoch, observe how loss and metrics vary
# and possibly tweak batch size and learning rate
num_epoch = 2
batch_size = 32
model.fit(X_train, y_train, epochs=num_epoch, batch_size=batch_size, verbose=1)

# export model (architecture)
model_path = "resources/models/{}_vocab_{}.json".format(corpus_name, vocabulary_size)
model_json = model.to_json()
with open(model_path, "w") as f:
    f.write(model_json)

# export model weights
weights_path = "resources/models/{}_epoch_{}.hdf5".format(corpus_name, 40)
model.save_weights(weights_path)

# export word indexes
index_to_word_path = 'resources/models/{}_idxs_vocab{}.txt'.format(corpus_name, vocabulary_size)
with...


I am working on the file "RNN with Keras - Text Generation.ipynb".
The first task is:
Text Generations
For the generation part I am relying on a generic utility class (TextGenModel) included in this repo.
Main operations for this step are:
* load previously trained model
* instantiate class with the model and model configuration (e.g. temperature, sent max length)
* generate new text with target class
* prettify generated text
For the text generation task a seed sentence can be provided, plus additional requirements on text length.
The class will then internally take care of predicting word after word until some criteria are met.
Can you write Python code for it?

model_path = "resources/models/jokes_vocab_5002.json"
weights_path = "resources/models/jokes_epoch_20.hdf5"
# Load previously saved model
with open(model_path, 'r') as f:
    model = model_from_json(f.read())
# Load weights into model
model.load_weights(weights_path)
# Load word indexes
with open(index_to_word_path, 'rb') as f:
    index_to_word = pickle.load(f, encoding='utf-8')

import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# instantiate generation class on our data
text_gen = TextGenModel(model, index_to_word, word_to_index, sent_max_len=sent_max_len,
                                    temperature=1.0,
                                    use_embeddings=True)
# generate N new sentences
n_sents = 10
original_sentences = [entry.lower() for entry in dataset]
for _ in range(n_sents):
    res = text_gen.pretty_print_sentence(text_gen.get_sentence(15), skip_last=True)
    if res in original_sentences:
        print("* {} SAME".format(res))
    else:
        print(res)


I am working on the file "RNN.ipynb".
The first task is:
Backward pass
The goal of the backward pass is to calculate all gradients.
Gradients tell in witch direction you have to move your parameter to make a better model.
The naive way to calculate all gradients would be to recalculate a loss for small variations for each parameters.
This is possible but would be time consuming. We have more than 20k parameters.
There is a technic to calculates all the gradients for all the parameters at once: the backdrop propagation.
Gradients are calculated in the oposite order of the forward pass, using simple technics.
For instance if we have:
python
loss = a.x + b
If we want to minimize loss, we need to calculate d(loss)/dx and use it to calculate the new_x value.
python
new_x = x - d(loss)/dx * step_size
If new_loss is smaller than loss, it is a win: we succeed to find a better x input.
Lets do the math:
d(loss)/dx = d(a.x)/dx +d(b)/dx
d(loss)/dx = (d(a)/dx)1 + ad(x)/dx + 0
d(loss)/dx = 0 + a*1
d(loss)/dx = a
Can you write Python code for it?

x = 10
a = 3
b = 7

loss = a+x + b
print 'initial loss =', loss
# dx stand for d(loss)/dx
dx = a #Calculate dx=d(loss)/dx analytically
step_size = 0.1
# use dx and step size to calculate new x
new_x = x - dx * step_size
new_loss = a+new_x + b
print 'new loss =',new_loss
if (new_loss<loss): print 'New loss is smaller, Yeah!'


I am working on the file "RNN.ipynb".
The first task is:
Encode/Decode char/vector
Neural networks can only works on vectors. (a vector is an array of float)
So we need a way to encode and decode a char as a vector.
For this we count the number of unique char (vocab_size). It will be the size of the vector.
The vector contain only zero exept for the position of the char wherae the value is 1.
First we calculate vocab_size:
Can you write Python code for it?

chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
print 'data has %d characters, %d unique.' % (data_size, vocab_size)


I am working on the file "RNN.ipynb".
The first task is:
Finaly we create a vector from a char like this:
The dictionary defined above allow us to create a vector of size 61 instead of 256.
Here and exemple for char 'a'
The vector contains only zero, except at position char_to_ix['a'] where we put a 1.
Can you write Python code for it?

%matplotlib notebook

import matplotlib
import matplotlib.pyplot as plt

vector_for_char_a = np.zeros((vocab_size, 1))
vector_for_char_a[char_to_ix['a']] = 1
#print vector_for_char_a
print vector_for_char_a.ravel()

x = range(0,len(chars))
plt.figure(figsize=(10,2))
plt.bar(x, vector_for_char_a.ravel(), 0.3)
plt.xticks(x, chars)
plt.show()


I am working on the file "RNN.ipynb".
The first task is:
Forward pass
The forward pass use the parameters of the model (Wxh, Whh, Why, bh, by) to calculate the next char given a char from the trainning set.
xs[t] is the vector that encode the char at position t
ps[t] is the probabilities for next char
python
hs[t] = np.tanh(np.dot(Wxh, xs[t]) + np.dot(Whh, hs[t-1]) + bh) # hidden state
ys[t] = np.dot(Why, hs[t]) + by # unnormalized log probabilities for next chars
ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # probabilities for next chars
or is dirty pseudo code for each char
python
hs = input*Wxh + last_value_of_hidden_state*Whh + bh
ys = hs*Why + by
ps = normalized(ys)
To dive into the code, we'll work on one char only (we set t=0 ; instead of the "for each" loop).
Can you write Python code for it?

# uncomment the print to get some details
xs, hs, ys, ps = {}, {}, {}, {}
hs[-1] = np.copy(hprev)
# forward pass
t=0 # for t in xrange(len(inputs)):
xs[t] = np.zeros((vocab_size,1)) # encode in 1-of-k representation
xs[t][inputs[t]] = 1
# print xs[t]
hs[t] = np.tanh(np.dot(Wxh, xs[t]) + np.dot(Whh, hs[t-1]) + bh) # hidden state
ys[t] = np.dot(Why, hs[t]) + by # unnormalized log probabilities for next chars
# print ys[t]
ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # probabilities for next chars
# print ps[t].ravel()

# Let's build a dict to see witch probablity is associated with witch char
probability_per_char =  { ch:ps[t].ravel()[i] for i,ch in enumerate(chars) }
# uncoment the next line to see the raw result
# print probability_per_char

# To print the probability in a way that is more easy to read.
for x in range(vocab_size):
    print 'p('+ ix_to_char[x] + ")=", "%.4f" % ps[t].ravel()[x],
    if (x%7==0):
        print ""
    else:
        print "",

x = range(0,len(chars))
plt.figure(figsize=(10,5))
plt.bar(x, ps[t], 0.3)
plt.xticks(x, chars)
plt.show()

# We can create the next char from the above distribution
ix = np.random.choice(range(vocab_size), p=ps[t].ravel())
print
print "Next char code is:", ix
print "Next char is:", ix_to_char[ix]


I am working on the file "RNN.ipynb".
The first task is:
The model parameters are adjusted during the trainning.
* Wxh are parameters to connect a vector that contain one input to the hidden layer.
* Whh are parameters to connect the hidden layer to itself. This is the Key of the Rnn: Recursion is done by injecting the previous values from the output of the hidden state, to itself at the next iteration.
* Why are parameters to connect the hidden layer to the output
* bh contains the hidden bias
* by contains the output bias
You'll see in the next section how theses parameters are used to create a sentence.
Create a sentence from the model
Can you write Python code for it?

def sample(h, seed_ix, n):
  """
  sample a sequence of integers from the model
  h is memory state, seed_ix is seed letter for first time step
  """
  x = np.zeros((vocab_size, 1))
  x[seed_ix] = 1
  ixes = []
  for t in xrange(n):
    h = np.tanh(np.dot(Wxh, x) + np.dot(Whh, h) + bh)
    y = np.dot(Why, h) + by
    p = np.exp(y) / np.sum(np.exp(y))
    ix = np.random.choice(range(vocab_size), p=p.ravel())
    x = np.zeros((vocab_size, 1))
    x[ix] = 1
    ixes.append(ix)
  txt = ''.join(ix_to_char[ix] for ix in ixes)
  print '----\n %s \n----' % (txt, )

hprev = np.zeros((hidden_size,1)) # reset RNN memory
sample(hprev,char_to_ix['a'],200)


I am working on the file "RNN.ipynb".
The first task is:
Then we create 2 dictionary to encode and decode a char to an int
Can you write Python code for it?

char_to_ix = { ch:i for i,ch in enumerate(chars) }
ix_to_char = { i:ch for i,ch in enumerate(chars) }
print char_to_ix
print ix_to_char


I am working on the file "RNN.ipynb".
The first task is:
goal is to calculate gradients for the forward formula:
python
hs = input*Wxh + last_value_of_hidden_state*Whh + bh
ys = hs*Why + by
This part need more work to explain the code, but here a great source to understand this technic in detail.
```python
Backdrop this: ys = hs*Why + by
dy=-1 # because the smaller the loss, the better is the model.
dWhy = np.dot(dy, hs.T)
dby = dy
dh = np.dot(Why.T, dy) + dhnext # backprop into h
dhraw = (1 - hs * hs) * dh # backprop through tanh nonlinearity
Backdrop this: hs = inputWxh + last_value_of_hidden_stateWhh + bh
dbh += dhraw
dWxh += np.dot(dhraw, xs.T)
dWhh += np.dot(dhraw, hs.T)
dhnext = np.dot(Whh.T, dhraw)
```
Can you write Python code for it?

# backward pass: compute gradients going backwards
dWxh, dWhh, dWhy = np.zeros_like(Wxh), np.zeros_like(Whh), np.zeros_like(Why)
dbh, dby = np.zeros_like(bh), np.zeros_like(by)
dhnext = np.zeros_like(hs[0])
t=0 #for t in reversed(xrange(len(inputs))):
dy = np.copy(ps[t])
dy[targets[t]] -= 1 # backprop into y
#print dy.ravel()
dWhy += np.dot(dy, hs[t].T)
#print dWhy.ravel()
dby += dy
#print dby.ravel()
dh = np.dot(Why.T, dy) + dhnext # backprop into h
dhraw = (1 - hs[t] * hs[t]) * dh # backprop through tanh nonlinearity
dbh += dhraw
dWxh += np.dot(dhraw, xs[t].T)
dWhh += np.dot(dhraw, hs[t-1].T)
dhnext = np.dot(Whh.T, dhraw)
for dparam in [dWxh, dWhh, dWhy, dbh, dby]:
  np.clip(dparam, -5, 5, out=dparam) # clip to mitigate exploding gradients
  #print dparam


I am working on the file "RNN101/Customized RNN.ipynb".
The first task is:
Create sample sequence
Can you write Python code for it?

sample_seq = np.array([[1,0],[0,1],[0,1]],dtype=np.float64)
sample_seq = np.concatenate([sample_seq]*(30), axis=0)
print("Sample sequence:\n{}".format(sample_seq))
train_input = sample_seq[0:5,:]
train_output = sample_seq[1:6,:]
test_input = sample_seq[:-1,:]
test_output = sample_seq[1:,:]


I am working on the file "RNN101/Customized RNN.ipynb".
The first task is:
Define MyRnnCell
The following property/methods should be correctly defined for an RNN cell.
* _call_ (method)
* output_size (property)
* state_size (property)
Can you write Python code for it?

class MyRnnCell(tf.nn.rnn_cell.RNNCell):
    def __init__(self, state_size, dtype):
        self._state_size = state_size
        self._dtype = dtype
        self._W_xh = tf.get_variable(shape=[self._state_size, self._state_size],
                                     dtype=self._dtype, name="W_xh", initializer=tf.truncated_normal_initializer())
        self._W_hh = tf.get_variable(shape=[self._state_size, self._state_size],
                                     dtype=self._dtype, name="W_hh", initializer=tf.truncated_normal_initializer())
        self._W_ho = tf.get_variable(shape=[self._state_size, self._state_size],
                                     dtype=self._dtype, name="W_ho", initializer=tf.truncated_normal_initializer())
        self._b_o = tf.get_variable(shape=[self._state_size], dtype=self._dtype,
                                    name="b_o", initializer=tf.truncated_normal_initializer())

    def __call__(self, _input, state, scope=None):
        new_state = tf.tanh(tf.matmul(_input, self._W_xh)+tf.matmul(state, self._W_hh))
        new_output = tf.tanh(tf.matmul(new_state, self._W_ho)+self._b_o)
        return new_output, new_state

    @property
    def output_size(self):
        return self._state_size

    @property
    def state_size(self):
        return self._state_size


I am working on the file "RNN101/Customized RNN.ipynb".
The first task is:
Training & Testing
Can you write Python code for it?

#state = np.zeros([1, 2])
inputs = tf.placeholder(shape=[None, 2], dtype=tf.float64)
targets = tf.placeholder(shape=[None, 2], dtype=tf.float64)
# One batch only
batch_inputs = tf.reshape(inputs, shape=np.array([1, -1, 2]))
outputs, final_state = tf.nn.dynamic_rnn(test_cell, batch_inputs, dtype=tf.float64)
# de-batch
outputs = tf.reshape(outputs, shape=[-1, 2])
loss = tf.nn.softmax_cross_entropy_with_logits(labels=targets, logits=outputs)
optimize_op = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(loss)
print("Training network")
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(20000):
        sess.run([optimize_op, outputs], feed_dict={inputs: train_input, targets: train_output})
    print("Testing network with input:\n{}".format(test_input))
    print("Expected outputs:\n{}\nNetwork activations:\n{}".format(test_output,
                                                                   sess.run(outputs, feed_dict={inputs: test_input})))


I am working on the file "RNN101/Text Generator.ipynb".
The first task is:
Check the generator
Can you write Python code for it?

seq_length = 100
reset_every = 2
batch_size = 2
batches = batch_generator(data=encoded_data,
                               codec=test_codec,
                               batch_size=batch_size,
                               seq_length=seq_length,
                               reset_every=reset_every)
for (x, y), i in zip(batches, range(reset_every * 2)):
    print("Batch {}".format(i))
    if (i % reset_every) == 0:
        print("Reset")
    for j in range(batch_size):
        decoded_x, decoded_y = test_codec.decode(x[j], strip=False), test_codec.decode(y[j], strip=False)
        print("Index of sub-sequence:\n{}\nSequence input:\n{}:\nSequence output:\n{}".format(j,
                                                                                decoded_x,
                                                                                decoded_y))
del seq_length, reset_every, batch_size, batches


I am working on the file "RNN101/Text Generator.ipynb".
The first task is:
Configurations
Can you write Python code for it?

vocab = (" $%'()+,-./0123456789:;=?ABCDEFGHIJKLMNOPQRSTUVWXYZ"
            "\\^_abcdefghijklmnopqrstuvwxyz{|}\n")
graph_path = r"./graphs"
test_text_path = os.path.normpath(r"../Dataset/arvix_abstracts.txt")
batch_size=50
model_param_path=os.path.normpath(r"./model_checkpoints")


I am working on the file "RNN101/Text Generator.ipynb".
The first task is:
Define model class
Can you write Python code for it?

class DRNN(tf.nn.rnn_cell.RNNCell):
    def __init__(self, input_dim, hidden_dim, output_dim, num_hidden_layer, dtype=tf.float32):
        super(tf.nn.rnn_cell.RNNCell, self).__init__(dtype=dtype)
        assert type(input_dim) == int and input_dim > 0, "Invalid input dimension. "
        self._input_dim = input_dim
        assert type(num_hidden_layer) == int and num_hidden_layer > 0, "Invalid number of hidden layer. "
        self._num_hidden_layer = num_hidden_layer
        assert type(hidden_dim) == int and hidden_dim > 0, "Invalid dimension of hidden states. "
        self._hidden_dim = hidden_dim
        assert type(output_dim) == int and output_dim > 0, "Invalid dimension of output dimension. "
        self._output_dim = output_dim
        self._state_is_tuple = True
        with tf.variable_scope("input_layer"):
            self._W_xh = tf.get_variable("W_xh", shape=[self._input_dim, self._hidden_dim])
            self._b_xh = tf.get_variable("b_xh", shape=[self._hidden_dim])
        with tf.variable_scope("rnn_layers"):
            self._cells = [tf.nn.rnn_cell.GRUCell(self._hidden_dim) for _ in range(num_hidden_layer)]
        with tf.variable_scope("output_layer"):
            self._W_ho_list = [tf.get_variable("W_h{}o".format(i), shape=[self._hidden_dim, self._output_dim])
                               for i in range(num_hidden_layer)]
            self._b_ho = tf.get_variable("b_ho", shape=[self._output_dim])

    @property
    def output_size(self):
        return self._output_dim

    @property
    def state_size(self):
        return (self._hidden_dim,) * self._num_hidden_layer

    def zero_state(self, batch_size, dtype):
        if self._state_is_tuple:
            return tuple(cell.zero_state(batch_size, dtype)for cell in self._cells)
        else:
            raise NotImplementedError("Not implemented yet.")

    def __call__(self, _input, state, scope=None):
        assert type(state) == tuple and len(state) == self._num_hidden_layer, "state must be a tuple of size {}".format(
           ...


I am working on the file "RNN101/Text Generator.ipynb".
The first task is:
Load data set
Can you write Python code for it?

with open(test_text_path, "r") as f:
    raw_text_list = "".join(f.readlines()).split("\n")
print("Loaded abstract from a total of {} theses.".format(len(raw_text_list)))
# See what we have loaded
sample_text_no = random.randint(0, len(raw_text_list)-1)
sample_text_raw = raw_text_list[sample_text_no]
print("A sample text in the data set:\n{}".format(sample_text_raw))
sample_text_encoded=test_codec.encode(sample_text_raw)
print("Encoded text:\n{}".format(sample_text_encoded))
print("Decoded text:\n{}".format(test_codec.decode(sample_text_encoded)))
encoded_data = test_codec.encode("\n".join(raw_text_list), start=False, stop=False)


I am working on the file "RNN101/Text Generator.ipynb".
The first task is:
Test
See how encoding and decoding work.
Can you write Python code for it?

test_codec=TextCodec(vocab)
test_text_encoded=test_codec.encode("Hello world!")
print("Encoded text looks like:\n{}".format(test_text_encoded))
test_text_decoded=test_codec.decode(nparray=test_text_encoded,strip=False)
print("Decoded text looks like:\n{}".format(test_text_decoded))


I am working on the file "RNN101/Text Generator.ipynb".
The first task is:
Test online inference
Can you write Python code for it?

def online_inference(cell, prime, sess, codec,
                     input_tensor,
                     init_state_tensor_tuple,
                     output_tensor,
                     final_state_tensor_tuple,
                     length):
    final_output = [prime]
    zero_states = sess.run(cell.zero_state(batch_size=1, dtype=tf.float32))
    feed_dict = {input_tensor: codec.encode(prime, start=True, stop=False)[np.newaxis, :, :]}  # prime
    for init_state_tensor, init_state_value in zip(init_state_tensor_tuple,
                                                   zero_states):
        feed_dict[init_state_tensor] = init_state_value
    output, final_states = sess.run([output_tensor, final_state_tensor_tuple], feed_dict=feed_dict)
    final_output.append(codec.decode(output[0, -1:, :], strip=False))
    for _ in range(length - len(prime)):
        feed_dict = {input_tensor: codec.encode(final_output[-1], start=False, stop=False)[np.newaxis, :, :]}
        for init_state_tensor, init_state_value in zip(init_state_tensor_tuple, final_states):
            feed_dict[init_state_tensor] = init_state_value
        output, final_states = sess.run([output_tensor, final_state_tensor_tuple], feed_dict=feed_dict)
        final_output.append(codec.decode(output[0], strip=False))
    return "".join(final_output)

saver = tf.train.Saver()
with tf.Session() as sess:
    ckpt = saver.last_checkpoints
    print(ckpt)
    print(online_inference(rnn_cell, "We propose",
                       sess, test_codec, seq_input, init_state, seq_output, final_states, 200))


I am working on the file "RNNLM.ipynb".
The first task is:
A few selected (censored) sentences. I added capitalization.

Anyway, to the city scene you're an idiot teenager.
What ? ! ! ! ! ignore!
Screw fitness, you're saying: https
Thanks for the advice to keep my thoughts around girls.
Yep, please disappear with the terrible generation.

Looking at the generated sentences there are a few interesting things to note. The model successfully learn syntax. It properly places commas (usually before and's and or's) and ends sentence with punctuation. Sometimes it mimics internet speech such as multiple exclamation marks or smileys.
However, the vast majority of generated sentences don't make sense or have grammatical errors. One reason could be that we did not train our network long enough (or didn't use enough training data). That may be true, but it's most likely not the main reason. Our vanilla RNN  can't generate meaningful text because it's unable to learn dependencies between words that are several steps apart. That's also why RNNs failed to gain popularity when they were first invented. They were beautiful in theory but didn't work well in practice, and we didn't immediately understand why.
Fortunately, the difficulties in training RNNs are much better understood now. In the next part of this tutorial we will explore the Backpropagation Through Time (BPTT) algorithm in more detail and demonstrate what's called the vanishing gradient problem. This will motivate our move to more sophisticated RNN models, such as LSTMs, which are the current state of the art for many tasks in NLP (and can generate much better reddit comments!).  Everything you learned in this tutorial also applies to LSTMs and other RNN models, so don't feel discouraged if the results for a vanilla RNN are worse then you expected.
Can you write Python code for it?

%run train.py

%run NER.py

%run train.py


I am working on the file "RNNLM.ipynb".
The first task is:
Above, word_dim is the size of our vocabulary, and hidden_dim is the size of our hidden layer (we can pick it). Don't worry about the bptt_truncate parameter for now, we'll explain what that is later.
Forward Propagation
Next, let's implement the forward propagation (predicting word probabilities) defined by our equations above:
Can you write Python code for it?

def forward_propagation(self, x):
    # The total number of time steps
    T = len(x)
    # During forward propagation we save all hidden states in s because need them later.
    # We add one additional element for the initial hidden, which we set to 0
    s = np.zeros((T + 1, self.hidden_dim))
    s[-1] = np.zeros(self.hidden_dim)
    # The outputs at each time step. Again, we save them for later.
    o = np.zeros((T, self.word_dim))
    # For each time step...
    for t in np.arange(T):
        # Note that we are indxing U by x[t]. This is the same as multiplying U with a one-hot vector.
        s[t] = np.tanh(self.U[:,x[t]] + self.W.dot(s[t-1]))
        o[t] = softmax(self.V.dot(s[t]))
    return [o, s]

RNNNumpy.forward_propagation = forward_propagation


I am working on the file "RNNLM.ipynb".
The first task is:
Building the RNN
For a general overview of RNNs take a look at first part of the tutorial.

Let's get concrete and see what the RNN for our language model looks like. The input $x$ will be a sequence of words (just like the example printed above) and each $x_t$ is a single word. But there's one more thing: Because of how matrix multiplication works we can't simply use a word index (like 36) as an input. Instead, we represent each word as a one-hot vector of size vocabulary_size. For example, the word with index 36 would be the vector of all 0's and a 1 at position 36. So, each $x_t$ will become a vector, and $x$ will be a matrix, with each row representing a word. We'll perform this transformation in our Neural Network code instead of doing it in the pre-processing. The output of our network $o$ has a similar format. Each $o_t$ is a vector of vocabulary_size elements, and each element represents the probability of that word being the next word in the sentence.
Let's recap the equations for the RNN from the first part of the tutorial:
$
\begin{aligned}
s_t &= \tanh(Ux_t + Ws_{t-1}) \
o_t &= \mathrm{softmax}(Vs_t)
\end{aligned}
$
I always find it useful to write down the dimensions of the matrices and vectors. Let's assume we pick a vocabulary size $C = 8000$ and a hidden layer size $H = 100$. You can think of the hidden layer size as the "memory" of our network. Making it bigger allows us to learn more complex patterns, but also results in additional computation. Then we have:
$
\begin{aligned}
x_t & \in \mathbb{R}^{8000} \
o_t & \in \mathbb{R}^{8000} \
s_t & \in \mathbb{R}^{100} \
U & \in \mathbb{R}^{100 \times 8000} \
V & \in \mathbb{R}^{8000 \times 100} \
W & \in \mathbb{R}^{100 \times 100} \
\end{aligned}
$
This is valuable information. Remember that $U,V$ and $W$ are the parameters of our network we want to learn from data. Thus, we need to learn a total of $2HC + H^2$ parameters. In the case of $C=8000$ and $H=100$ that's 1,610,000.  The dimensions also tell us the bottleneck of our model. Note that...
Can you write Python code for it?

class RNNNumpy:

    def __init__(self, word_dim, hidden_dim=100, bptt_truncate=4):
        # Assign instance variables
        self.word_dim = word_dim
        self.hidden_dim = hidden_dim
        self.bptt_truncate = bptt_truncate
        # Randomly initialize the network parameters
        self.U = np.random.uniform(-np.sqrt(1./word_dim), np.sqrt(1./word_dim), (hidden_dim, word_dim))
        self.V = np.random.uniform(-np.sqrt(1./hidden_dim), np.sqrt(1./hidden_dim), (word_dim, hidden_dim))
        self.W = np.random.uniform(-np.sqrt(1./hidden_dim), np.sqrt(1./hidden_dim), (hidden_dim, hidden_dim))


I am working on the file "RNNLM.ipynb".
The first task is:
Calculating the Loss
To train our network we need a way to measure the errors it makes. We call this the loss function $L$, and our goal is find the parameters $U,V$ and $W$ that minimize the loss function for our training data. A common choice for the loss function is the cross-entropy loss. If we have $N$ training examples (words in our text) and $C$ classes (the size of our vocabulary) then the loss with respect to our predictions $o$ and the true labels $y$ is given by:
$
\begin{aligned}
L(y,o) = - \frac{1}{N} \sum_{n \in N} y_{n} \log o_{n}
\end{aligned}
$
The formula looks a bit complicated, but all it really does is sum over our training examples and add to the loss based on how off our prediction are. The further away $y$ (the correct words) and $o$ (our predictions), the greater the loss will be. We implement the function calculate_loss:
Can you write Python code for it?

def calculate_total_loss(self, x, y):
    L = 0
    # For each sentence...
    for i in np.arange(len(y)):
        o, s = self.forward_propagation(x[i])
        # We only care about our prediction of the "correct" words
        correct_word_predictions = o[np.arange(len(y[i])), y[i]]
        # Add to the loss based on how off we were
        L += -1 * np.sum(np.log(correct_word_predictions))
    return L

def calculate_loss(self, x, y):
    # Divide the total loss by the number of training examples
    N = np.sum((len(y_i) for y_i in y))
    return self.calculate_total_loss(x,y)/N

RNNNumpy.calculate_total_loss = calculate_total_loss
RNNNumpy.calculate_loss = calculate_loss


I am working on the file "RNNLM.ipynb".
The first task is:
Done! Let's try to get a sense of how long it would take to train our network:
Can you write Python code for it?

np.random.seed(10)
model = RNNNumpy(vocabulary_size)
%timeit model.sgd_step(X_train[10], y_train[10], 0.005)


I am working on the file "RNNLM.ipynb".
The first task is:
For each word in the sentence (45 above), our model made 8000 predictions representing probabilities of the next word. Note that because we initialized $U,V,W$ to random values these predictions are completely random right now. The following gives the indices of the highest probability predictions for each word:
Can you write Python code for it?

predictions = model.predict(X_train[10])
print predictions.shape
print predictions


I am working on the file "RNNLM.ipynb".
The first task is:
Generating Text
Now that we have our model we can ask it to generate new text for us! Let's implement a helper function to generate new sentences:
Can you write Python code for it?

def generate_sentence(model):
    # We start the sentence with the start token
    new_sentence = [word_to_index[sentence_start_token]]
    # Repeat until we get an end token
    while not new_sentence[-1] == word_to_index[sentence_end_token]:
        next_word_probs = model.forward_propagation(new_sentence)
        sampled_word = word_to_index[unknown_token]
        # We don't want to sample unknown words
        while sampled_word == word_to_index[unknown_token]:
            samples = np.random.multinomial(1, next_word_probs[-1])
            sampled_word = np.argmax(samples)
        new_sentence.append(sampled_word)
    sentence_str = [index_to_word[x] for x in new_sentence[1:-1]]
    return sentence_str

num_sentences = 10
senten_min_length = 7

for i in range(num_sentences):
    sent = []
    # We want long sentences, not sentences with one or two words
    while len(sent) < senten_min_length:
        sent = generate_sentence(model)
    print " ".join(sent)


I am working on the file "RNNLM.ipynb".
The first task is:
Good, it seems like our implementation is at least doing something useful and decreasing the loss, just like we wanted.
Training our Network with Theano and the GPU
I have previously written a tutorial on Theano, and since all our logic will stay exactly the same I won't go through optimized code here again. I defined a RNNTheano class that replaces the numpy calculations with corresponding calculations in Theano. Just like the rest of this post, the code is also available Github.
Can you write Python code for it?

from rnn_theano import RNNTheano, gradient_check_theano

np.random.seed(10)
# To avoid performing millions of expensive calculations we use a smaller vocabulary size for checking.
grad_check_vocab_size = 5
model = RNNTheano(grad_check_vocab_size, 10)
gradient_check_theano(model, [0,1,2,3], [1,2,3,4])

np.random.seed(10)
model = RNNTheano(vocabulary_size)
%timeit model.sgd_step(X_train[10], y_train[10], 0.005)


I am working on the file "RNNLM.ipynb".
The first task is:
Gradient Checking
Whenever you implement backpropagation it is good idea to also implement gradient checking, which is a way of verifying that your implementation is correct. The idea behind gradient checking is that derivative of a parameter is equal to the slope at the point, which we can approximate by slightly changing the parameter and then dividing by the change:
$
\begin{aligned}
\frac{\partial L}{\partial \theta} \approx \lim_{h \to 0} \frac{J(\theta + h) - J(\theta -h)}{2h}
\end{aligned}
$
We then compare the gradient we calculated using backpropagation to the gradient we estimated with the method above. If there's no large difference we are good. The approximation needs to calculate the total loss for every parameter, so that gradient checking is very expensive (remember, we had more than a million parameters in the example above). So it's a good idea to perform it on a model with a smaller vocabulary.
Can you write Python code for it?

def gradient_check(self, x, y, h=0.001, error_threshold=0.01):
    # Calculate the gradients using backpropagation. We want to checker if these are correct.
    bptt_gradients = model.bptt(x, y)
    # List of all parameters we want to check.
    model_parameters = ['U', 'V', 'W']
    # Gradient check for each parameter
    for pidx, pname in enumerate(model_parameters):
        # Get the actual parameter value from the mode, e.g. model.W
        parameter = operator.attrgetter(pname)(self)
        print "Performing gradient check for parameter %s with size %d." % (pname, np.prod(parameter.shape))
        # Iterate over each element of the parameter matrix, e.g. (0,0), (0,1), ...
        it = np.nditer(parameter, flags=['multi_index'], op_flags=['readwrite'])
        while not it.finished:
            ix = it.multi_index
            # Save the original value so we can reset it later
            original_value = parameter[ix]
            # Estimate the gradient using (f(x+h) - f(x-h))/(2*h)
            parameter[ix] = original_value + h
            gradplus = model.calculate_total_loss([x],[y])
            parameter[ix] = original_value - h
            gradminus = model.calculate_total_loss([x],[y])
            estimated_gradient = (gradplus - gradminus)/(2*h)
            # Reset parameter to original value
            parameter[ix] = original_value
            # The gradient for this parameter calculated using backpropagation
            backprop_gradient = bptt_gradients[pidx][ix]
            # calculate The relative error: (|x - y|/(|x| + |y|))
            relative_error = np.abs(backprop_gradient - estimated_gradient)/(np.abs(backprop_gradient) + np.abs(estimated_gradient))
            # If the error is to large fail the gradient check
            if relative_error > error_threshold:
                print "Gradient Check ERROR: parameter=%s ix=%s" % (pname, ix)
                print "+h Loss: %f" % gradplus
                print "-h Loss: %f" % gradminus
                print "Estimated_gradient: %f" %...


I am working on the file "RNNLM.ipynb".
The first task is:
Let's take a step back and think about what the loss should be for random predictions. That will give us a baseline and make sure our implementation is correct. We have $C$ words in our vocabulary, so each word should be (on average) predicted with probability $1/C$, which would yield a loss of $L = -\frac{1}{N} N \log\frac{1}{C} = \log C$:
Can you write Python code for it?

# Limit to 1000 examples to save time
print "Expected Loss for random predictions: %f" % np.log(vocabulary_size)
print "Actual loss: %f" % model.calculate_loss(X_train[:1000], y_train[:1000])


I am working on the file "RNNLM.ipynb".
The first task is:
Let's try our newly implemented methods and see an example output:
Can you write Python code for it?

np.random.seed(10)
model = RNNNumpy(vocabulary_size)
o, s = model.forward_propagation(X_train[10])
print o.shape
print o


I am working on the file "RNNLM.ipynb".
The first task is:
Pretty close! Keep in mind that evaluating the loss on the full dataset is an expensive operation and can take hours if you have a lot of data!
Training the RNN with SGD and Backpropagation Through Time (BPTT)
Remember that we want to find the parameters $U,V$ and $W$ that minimize the total loss on the training data. The most common way to do this is SGD, Stochastic Gradient Descent. The idea behind SGD is pretty simple. We iterate over all our training examples and during each iteration we nudge the parameters into a direction that reduces the error. These directions are given by the gradients on the loss: $\frac{\partial L}{\partial U}, \frac{\partial L}{\partial V}, \frac{\partial L}{\partial W}$. SGD also needs a learning rate, which defines how big of a step we want to make in each iteration. SGD is the most popular optimization method not only for Neural Networks, but also for many other Machine Learning algorithms. As such there has been a lot of research on how to optimize SGD using batching, parallelism and adaptive learning rates. Even though the basic idea is simple, implementing SGD in a really efficient way can become very complex. If you want to learn more about SGD this is a good place to start. Due to its popularity there are a wealth of tutorials floating around the web, and I don't want to duplicate them here. I'll implement a simple version of SGD that should be understandable even without a background in optimization.
But how do we calculate those gradients we mentioned above? In a traditional Neural Network we do this through the backpropagation algorithm. In RNNs we use a slightly modified version of the this algorithm called Backpropagation Through Time (BPTT). Because the parameters are shared by all time steps in the network, the gradient at each output depends not only on the calculations of the current time step, but also the previous time steps. If you know calculus, it really is just applying the chain rule. The next part of the tutorial will be all about BPTT, so I won't go into...
Can you write Python code for it?

def bptt(self, x, y):
    T = len(y)
    # Perform forward propagation
    o, s = self.forward_propagation(x)
    # We accumulate the gradients in these variables
    dLdU = np.zeros(self.U.shape)
    dLdV = np.zeros(self.V.shape)
    dLdW = np.zeros(self.W.shape)
    delta_o = o
    delta_o[np.arange(len(y)), y] -= 1.
    # For each output backwards...
    for t in np.arange(T)[::-1]:
        dLdV += np.outer(delta_o[t], s[t].T)
        # Initial delta calculation
        delta_t = self.V.T.dot(delta_o[t]) * (1 - (s[t] ** 2))
        # Backpropagation through time (for at most self.bptt_truncate steps)
        for bptt_step in np.arange(max(0, t-self.bptt_truncate), t+1)[::-1]:
            # print "Backpropagation step t=%d bptt step=%d " % (t, bptt_step)
            dLdW += np.outer(delta_t, s[bptt_step-1])
            dLdU[:,x[bptt_step]] += delta_t
            # Update delta for next step
            delta_t = self.W.T.dot(delta_t) * (1 - s[bptt_step-1] ** 2)
    return [dLdU, dLdV, dLdW]

RNNNumpy.bptt = bptt


I am working on the file "RNNLM.ipynb".
The first task is:
SGD Implementation
Now that we are able to calculate the gradients for our parameters we can implement SGD. I like to do this in two steps: 1. A function sdg_step that calculates the gradients and performs the updates for one batch. 2. An outer loop that iterates through the training set and adjusts the learning rate.
Can you write Python code for it?

# Performs one step of SGD.
def numpy_sdg_step(self, x, y, learning_rate):
    # Calculate the gradients
    dLdU, dLdV, dLdW = self.bptt(x, y)
    # Change parameters according to gradients and learning rate
    self.U -= learning_rate * dLdU
    self.V -= learning_rate * dLdV
    self.W -= learning_rate * dLdW

RNNNumpy.sgd_step = numpy_sdg_step

# Outer SGD Loop
# - model: The RNN model instance
# - X_train: The training data set
# - y_train: The training data labels
# - learning_rate: Initial learning rate for SGD
# - nepoch: Number of times to iterate through the complete dataset
# - evaluate_loss_after: Evaluate the loss after this many epochs
def train_with_sgd(model, X_train, y_train, learning_rate=0.005, nepoch=100, evaluate_loss_after=5):
    # We keep track of the losses so we can plot them later
    losses = []
    num_examples_seen = 0
    for epoch in range(nepoch):
        # Optionally evaluate the loss
        if (epoch % evaluate_loss_after == 0):
            loss = model.calculate_loss(X_train, y_train)
            losses.append((num_examples_seen, loss))
            time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
            print "%s: Loss after num_examples_seen=%d epoch=%d: %f" % (time, num_examples_seen, epoch, loss)
            # Adjust the learning rate if loss increases
            if (len(losses) > 1 and losses[-1][1] > losses[-2][1]):
                learning_rate = learning_rate * 0.5
                print "Setting learning rate to %f" % learning_rate
            sys.stdout.flush()
        # For each training example...
        for i in range(len(y_train)):
            # One SGD step
            model.sgd_step(X_train[i], y_train[i], learning_rate)
            num_examples_seen += 1


I am working on the file "RNNLM.ipynb".
The first task is:
This the second part of the Recurrent Neural Network Tutorial. The first part is here.
In this part we will implement a full Recurrent Neural Network from scratch using Python and optimize our implementation using Theano, a library to perform operations on a GPU. The full code is available on Github. I will skip over some boilerplate code that is not essential to understanding Recurrent Neural Networks, but all of that is also on Github.
Language Modeling
Our goal is to build a Language Model using a Recurrent Neural Network. Here's what that means. Let's say we have sentence  of $m$ words. Language Model allows us to predict the probability of observing the sentence (in a given dataset) as:
$
\begin{aligned}
P(w_1,...,w_m) = \prod_{i=1}^{m}P(w_i \mid w_1,..., w_{i-1})
\end{aligned}
$
In words, the probability of a sentence is the product of probabilities of each word given the words that came before it. So, the probability of the sentence "He went to buy some chocolate" would be the probability of "chocolate" given "He went to buy some", multiplied by the probability of "some" given "He went to buy", and so on.
Why is that useful? Why would we want to assign a probability to observing a sentence?
First, such a model can be used as a scoring mechanism. For example, a Machine Translation system typically generates multiple candidates for an input sentence. You could use a language model to pick the most probable sentence. Intuitively, the most probable sentence is likely to be grammatically correct. Similar scoring happens in speech recognition systems.
But solving the Language Modeling problem also has a cool side effect. Because we can predict the probability of a word given the preceding words, we are able to generate new text. It's a generative model. Given an existing sequence of words we sample a next word from the predicted probabilities, and repeat the process until we have a full sentence. Andrew Karparthy has a great post that demonstrates what language models are capable of. His models are trained...
Can you write Python code for it?

vocabulary_size = 8000
unknown_token = "UNKNOWN_TOKEN"
sentence_start_token = "SENTENCE_START"
sentence_end_token = "SENTENCE_END"

# Read the data and append SENTENCE_START and SENTENCE_END tokens
print "Reading CSV file..."
with open('data/reddit-comments-2015-08.csv', 'rb') as f:
    reader = csv.reader(f, skipinitialspace=True)
    reader.next()
    # Split full comments into sentences
    sentences = itertools.chain(*[nltk.sent_tokenize(x[0].decode('utf-8').lower()) for x in reader])
    # Append SENTENCE_START and SENTENCE_END
    sentences = ["%s %s %s" % (sentence_start_token, x, sentence_end_token) for x in sentences]
print "Parsed %d sentences." % (len(sentences))

# Tokenize the sentences into words
tokenized_sentences = [nltk.word_tokenize(sent) for sent in sentences]

# Count the word frequencies
word_freq = nltk.FreqDist(itertools.chain(*tokenized_sentences))
print "Found %d unique words tokens." % len(word_freq.items())

# Get the most common words and build index_to_word and word_to_index vectors
vocab = word_freq.most_common(vocabulary_size-1)
index_to_word = [x[0] for x in vocab]
index_to_word.append(unknown_token)
word_to_index = dict([(w,i) for i,w in enumerate(index_to_word)])

print "Using vocabulary size %d." % vocabulary_size
print "The least frequent word in our vocabulary is '%s' and appeared %d times." % (vocab[-1][0], vocab[-1][1])

# Replace all words not in our vocabulary with the unknown token
for i, sent in enumerate(tokenized_sentences):
    tokenized_sentences[i] = [w if w in word_to_index else unknown_token for w in sent]

print "\nExample sentence: '%s'" % sentences[0]
print "\nExample sentence after Pre-processing: '%s'" % tokenized_sentences[0]

# Create the training data
X_train = np.asarray([[word_to_index[w] for w in sent[:-1]] for sent in tokenized_sentences])
y_train = np.asarray([[word_to_index[w] for w in sent[1:]] for sent in tokenized_sentences])


I am working on the file "RNNLM.ipynb".
The first task is:
This time, one SGD step takes 70ms on my Mac (without GPU) and 23ms on a g2.2xlarge Amazon EC2 instance with GPU. That's a 15x improvement over our initial implementation and means we can train our model in hours/days instead of weeks. There are still a vast number of optimizations we could make, but we're good enough for now.
To help you avoid spending days training a model I have pre-trained a Theano model with a hidden layer dimensionality of 50 and a vocabulary size of 8000. I trained it for 50 epochs in about 20 hours. The loss was was still decreasing and training longer would probably have resulted in a better model, but I was running out of time and wanted to publish this post. Feel free to try it out yourself and trian for longer. You can find the model parameters in data/trained-model-theano.npz in the Github repository and load them using the load_model_parameters_theano method:
Can you write Python code for it?

from utils import load_model_parameters_theano, save_model_parameters_theano

model = RNNTheano(vocabulary_size, hidden_dim=50)
# losses = train_with_sgd(model, X_train, y_train, nepoch=50)
# save_model_parameters_theano('./data/trained-model-theano.npz', model)
load_model_parameters_theano('./data/trained-model-theano.npz', model)


I am working on the file "RNNLM.ipynb".
The first task is:
Uh-oh, bad news. One step of SGD takes approximately 350 milliseconds on my laptop. We have about 80,000 examples in our training data, so one epoch (iteration over the whole data set) would take several hours. Multiple epochs would take days, or even weeks! And we're still working with a small dataset compared to what's being used by many of the companies and researchers out there. What now?
Fortunately there are many ways to speed up our code. We could stick with the same model and make our code run faster, or we could modify our model to be less computationally expensive, or both. Researchers have identified many ways to make models less computationally expensive, for example by using a hierarchical softmax or adding projection layers to avoid the large matrix multiplications (see also here or here). But I want to keep our model simple and go the first route: Make our implementation run faster using a GPU. Before doing that though, let's just try to run SGD with a small dataset and check if the loss actually decreases:
Can you write Python code for it?

np.random.seed(10)
# Train on a small subset of the data to see what happens
model = RNNNumpy(vocabulary_size)
losses = train_with_sgd(model, X_train[:100], y_train[:100], nepoch=10, evaluate_loss_after=1)


I am working on the file "RNNLM.ipynb".
The first task is:
We not only return the calculated outputs, but also the hidden states. We will use them later to calculate the gradients, and by returning them here we avoid duplicate computation. Each $o_t$ is a vector of probabilities representing the words in our vocabulary, but sometimes, for example when evaluating our model, all we want is the next word with the highest probability. We call this function predict:
Can you write Python code for it?

def predict(self, x):
    # Perform forward propagation and return index of the highest score
    o, s = self.forward_propagation(x)
    return np.argmax(o, axis=1)

RNNNumpy.predict = predict


I am working on the file "RNN_project.ipynb".
The first task is:
1.2  Cutting our time series into sequences
Remember, our time series is a sequence of numbers that we can represent in general mathematically as
$$s_{0},s_{1},s_{2},...,s_{P}$$
where $s_{p}$ is the numerical value of the time series at time period $p$ and where $P$ is the total length of the series.  In order to apply our RNN we treat the time series prediction problem as a regression problem, and so need to use a sliding window to construct a set of associated input/output pairs to regress on.  This process is animated in the gif below.

For example - using a window of size T = 5 (as illustrated in the gif above) we produce a set of input/output pairs like the one shown in the table below
$$\begin{array}{c|c}
\text{Input} & \text{Output}\
\hline \color{CornflowerBlue} {\langle s_{1},s_{2},s_{3},s_{4},s_{5}\rangle} & \color{Goldenrod}{ s_{6}} \
\ \color{CornflowerBlue} {\langle s_{2},s_{3},s_{4},s_{5},s_{6} \rangle } & \color{Goldenrod} {s_{7} } \
\color{CornflowerBlue}  {\vdots} & \color{Goldenrod} {\vdots}\
\color{CornflowerBlue} { \langle s_{P-5},s_{P-4},s_{P-3},s_{P-2},s_{P-1} \rangle } & \color{Goldenrod} {s_{P}}
\end{array}$$
Notice here that each input is a sequence (or vector) of length 4 (and in general has length equal to the window size T) while each corresponding output is a scalar value.  Notice also how given a time series of length P and window size T = 5 as shown above, we created P - 5  input/output pairs.  More generally, for a window size T we create P - T such pairs.
Now its time for you to window the input time series as described above!

TODO: Fill in the function below - called window_transform_series - that runs a sliding window along the input series and creates associated input/output pairs.    Note that this function should input a) the series and b) the window length, and return the input/output subsequences.  Make sure to format returned input/output as generally shown in table above...
Can you write Python code for it?


I am working on the file "RNN_project.ipynb".
The first task is:
1.3  Splitting into training and testing sets
In order to perform proper testing on our dataset we will lop off the last 1/3 of it for validation (or testing).  This is that once we train our model we have something to test it on (like any regression problem!).  This splitting into training/testing sets is done in the cell below.
Note how here we are not splitting the dataset randomly as one typically would do when validating a regression model.  This is because our input/output pairs are related temporally.   We don't want to validate our model by training on a random subset of the series and then testing on another random subset, as this simulates the scenario that we receive new points within the timeframe of our training set.
We want to train on one solid chunk of the series (in our case, the first full 2/3 of it), and validate on a later chunk (the last 1/3) as this simulates how we would predict future values of a time series.
Can you write Python code for it?

# split our dataset into training / testing sets
train_test_split = int(np.ceil(2*len(y)/float(3)))   # set the split point

# partition the training set
X_train = X[:train_test_split,:]
y_train = y[:train_test_split]

# keep the last chunk for testing
X_test = X[train_test_split:,:]
y_test = y[train_test_split:]

# NOTE: to use keras's RNN LSTM module our input must be reshaped to [samples, stepsize, window size]
X_train = np.asarray(np.reshape(X_train, (X_train.shape[0], 1, window_size)))
X_test = np.asarray(np.reshape(X_test, (X_test.shape[0], 1, window_size)))


I am working on the file "RNN_project.ipynb".
The first task is:
2.3  Cutting data into input/output pairs
Now that we have our text all cleaned up, how can we use it to train a model to generate sentences automatically?  First we need to train a machine learning model - and in order to do that we need a set of input/output pairs for a model to train on.  How can we create a set of input/output pairs from our text to train on?
Remember in part 1 of this notebook how we used a sliding window to extract input/output pairs from a time series?  We do the same thing here!  We slide a window of length $T$ along our giant text corpus - everything in the window becomes one input while the character following becomes its corresponding output.  This process of extracting input/output pairs is illustrated in the gif below on a small example text using a window size of T = 4.
<img src="images/text_windowing_training.gif" width=400 height=400/>
Notice one aspect of the sliding window in this gif that does not mirror the analaogous gif for time series shown in part 1 of the notebook - we do not need to slide the window along one character at a time but can move by a fixed step size $M$ greater than 1 (in the gif indeed $M = 1$).  This is done with large input texts (like ours which has over 500,000 characters!) when sliding the window along one character at a time we would create far too many input/output pairs to be able to reasonably compute with.
More formally lets denote our text corpus - which is one long string of characters - as follows
$$s_{0},s_{1},s_{2},...,s_{P}$$
where $P$ is the length of the text (again for our text $P \approx 500,000!$).  Sliding a window of size T = 5 with a step length of M = 1 (these are the parameters shown in the gif above) over this sequence produces the following list of input/output pairs
$$\begin{array}{c|c}
\text{Input} & \text{Output}\
\hline \color{CornflowerBlue} {\langle s_{1},s_{2},s_{3},s_{4}\rangle} & \color{Goldenrod}{ s_{5}} \
\ \color{CornflowerBlue} {\langle s_{2},s_{3},s_{4},s_{5} \rangle } & \color{Goldenrod} {s_{6} }...
Can you write Python code for it?

### TODO: fill out the function below that transforms the input text and window-size into a set of input/output pairs for use with our RNN model
def window_transform_series(text,window_size,step_size):  # defines the function
    # containers for input/output pairs
    inputs = []
    outputs = []
    for i in range(window_size, len(text), step_size):
        inputs.append(text[i - window_size:i])
        outputs.append(text[i])

    return inputs,outputs


I am working on the file "RNN_project.ipynb".
The first task is:
2.7  Training our RNN model for text generation
With our RNN setup we can now train it!  Lets begin by trying it out on a small subset of the larger version.  In the next cell we take the first 10,000 input/output pairs from our training database to learn on.
Can you write Python code for it?

# a small subset of our input/output pairs
Xsmall = X[:10000,:,:]
ysmall = y[:10000,:]


I am working on the file "RNN_project.ipynb".
The first task is:

1.4  Build and run an RNN regression model
Having created input/output pairs out of our time series and cut this into training/testing sets, we can now begin setting up our RNN.  We use Keras to quickly build a two hidden layer RNN of the following specifications

layer 1 uses an LSTM module with 5 hidden units (note here the input_shape = (1,window_size))
layer 2 uses a fully connected module with one unit
the 'mean_squared_error' loss should be used (remember: we are performing regression here)

This can be constructed using just a few lines - see e.g., the general Keras documentation and the LTSM documentation in particular for examples of how to quickly use Keras to build neural network models.  Make sure you are initializing your optimizer given the keras-recommended approach for RNNs (given in the cell below).
Can you write Python code for it?

### TODO: create required RNN model
# import keras network libraries
from keras.models import Sequential # this imports the sequential starter for the model
from keras.layers import Dense, Dropout # this imports the fully connected and dropout layers
from keras.layers import LSTM # this imports our RNN LSTM layer
import keras # this imports the whole keras library

# given - fix random seed - so we can all reproduce the same results on our default time series
np.random.seed(0)

# TODO: build an RNN to perform regression on our time series input/output data

model = Sequential()  # this creates a blank model
model.add(LSTM(5, input_shape=X_train.shape[1:]))  # this adds one LSTM layer with a shape of (1, window_size)
model.add(Dense(1))  # this creates an output layer with 1 output to predict price

# build model using keras documentation recommended optimizer initialization
optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)

# compile the model
model.compile(loss='mean_squared_error', optimizer=optimizer)


I am working on the file "RNN_project.ipynb".
The first task is:

TODO: finish cleaning the text
Lets make sure we haven't left any other non-English/proper punctuation (commas, periods, etc., are ok) characters lurking around in the depths of the text.  You can do this by ennumerating all the text's unique characters, examining them, and then replacing any unwanted (non-english) characters with empty spaces!  Once we find all of the text's unique characters, we can remove all of the non-English/proper punctuation ones in the next cell.  Note: don't remove necessary punctuation marks!
Can you write Python code for it?

### TODO: list all unique characters in the text and remove any non-english ones
# find all unique characters in the text
unique_chars = set(text)  # this creates a unique set of all characters in the text

# this gives us strings of all letters, and punctuation.
# then we combine them into a set so we can use .difference to get characters
# that are in unique_chars but not in the chars we want to keep
# then we replace those chars with ' '
import string
letters = string.ascii_lowercase  # this creates a string with all letters
punc = '()[]&.,"\'!?:-'  # this creates a string with all reasonable punctuation -- says in the instructions that:
# commas, periods, etc., are ok
# digits = string.digits  # creates a string with all digits; digits are an acceptable part of english speech
# I at first included digits because I thought it would be fine, but removed them because there seemed to be some really long numbers in there
keepers = set(letters + punc + ' ')  # this combines the three previously made strings and adds a space
# for characters to keep
# remove as many non-english characters and character sequences as you can
print(unique_chars.difference(keepers))  # prints the difference of unique_chars from keepers for debugging
for l in unique_chars.difference(keepers):  # goes through a loop of every character not in keepers but in the text
    text = text.replace(l, ' ')  # replaces the bad characters with a space

# shorten any extra dead space created above
text = text.replace('  ',' ')

'à' in set(text) # making sure a bad character was removed

set(text) # checking the characters in the set to make sure it looks good

len(set(text))  # checking how many characters are in the unique set


I am working on the file "RNN_project.ipynb".
The first task is:

2.6 Setting up our RNN
With our dataset loaded and the input/output pairs extracted / transformed we can now begin setting up our RNN for training.  Again we will use Keras to quickly build a single hidden layer RNN - where our hidden layer consists of LTSM modules.
Time to get to work: build a 3 layer RNN model of the following specification

layer 1 should be an LSTM module with 200 hidden units --> note this should have input_shape = (window_size,len(chars)) where len(chars) = number of unique characters in your cleaned text
layer 2 should be a linear module, fully connected, with len(chars) hidden units --> where len(chars) = number of unique characters in your cleaned text
layer 3 should be a softmax activation ( since we are solving a multiclass classification)
Use the categorical_crossentropy loss

This network can be constructed using just a few lines - as with the RNN network you made in part 1 of this notebook.  See e.g., the general Keras documentation and the LTSM documentation in particular for examples of how to quickly use Keras to build neural network models.
Can you write Python code for it?

### necessary functions from the keras library
from keras.models import Sequential # imports model creator
from keras.layers import Dense, Activation, LSTM  # imports some neural net layers
from keras.optimizers import RMSprop  # imports rmsprop optimizer
from keras.utils.data_utils import get_file  # imports utility for getting file
import keras  # imports whole keras library
import random  # imports random library
# blank line
# TODO build the required RNN model: a single LSTM hidden layer with softmax activation, categorical_crossentropy loss
model = Sequential()  # creates blank model
model.add(LSTM(200, input_shape=X.shape[1:]))  # 200-dim LSTM layer with shape 7
model.add(Dense(len(chars)))  # creates output layer to predict character
model.add(Activation('softmax'))  # softmax to output probabilities
# blank line
# initialize optimizer
optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)
# blank line
# compile model --> make sure initialized optimizer and callbacks - as defined above - are used
model.compile(loss='categorical_crossentropy', optimizer=optimizer)


I am working on the file "RNN_project.ipynb".
The first task is:

With your trained model try a few subsets of the complete text as input - note the length of each must be exactly equal to the window size.  For each subset us the function above to predict the next 100 characters that follow each input.
Can you write Python code for it?

# TODO: choose an input sequence and use the prediction function in the previous Python cell to
# predict 100 characters following it.
# get an appropriately sized chunk of characters from the text
start_inds = [100, 105, 500, 2000]

# load in weights
model.load_weights('model_weights/best_RNN_small_textdata_weights.hdf5')
for s in start_inds:
    start_index = s
    input_chars = text[start_index: start_index + window_size]

    # use the prediction function
    predict_input = predict_next_chars(model,input_chars,num_to_predict = 100)

    # print out input characters
    print('------------------')
    input_line = 'input chars = ' + '\n' +  input_chars + '"' + '\n'
    print(input_line)

    # print out predicted characters
    line = 'predicted chars = ' + '\n' +  predict_input + '"' + '\n'
    print(line)


I am working on the file "RNN_project.ipynb".
The first task is:
Activating the next cell plots the original data, as well as both predictions on the training and testing sets.
Can you write Python code for it?

### Plot everything - the original series as well as predictions on training and testing sets
import matplotlib.pyplot as plt
%matplotlib inline

# plot original series
plt.plot(dataset,color = 'k')

# plot training set prediction
split_pt = train_test_split + window_size
plt.plot(np.arange(window_size,split_pt,1),train_predict,color = 'b')

# plot testing set prediction
plt.plot(np.arange(split_pt,split_pt + len(test_predict),1),test_predict,color = 'r')

# pretty up graph
plt.xlabel('day')
plt.ylabel('(normalized) price of Apple stock')
plt.legend(['original series','training fit','testing fit'],loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()


I am working on the file "RNN_project.ipynb".
The first task is:
How do we make a given number of predictions (characters) based on this fitted model?
First we predict the next character after following any chunk of characters in the text of length equal to our chosen window size.  Then we remove the first character in our input sequence and tack our prediction onto the end.  This gives us a slightly changed sequence of inputs that still has length equal to the size of our window.  We then feed in this updated input sequence into the model to predict the another character.  Together then we have two predicted characters following our original input sequence.  Repeating this process N times gives us N predicted characters.
In the next Python cell we provide you with a completed function that does just this - it makes predictions when given a) a trained RNN model, b) a subset of (window_size) characters from the text, and c) a number of characters to predict (to follow our input subset).
Can you write Python code for it?

# function that uses trained model to predict a desired number of future characters
def predict_next_chars(model,input_chars,num_to_predict):
    # create output
    predicted_chars = ''
    for i in range(num_to_predict):
        # convert this round's predicted characters to numerical input
        x_test = np.zeros((1, window_size, len(chars)))
        for t, char in enumerate(input_chars):
            x_test[0, t, chars_to_indices[char]] = 1.

        # make this round's prediction
        test_predict = model.predict(x_test,verbose = 0)[0]

        # translate numerical prediction back to characters
        r = np.argmax(test_predict)                           # predict class of each test input
        d = indices_to_chars[r]

        # update predicted_chars and input
        predicted_chars+=d
        input_chars+=d
        input_chars = input_chars[1:]
    return predicted_chars


I am working on the file "RNN_project.ipynb".
The first task is:
In the next cell we compute training and testing errors using our trained model - you should be able to achieve at least
training_accuracy < 0.02
and
testing_accuracy < 0.02
with your fully trained model.
If either or both of your accuracies are larger than 0.02 re-train your model - increasing the number of epochs you take (a maximum of around 1,000 should do the job) and/or adjusting your batch_size.
Can you write Python code for it?

# print out training and testing errors
training_accuracy = model.evaluate(X_train, y_train, verbose=0)
print('training accuracy = ' + str(training_accuracy))

testing_accuracy = model.evaluate(X_test, y_test, verbose=0)
print('testing accuracy = ' + str(testing_accuracy))


I am working on the file "RNN_project.ipynb".
The first task is:
Lets print out a few input/output pairs to verify that we have made the right sort of stuff!
Can you write Python code for it?

# print out a few of the input/output pairs to verify that we've made the right kind of stuff to learn from
print('input = ' + inputs[2])
print('output = ' + outputs[2])
print('--------------')
print('input = ' + inputs[100])
print('output = ' + outputs[100])


I am working on the file "RNN_project.ipynb".
The first task is:
Lets see how the first 1000 characters of our text looks now!
Can you write Python code for it?

### print out the first 1000 characters of the raw text to get a sense of what we need to throw out
text[:1000]


I am working on the file "RNN_project.ipynb".
The first task is:
Looks good!
2.4  Wait, what kind of problem is text generation again?
In part 1 of this notebook we used the same pre-processing technique - the sliding window - to produce a set of training input/output pairs to tackle the problem of time series prediction by treating the problem as one of regression.  So what sort of problem do we have here now, with text generation?  Well, the time series prediction was a regression problem because the output (one value of the time series) was a continuous value.  Here - for character-by-character text generation - each output is a single character.  This isn't a continuous value - but a distinct class - therefore character-by-character text generation is a classification problem.
How many classes are there in the data?  Well, the number of classes is equal to the number of unique characters we have to predict!  How many of those were there in our dataset again?  Lets print out the value again.
Can you write Python code for it?

# print out the number of unique characters in the dataset
chars = sorted(list(set(text)))
print ("this corpus has " +  str(len(chars)) + " unique characters")
print ('and these characters are ')
print (chars)


I am working on the file "RNN_project.ipynb".
The first task is:
Next, lets examine a bit of the raw text.  Because we are interested in creating sentences of English words automatically by building up each word character-by-character, we only want to train on valid English words.  In other words - we need to remove all of the other junk characters that aren't words!
Can you write Python code for it?

### print out the first 1000 characters of the raw text to get a sense of what we need to throw out
text[:2000]


I am working on the file "RNN_project.ipynb".
The first task is:
Now that we have thrown out a good number of non-English characters/character sequences lets print out some statistics about the dataset - including number of total characters and number of unique characters.
Can you write Python code for it?

# count the number of unique characters in the text
chars = sorted(list(set(text)))

# print some of the text, as well as statistics
print ("this corpus has " +  str(len(text)) + " total number of characters")
print ("this corpus has " +  str(len(chars)) + " unique characters")


I am working on the file "RNN_project.ipynb".
The first task is:
Rockin' - so we have a multi-class classification problem on our hands!
2.5  One-hot encoding characters
There's just one last issue we have to deal with before tackle: machine learning algorithm deal with numerical data and all of our input/output pairs are characters.  So we just need to transform our characters into equivalent numerical values.  The most common way of doing this is via a 'one-hot encoding' scheme.  Here's how it works.
We transform each character in our inputs/outputs into a vector with length equal to the number of unique characters in our text.  This vector is all zeros except one location where we place a 1 - and this location is unique to each character type.  e.g., we transform 'a', 'b', and 'c' as follows
$$a\longleftarrow\left[\begin{array}{c}
1\
0\
0\
\vdots\
0\
0
\end{array}\right]\,\,\,\,\,\,\,b\longleftarrow\left[\begin{array}{c}
0\
1\
0\
\vdots\
0\
0
\end{array}\right]\,\,\,\,\,c\longleftarrow\left[\begin{array}{c}
0\
0\
1\
\vdots\
0\
0
\end{array}\right]\cdots$$
where each vector has 32 entries (or in general: number of entries = number of unique characters in text).
The first practical step towards doing this one-hot encoding is to form a dictionary mapping each unique character to a unique integer, and one dictionary to do the reverse mapping.  We can then use these dictionaries to quickly make our one-hot encodings, as well as re-translate (from integers to characters) the results of our trained RNN classification model.
Can you write Python code for it?

# this dictionary is a function mapping each unique character to a unique integer
chars_to_indices = dict((c, i) for i, c in enumerate(chars))  # map each unique character to unique integer

# this dictionary is a function mapping each unique integer back to a unique character
indices_to_chars = dict((i, c) for i, c in enumerate(chars))  # map each unique integer back to unique character


I am working on the file "RNN_project.ipynb".
The first task is:
This looks ok, but not great.  Now lets try the same experiment with a larger chunk of the data - with the first 100,000 input/output pairs.
Tuning RNNs for a typical character dataset like the one we will use here is a computationally intensive endeavour and thus timely on a typical CPU.  Using a reasonably sized cloud-based GPU can speed up training by a factor of 10.  Also because of the long training time it is highly recommended that you carefully write the output of each step of your process to file.  This is so that all of your results are saved even if you close the web browser you're working out of, as the processes will continue processing in the background but variables/output in the notebook system will not update when you open it again.
In the next cell we show you how to create a text file in Python and record data to it.  This sort of setup can be used to record your final predictions.
Can you write Python code for it?

### A simple way to write output to file
f = open('my_test_output.txt', 'w')              # create an output file to write too
f.write('this is only a test ' + '\n')           # print some output text
x = 2
f.write('the value of x is ' + str(x) + '\n')    # record a variable value
f.close()

# print out the contents of my_test_output.txt
f = open('my_test_output.txt', 'r')              # create an output file to write too
f.read()


I am working on the file "RNN_project.ipynb".
The first task is:
With our function complete we can now use it to produce input/output pairs!  We employ the function in the next cell, where the window_size = 50 and step_size = 5.
Can you write Python code for it?

# run your text window-ing function
window_size = 100
step_size = 5
inputs, outputs = window_transform_series(text,window_size,step_size)


I am working on the file "RNN_project.ipynb".
The first task is:
With this recording devices we can now more safely perform experiments on larger portions of the text.  In the next cell we will use the first 100,000 input/output pairs to train our RNN model.
First we fit our model to the dataset, then generate text using the trained model in precisely the same generation method applied before on the small dataset.
Note: your generated words should be - by and large - more realistic than with the small dataset, but you won't be able to generate perfect English sentences even with this amount of data.  A rule of thumb: your model is working well if you generate sentences that largely contain real English words.
Can you write Python code for it?

# a small subset of our input/output pairs
Xlarge = X[:100000,:,:]
ylarge = y[:100000,:]

# TODO: fit to our larger dataset
model.fit(Xlarge, ylarge, batch_size=500, epochs=30, verbose = 0, callbacks=[TQDMNotebookCallback()])

# save weights
model.save_weights('model_weights/best_RNN_large_textdata_weights.hdf5')

# TODO: choose an input sequence and use the prediction function in the previous
# Python cell to predict 100 characters following it
# get an appropriately sized chunk of characters from the text
start_inds = [100, 105, 500, 2000]  # chose the same test indices as before

# save output
f = open('text_gen_output/RNN_large_textdata_output.txt', 'w')  # create an output file to write too

# load weights
model.load_weights('model_weights/best_RNN_large_textdata_weights.hdf5')
for s in start_inds:
    start_index = s
    input_chars = text[start_index: start_index + window_size]

    # use the prediction function
    predict_input = predict_next_chars(model,input_chars,num_to_predict = 100)

    # print out input characters
    line = '-------------------' + '\n'
    print(line)
    f.write(line)

    input_line = 'input chars = ' + '\n' +  input_chars + '"' + '\n'
    print(input_line)
    f.write(input_line)

    # print out predicted characters
    predict_line = 'predicted chars = ' + '\n' +  predict_input + '"' + '\n'
    print(predict_line)
    f.write(predict_line)
f.close()

# a small subset of our input/output pairs
Xlarge = X[:100000,:,:]
ylarge = y[:100000,:]

# TODO: fit to our larger dataset
model.fit(Xlarge, ylarge, batch_size=100, epochs=20, verbose = 0, callbacks=[TQDMNotebookCallback()])

# save weights
model.save_weights('model_weights/best_RNN_large_textdata_weights.hdf5')

# TODO: choose an input sequence and use the prediction function in the previous
# Python cell to predict 100 characters following it
# get an appropriately sized chunk of characters from the text
start_inds = [100, 105, 500, 2000]  # chose the same test indices as before

# save output
f =...


I am working on the file "RNN_project.ipynb".
The first task is:
With your chosen characters removed print out the first few hundred lines again just to double check that everything looks good.
Can you write Python code for it?

### print out the first 2000 characters of the raw text to get a sense of what we need to throw out
text[:2000]


I am working on the file "RNN_project.ipynb".
The first task is:
With your model built you can now fit the model by activating the cell below!  Note: the number of epochs (np_epochs) and batch_size are preset (so we can all produce the same results).  You can choose to toggle the verbose parameter - which gives you regular updates on the progress of the algorithm - on and off by setting it to 1 or 0 respectively.
Can you write Python code for it?

# run your model!
model.fit(X_train, y_train, epochs=1000, batch_size=50, verbose=0)


I am working on the file "RNN_project.ipynb".
The first task is:
Wow - there's a lot of junk here (i.e., weird uncommon character combinations - as this first character chunk contains the title and author page, as well as table of contents)!  e.g., all the carriage return and newline sequences '\n' and '\r' sequences.  We want to train our RNN on a large chunk of real english sentences - we don't want it to start thinking non-english words or strange characters are valid! - so lets clean up the data a bit.
First, since the dataset is so large and the first few hundred characters contain a lot of junk, lets cut it out.  Lets also find-and-replace those newline tags with empty spaces.
Can you write Python code for it?

### find and replace '\n' and '\r' symbols - replacing them
text = text[1302:]
text = text.replace('\n',' ')    # replacing '\n' with '' simply removes the sequence
text = text.replace('\r',' ')


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
3. Make useful categorical variables out of the single column data by one-hot encoding it.
Can you write Python code for it?

y_train = keras.utils.to_categorical(y_train, 4)
y_val = keras.utils.to_categorical(y_val, 4)


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
4. Set Global Parameters
Can you write Python code for it?

epochsize = 150
batchsize = 24
shuffle = False
dropout = 0.1
num_classes = 4
input_dim = x_train.shape[1]
hidden1_dim = 30
hidden2_dim = 30
class_names = mapping.keys()


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Architecture
Can you write Python code for it?

plot_model(auto_classifier, to_file='images/robo2_auto_class_LR.png', show_shapes=True, show_layer_names=True, rankdir='LR')

IPython.display.Image("images/robo2_auto_class_LR.png")


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Comparison
The following data is from a paper published in March 2017. You can find that here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5375835/
Can you write Python code for it?

IPython.display.Image("images/2018-01-25 18_44_01-PubMed Central, Table 2_ Sensors (Basel). 2017 Mar; 17(3)_ 549. Published online.png")


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Dimensionality reduction with stacked (multi hidden layer) autoencoder
Can you write Python code for it?

encoder_dim1 = 16
encoder_dim2 = 8
decoder_dim1 = 16

main_input = Input(shape=(input_dim,), dtype='float32', name='main_input')

encoding_layer1 = Dense(encoder_dim1, activation='relu', kernel_initializer='normal')(main_input)
encoding_layer2 = Dense(encoder_dim2, activation='relu', kernel_initializer='normal')(encoding_layer1)

decoding_layer1 = Dense(decoder_dim1
                        ,activation='relu'
                        ,kernel_initializer='normal')(encoding_layer2)
decoding_layer2 = Dense(input_dim
                        ,activation='sigmoid'
                        ,name='decoder_output'
                        ,kernel_initializer='normal')(decoding_layer1)

x = Dense(hidden1_dim, activation='relu', kernel_initializer='normal')(encoding_layer2)
x = Dropout(dropout)(x)
x = Dense(hidden2_dim, activation='relu', kernel_initializer='normal')(x)
x = Dropout(dropout)(x)

classifier_output = Dense(num_classes
                          ,activation='softmax'
                          ,name='main_output'
                          ,kernel_initializer='normal')(x)

stacked_auto_classifier = Model(inputs=main_input, outputs=[classifier_output, decoding_layer2])

stacked_auto_classifier.compile(optimizer=RMSprop(),
                        loss={'main_output': 'categorical_crossentropy', 'decoder_output': 'mean_squared_error'},
                        loss_weights={'main_output': 1., 'decoder_output': 1.},
                        metrics=['accuracy'])

plot_model(stacked_auto_classifier, to_file='images/stacked__auto_class.png', show_shapes=True, show_layer_names=True)

IPython.display.Image("images/stacked__auto_class.png")

stacked_auto_classifier.fit({'main_input': x_train},
                            {'main_output': y_train, 'decoder_output': x_train},
                            epochs=epochsize,
                            batch_size=batchsize,
                            shuffle=shuffle,
                            validation_split=0.05,
                            verbose=0)

stacked_score =...


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Import and basic data inspection
Can you write Python code for it?

# import
data_raw = pd.read_csv('data/sensor_readings_24.csv', sep=",", header=None)
data = data_raw.copy()


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
One can easily see that our results are better.
Dimensionality Reduction with Single Hidden Layer Autoencoder
Can you write Python code for it?

encoder_dim = 8
hidden1_dim = 30
hidden2_dim = 30

main_input = Input(shape=(input_dim,), dtype='float32', name='main_input')

encoding_layer = Dense(encoder_dim, activation='relu', kernel_initializer='normal')
encoding_layer_output = encoding_layer(main_input)
decoding_layer_output = Dense(input_dim
                              ,activation='sigmoid'
                              ,name='decoder_output'
                              ,kernel_initializer='normal')(encoding_layer_output)

x = Dense(hidden1_dim, activation='relu', kernel_initializer='normal')(encoding_layer_output)
x = Dropout(dropout)(x)
x = Dense(hidden2_dim, activation='relu', kernel_initializer='normal')(x)
x = Dropout(dropout)(x)

classifier_output = Dense(num_classes
                          ,activation='softmax'
                          ,name='main_output'
                          ,kernel_initializer='normal')(x)

auto_classifier = Model(inputs=main_input, outputs=[classifier_output, decoding_layer_output])

auto_classifier.compile(optimizer=RMSprop(),
                        loss={'main_output': 'categorical_crossentropy', 'decoder_output': 'mean_squared_error'},
                        loss_weights={'main_output': 1., 'decoder_output': 1.},
                        metrics=['accuracy'])

plot_model(auto_classifier, to_file='images/robo4_auto_class_LR.png', show_shapes=True, show_layer_names=True)

IPython.display.Image("images/robo4_auto_class_LR.png")

auto_classifier.fit({'main_input': x_train},
                    {'main_output': y_train, 'decoder_output': x_train},
                    epochs=epochsize,
                    batch_size=batchsize,
                    shuffle=shuffle,
                    validation_split=0.05,
                    verbose=0)

score = auto_classifier.evaluate(x=x_val, y=[y_val, x_val], verbose=1)[3]
print(score)

fig = plt.figure(figsize=(20,10))
plt.plot(auto_classifier.history.history['val_main_output_acc'])
plt.plot(auto_classifier.history.history['main_output_acc'])
plt.axhline(y=score,...


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Prediction for a classifier with a dimension of 16 (not for report)
Can you write Python code for it?

# Compute confusion matrix
cnf_matrix = confusion_matrix(y_eval, auto_classifier.predict(x_val)[0].argmax(axis=-1))
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure(figsize=(20,10))
plot_confusion_matrix(cnf_matrix, classes=class_names,
                      title='Confusion matrix, without normalization')

# Plot normalized confusion matrix
plt.figure(figsize=(20,10))
plot_confusion_matrix(cnf_matrix, classes=class_names, normalize=True,
                      title='Normalized confusion matrix')

result_df.to_csv('results/robo_results.csv')


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Summary
Can you write Python code for it?


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
The dataframe consists of only positive values and the classes are encoded as strings in the variable with index 24
Can you write Python code for it?


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
The idea is:
- reduce the number of features encoding_dim iteratively by training an auto_classifier network
- after training, the weights of the first layer are analyzed.
- therefore sum up the weights for each input node and take the encoding_dim number as new features
- test the new features with the pre- defined neural net and see which selection is best.
Can you write Python code for it?

while encoding_dim > 0:
    main_input = Input(shape=(input_dim,), dtype='float32', name='main_input')

    encoding_layer = Dense(encoding_dim, activation='relu', name='encoder', kernel_initializer='normal')
    encoding_layer_output = encoding_layer(main_input)
    decoding_layer_output = Dense(input_dim, activation='sigmoid'
                                  ,name='decoder_output'
                                  ,kernel_initializer='normal')(encoding_layer_output)

    x = Dense(hidden1_dim, activation='relu', kernel_initializer='normal')(encoding_layer_output)
    x = Dense(hidden2_dim, activation='relu', kernel_initializer='normal')(x)

    classifier_output = Dense(num_classes, activation='softmax', name='main_output', kernel_initializer='normal')(x)

    auto_classifier = Model(inputs=main_input, outputs=[classifier_output, decoding_layer_output])

    auto_classifier.compile(optimizer=RMSprop(),
                            loss={'main_output': 'categorical_crossentropy', 'decoder_output': 'mean_squared_error'},
                            loss_weights={'main_output': 1., 'decoder_output': 1.},
                            metrics=['accuracy'])

    auto_classifier.fit({'main_input': x_train},
                        {'main_output': y_train, 'decoder_output': x_train},
                        epochs=epoch_size,
                        batch_size=batch_size,
                        shuffle=shuffle,
                        validation_split=0.1,
                        verbose=0)

    accuracy = auto_classifier.evaluate(x=x_val, y=[y_val, x_val], verbose=1)[3]
    result2['encoding_dim/features'].append(encoding_dim)
    result2['compression_level'].append(1 - encoding_dim/24)
    result2['auto_classifier_acc'].append(accuracy)

    encoding_weights = encoding_layer.get_weights()
    sum_of_weights = {index: item.sum() for (index, item) in enumerate(encoding_weights[0])}

    weights = sum_of_weights
    features = []
    for i in range(encoding_dim):
        max_key = max(weights, key=lambda key:...


I am working on the file "ROBO_Autoencoder.ipynb".
The first task is:
Whats the distribution of the classes?
Can you write Python code for it?

df_tab = data_raw
df_tab[24] = df_tab[24].astype('category')
tab = pd.crosstab(index=df_tab[24], columns="frequency")
tab.index.name = 'Class/Direction'
tab/tab.sum()


I am working on the file "ROBO_SAE.ipynb".
The first task is:
1. Take a random sample of 90% of the rows from the dataframe. To ensure reproducability the random_state variable is set. The other 10% are placed aside for validation after training. The last column is the class column and is stored in the y variables respectively.
Can you write Python code for it?

data_train = data.sample(frac=0.9, random_state=42)
data_val = data.drop(data_train.index)

df_x_train = data_train.iloc[:,:-1]
df_y_train = data_train.iloc[:,-1]

df_x_val = data_val.iloc[:,:-1]
df_y_val = data_val.iloc[:,-1]


I am working on the file "ROBO_SAE.ipynb".
The first task is:
2. Normalization between 0 and 1
Can you write Python code for it?

x_train = df_x_train.values
x_train = (x_train - x_train.min()) / (x_train.max() - x_train.min())
y_train = df_y_train.values
y_train_cat = y_train

x_val = df_x_val.values
x_val = (x_val - x_val.min()) / (x_val.max() - x_val.min())
y_val = df_y_val.values
y_eval = y_val


I am working on the file "ROBO_SAE.ipynb".
The first task is:
Classification
Can you write Python code for it?

input_img = Input(shape=(input_dim,))
sae_classifier_encoded1 = Dense(16, activation='relu')(input_img)
sae_classifier_encoded2 = Dense(8, activation='relu')(sae_classifier_encoded1)
class_layer = Dense(num_classes, activation='softmax')(sae_classifier_encoded2)

sae_classifier = Model(inputs=input_img, outputs=class_layer)

sae_classifier.layers[1].set_weights(autoencoder1.layers[1].get_weights())
sae_classifier.layers[2].set_weights(autoencoder2.layers[1].get_weights())
sae_classifier.compile(loss='binary_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])

sae_classifier.fit(x_train, y_train
               , epochs=50
                , verbose=True
               , batch_size=24
               , shuffle=True)

score5 = sae_classifier.evaluate(x_val, y_val)
print('Test accuracy:', score5[1])


I am working on the file "ROBO_SAE.ipynb".
The first task is:
Import and basic data inspection
Can you write Python code for it?


I am working on the file "ROBO_SAE.ipynb".
The first task is:
One can easily see that our results are better. So we go further with that result and check how good our SAW might become.
Stacked Autoencoder
For this dataset we decided to go with a 24-16-8-16-24 architecture.
First layer
Can you write Python code for it?

input_img = Input(shape=(input_dim,))
encoded1 = Dense(16, activation='relu')(input_img)
decoded1 = Dense(input_dim, activation='relu')(encoded1)
class1 = Dense(num_classes, activation='softmax')(decoded1)

autoencoder1 = Model(input_img, class1)
autoencoder1.compile(optimizer=RMSprop(), loss='binary_crossentropy', metrics=['accuracy'])
encoder1 = Model(input_img, encoded1)
encoder1.compile(optimizer=RMSprop(), loss='binary_crossentropy')

autoencoder1.fit(x_train
                 , y_train
                 , epochs=50
                 , batch_size=24
                 , shuffle=True
                 , verbose=False
                 )

score1 = autoencoder1.evaluate(x_val, y_val, verbose=0)
print('Test accuracy:', score1[1])


I am working on the file "ROBO_SAE.ipynb".
The first task is:
Plot a two dimensional representation of the data
Can you write Python code for it?

third_layer_code = encoder2.predict(encoder1.predict(x_train))

encoded_4_input = Input(shape=(8,))
encoded4 = Dense(2, activation='sigmoid')(encoded_4_input)
decoded4 = Dense(8, activation='sigmoid')(encoded4)
class4 = Dense(num_classes, activation='softmax')(decoded4)

autoencoder4 = Model(encoded_4_input, class4)
autoencoder4.compile(optimizer=RMSprop(), loss='binary_crossentropy', metrics=['accuracy'])
encoder4 = Model(encoded_4_input, encoded4)
encoder4.compile(optimizer=RMSprop(), loss='binary_crossentropy')

autoencoder4.fit(third_layer_code
                 , y_train
                 , epochs=100
                 , batch_size=24
                 , shuffle=True
                 , verbose=True
                 )

third_layer_code_val = encoder2.predict(encoder1.predict(x_val))

score4 = autoencoder4.evaluate(third_layer_code_val, y_val, verbose=0)
print('Test loss:', score4[0])
print('Test accuracy:', score4[1])

fourth_layer_code = encoder4.predict(encoder2.predict(encoder1.predict(x_train)))

value1 = [x[0] for x in fourth_layer_code]
value2 = [x[1] for x in fourth_layer_code]
y_classes = y_train_cat

data = {'value1': value1, 'value2': value2, 'class' : y_classes}
data = pd.DataFrame.from_dict(data)
data.head()

groups = data.groupby('class')

# Plot
fig, ax = plt.subplots(figsize=(20,10))
# plt.figure(figsize=(20,10))
ax.margins(0.05) # Optional, just adds 5% padding to the autoscaling
for name, group in groups:
    ax.plot(group.value1, group.value2, marker='o', linestyle='', ms=3, label=name, alpha=0.7)
ax.legend()

plt.show()


I am working on the file "ROBO_SAE.ipynb".
The first task is:
The Move_Forward and the Sharp-Right-Turn Class combine nearly 80% of all observated classes. So it might happen, that the accuracy may still be high with around 75% although most of the features are eliminated.
Preprocessing
0. Mapping integer values to the classes.
Can you write Python code for it?

mapping = {key: value for (key, value) in zip(data[24].unique(), range(len(data[24].unique())))}
print(mapping)
data.replace({24:mapping}, inplace=True)

data[24].unique()


I am working on the file "ROBO_SAE.ipynb".
The first task is:
The dataframe consists of only positive values and the classes are encoded as strings in the variable with index 24
Can you write Python code for it?


I am working on the file "ROBO_SAE.ipynb".
The first task is:
Whats the distribution of the classes?
Can you write Python code for it?


I am working on the file "ROBO_SAE_Comments.ipynb".
The first task is:
Comparison
The following data is from a paper published in March 2017. You can find that here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5375835/
Can you write Python code for it?


I am working on the file "ROBO_SAE_Comments.ipynb".
The first task is:
The classifier yields a test accuracy of 0.945054945055.
Can you write Python code for it?

# Plot normalized confusion matrix
plt.figure(figsize=(20,10))
plot_confusion_matrix(cnf_matrix, classes=class_names, normalize=True,
                      title='Normalized confusion matrix')


I am working on the file "ROC Curve and the UCI German Credit Data Set.ipynb".
The first task is:
"Leave one out", n-1 dummy variables fully describe the categorical feature.
Can you write Python code for it?

drop_dummies = ['CkgAcct_A14', 'CreditHistory_A32', 'Purpose_A43', 'Savings_A61', 'EmploymentTime_A73',
                'PersStatus_A93', 'OtherDebts_A101', 'Property_A123', 'OtherLoan_A143', 'Housing_A152',
                'Job_A173']
df.drop(drop_dummies, axis=1, inplace=True)


I am working on the file "ROC Curve and the UCI German Credit Data Set.ipynb".
The first task is:
Compute ROC curve parameters. ROC code from:
http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py
Can you write Python code for it?

n_classes = 2
fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(y_test_expanded[:, i], pred_proba[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

# Compute micro-average ROC curve and ROC area
fpr["micro"], tpr["micro"], _ = roc_curve(y_test_expanded.ravel(), pred_proba.ravel())
roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])