Simple RNN Deciphers Code
By: Kat He
Tags: Recurrent Neural Network, Naturallanguageprocessing, Cipher, Machine Learning, Neural Networks

Table of Contents

Overview Pipeline Tokenization Padding Define RNN Postprocessing Output at Every Step Conclusion

Overview

In this article, we will define a simple RNN to decrypt a Caesar Cipher. If you’re not familiar with RNNs, take a look at RNN Numerical Walkthrough + Code.

Caesar Cipher

A Caesar Cipher is a cipher that encodes sentences by replacing the letters by other letters shifted by a fixed size. For example, a Caesar Cipher with a left shift value of 3 will result in the following:

Input: ABCDEFGHIJKLMNOPQRSTUVWXYZ

Cipher: DEFGHIJKLMNOPQRSTUVWXYZABC

Notice that there is a 1-to-1 mapping for every character, where every input letter maps to the letter below it. Because of this property, we will use a character-level RNN for this cipher, although word-level RNNs may be more common in practice.

Why use an RNN?

A Caesar Cipher can be solved as a multiclass classification problem using a fully-connected feedforward neural network since each letter X maps to its cipher value Y. However, an RNN will outperform a fully-connected network for text input because it is able to learn temporal dependencies between the letters that form the words in sentences. Therefore, even though the Caesar Cipher is a toy example, an RNN can still learn this cipher faster and with greater accuracy than a basic fully-connected neural network.

Data

Plain text:

Data matures like wine, applications like fish.

If we have data, let’s look at data. If all we have are opinions, let’s go with mine.

Cipher text:

‘Gdwd pdwxuhv olnh zlqh, dssolfdwlrqv olnh ilvk.’

“Li zh kdyh gdwd, ohw’v orrn dw gdwd. Li doo zh kdyh duh rslqlrqv, ohw’v jr zlwk plqh.”

The RNN will receive as input the preprocessed cipher text, character by character, and output the predictions character by character. Accordingly, our defined RNN will have a character-level many-to-many (matching) architecture.

Pipeline

Preprocessing

Before the input can be given to the neural network, it needs to be preprocessed into a format that the model can understand. There are essentially 2 main steps in our preprocessing pipeline: Tokenization and Padding.

Postprocessing

Since neural networks map numbers to numbers, the numerical output of the model will need to mapped back to a letter in order to re-form the sentence.

Pipeline

Input → Tokenization → Padding → MODEL→ Num-to-Text

Now, we’ll send the following sentence through the pipeline:

Torture the data, and it will confess to anything.

Tokenization

Tokenization is essentially splitting a phrase, sentence, paragraph, or an entire text document into smaller units, such as individual words or terms. Each of these smaller units are called tokens

We are performing character-level tokenization in this article as opposed to word-level tokenization which may be more common in practice.

Convert to character tokens

Since there is a 1-to-1 mapping between characters, the input at each timestep is a single character. The characters need to be individually isolated and converted to lowercase. Punctuation marks such as commas and periods should map to themselves.

def generate_character_tokens(x):

return [c for c in x.lower()]

Output

[‘t’, ‘o’, ‘r’, ‘t’, ‘u’, ‘r’, ‘e’, ‘ ’, ‘t’, ‘h’, ‘e’, ‘ ’, ‘d’, ‘a’, ‘t’, ‘a’, ‘,’, ‘ ’, ‘a’, ’n’, ‘d’, ‘ ’, ‘i’, ‘t’, ‘ ’, ‘w’, ‘i’, ‘l’, ‘l’, ‘ ’, ‘c’, ‘o’, ’n’, ‘f’, ‘e’, ‘s’, ‘s’, ‘ ’, ‘t’, ‘o’, ‘ ’, ‘a’, ‘n’, ‘y’, ‘t’, ‘h’, ‘i’, ’n’, ‘g’, ‘.’]

Create Character IDs

Now that the input has been converted to individual character tokens, these tokens need to be mapped to integers. Every character will be assigned an integer value, non-letters included. Assuming that the only non-letters are commas, spaces, and periods, a potential mapping could be as follows:

char_to_int = {' ': 1, 'k': 2, 'f': 3, 'g': 4, 's': 5, 'h': 6, 'r': 7, 'y': 8, 'a': 9, 'c': 10, 'n': 11, 't': 12, 'u': 13, 'p': 14, ',': 14, '.': 15, 't': 16, 'q': 17, 'e': 18, 'w': 19, 'o': 20, 'x': 21, 'm': 22, 'v': 23, 'l': 24, 'z': 25, 'd': 26, 'i': 27, 'b': 28, 'j': 29} def get_character_ids(tokens):

return [char_to_int[t] for t in tokens]

Putting it all together

def our_tokenizer(x):

tokens = generate_character_tokens(x)

character_ids = get_character_ids(tokens)

return character_ids

Using Keras

Keras has a tokenizer function that we could use instead which will essentially give us the same output with a potentially different character to integer mapping.

from keras.preprocessing.text import Tokenizer def keras_tokenizer(x):

tokenizer = Tokenizer(character_level=True, lower=True)

tokenizer.fit_on_texts(x)

return tokenizer.texts_to_sequences(x)

Output

[16, 20, 7, 16, 13, 7, 18, 1, 16, 6, 18, 1, 26, 9, 16, 9, 14, 1, 9, 11, 26, 1, 27, 16, 1, 19, 27, 24, 24, 1, 10, 20, 11, 3, 18, 5, 5, 1, 16, 20, 1, 9, 11, 8, 16, 6, 27, 11, 4, 15]

Padding

When defining the model, we specify the size of the input layer so that the model knows what length to expect. After converting the text into character ids, we’ll convert all the sequences to the same length by adding padding to the shorter sequences.

First, we need to find the length of the longest sequence, then we’ll add or pad 0’s to the ends of the shorter sequences.

def my_padding(x, max_length):

pad_length = max(0, max_length - len(x))

return x + [0]*pad_length

Using Keras

Keras also has a pad_sequences function that we can use.

from keras.preprocessing.sequence import pad_sequences def keras_padding(x, max_length):

return pad_sequences(x, maxlen=max_length, padding='post')

Output

[16, 20, 7, 16, 13, 7, 18, 1, 16, 6, 18, 1, 26, 9, 16, 9, 14, 1, 9, 11, 26, 1, 27, 16, 1, 19, 27, 24, 24, 1, 10, 20, 11, 3, 18, 5, 5, 1, 16, 20, 1, 9, 11, 8, 16, 6, 27, 11, 4, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Model

The input sequences can now be given to the model. Here we’ve defined a simple RNN with one recurrent layer and one dense layer for outputting values at each time step.

from keras.layers import SimpleRNN, Input, Dense, TimeDistributed

from keras.models import Model

from keras.layers import Activation

from keras.optimizers import Adam

from keras.losses import sparse_categorical_crossentropy def simple_rnn(input_shape, learning_rate=1e-3):

input_seq = Input(input_shape)

rnn = SimpleRNN(64, activation='relu', return_sequences=True)(input_seq)

logits = TimeDistributed(Dense(plaintext_vocab_size))(rnn)

out = Activation('softmax')(logits) model = Model(inputs=input_seq, outputs=out)

model.compile(loss=sparse_categorical_crossentropy,

optimizer=Adam(learning_rate),

metrics=['accuracy'])



return model

input_shape: this is the shape of the padded vector of numbers, not including the batch size

return_sequences: specifies to return the entire output sequence, not just the last output in the output sequence

TimeDistributed( ): Applies the same Dense layer with the same weights to every time step

Activation(‘softmax’): squishes the logits into values between 0 and 1 that also sum to 1. Essentially, this activation function converts the logits into a categorical distribution.

The RNN achieves a > 99% accuracy after 4 epochs, trained on 8000 samples and validated on 2000 samples.

Postprocessing

The RNN outputs a vector of numbers which need to be converted back to characters.

[ 2.76532210e-03, 9.61597681e-01, 2.77274812e-04, 2.80881330e-04, 3.43341962e-04, ………, 4.12466412e-04, 4.33523121e-04]

Each vector is a categorical distribution over the possible unique characters. To get the prediction, we choose the argmax of the vector. Then, we map that index back to the original index by reversing the original mapping.

Recall that the original mapping is:

char_to_int = {' ': 1, 'k': 2, 'f': 3, 'g': 4, 's': 5, 'h': 6, 'r': 7, 'y': 8, 'a': 9, 'c': 10, 'n': 11, 't': 12, 'u': 13, 'p': 14, ',': 14, '.': 15, 't': 16, 'q': 17, 'e': 18, 'w': 19, 'o': 20, 'x': 21, 'm': 22, 'v': 23, 'l': 24, 'z': 25, 'd': 26, 'i': 27, 'b': 28, 'j': 29}

Then the inverse is:

int_to_char = {1: ' ', 2: 'k', 3: 'f', 4: 'g', 5: 's', 6: 'h', 7: 'r', 8: 'y', 9: 'a', 10: 'c', 11: 'n', 16: 't', 13: 'u', 14: ',', 15: '.', 17: 'q', 18: 'e', 19: 'w', 20: 'o', 21: 'x', 22: 'm', 23: 'v', 24: 'l', 25: 'z', 26: 'd', 27: 'i', 28: 'b', 29: 'j'} int_to_char[0] = '<padding'>

Recall that after the tokenization step, we added 0’s to pad our vectors. This padded value needs to accounted for in the mapping or we will get a KeyError.

We can simply grab the characters using the argmax function, then index the mapping to get the characters.

import numpy as np def prediction_to_text(y):

# grab predictions

indices = np.argmax(y) # convert ints to characters

characters = []

for idx in indices:

c = int_to_char[idx]

characters.append(c)



return ' '.join(characters) # return sequence joined by space

Using Keras

Alternatively, the Keras tokenizer can take care of this part.

def prediction_to_text(y, tokenizer): # for entire sequence

# create mapping from tokenizer

int_to_char = {i: c for c, i in tokenizer.word_index.items()}

int_to_char[0] = '<padding>' # grab predictions

indices = np.argmax(y)



# convert ints to characters

characters = []

for idx in indices:

c = int_to_char[idx]

characters.append(c)



return ' '.join(characters) # return sequence joined by space

Output at Every Step

Original Input:

‘gdwd pdwxuhv olnh zlqh, dssolfdwlrqv olnh ilvk.’

After Preprocessing:

tokens: [g, d, w, d, , , p, d, w, x, u, h, v, , ,o , l, n, h, z, ,,, , l, q, h, d, s, s, o, l, f, d, w, l, r, q, v, , , o, l, n, h, i, l, v, k, ,. ,]

ids: [4, 26, 19, 26, 1, 14, 26, 19, 21, 13, 6, 23, 1, 20, 24, 11, 6, 1, 25, 24, 17, 6, 14, 1, 26, 5, 5, 20, 24, 3, 26, 19, 24, 7, 17, 23, 1, 20, 24, 11, 6, 1, 27, 24, 23, 2, 15]

Model Predictions for every character:

[26, 9, 16, 9, 1, 22, 9, 16, 13, 7, 18, 5, 1, 24, 27, 2, 18, 1, 19, 27, 11, 18, 14, 1, 9, 14, 14, 24, 27, 10, 9, 16, 27, 20, 11, 5, 1, 24, 27, 2, 18, 1, 3, 27, 5, 6, 15]

After Postprocessing:

‘data matures like wine, applications like fish.’

Conclusion

Although this may be a simple example, RNNs are advantageous for data with temporal dependencies. Language, especially, has these dependencies through the structure of characters in the words. If a cipher were to be encoded as a series or pattern, both of which depend on a past history, the advantages would be even greater.

I hope this article was informative. Please let me know if there are any errors or any improvements that can be made.