Understanding Bigrams: Cipher Decryption with Language Modeling and Genetic Algorithm
By: Bowen Chen
Tags: NLP, Genetic Algorithm, Data Science

Natural Language Processing

Understanding Bigrams: Cipher Decryption with Language Modeling and Genetic Algorithm

How this project helped me understand those legendary word bigrams

Photo by Philipp Katzenberger on Unsplash

Time to start some new learning adventures in natural language processing.

What? Isn’t this supposed to be kickstarted by a spam detection project? As you might probably think, cipher decryption might be a curious place to start an NLP series. This starting place has strong ties to do my desire to understand the word embedding from a deeper level. With the help from open-source NLP packages like SpaCy and NLTK, we are able to generate a fully functional word-embedding from a giant training text corpus, with a single line of code — seems like magic to me. While it is absolutely wonderful to leverage the power of these tools when we would like to achieve some convoluted tasks in a relatively short period of time without worrying too much about the details of the very first step, it is certainly not ideal for folks who are still trying to grasp the essence of this fascinating realm of natural language processing. With the help of an amazing course developed by Lazy Programmer (link to course here), I was able to implement this interesting project that enabled me to fully understand the underlying algorithm that built most of those word embedding — bigrams.

How is this possible? Let’s briefly talk about what cipher decryption is, and its applications.

What is Cipher Decryption?

Cipher decryption, in simple terms, is a process of substitution. We have 26 letters in English — and those letters form words, while words form sequences that convey meanings. If we randomly shuffle the order of the letters in these sentences, the meaning of these sequences will become murky. However, if we know a precise order of these random shuffles, we will be able to reverse the shuffle of these letters to make the sentence eloquent again. The letters mapping of this shuffling process is the cipher decryption map. An illustrative example is shown below.

Source: Matt Crypto from Wikipedia

We substitute a with d, b with e, … The word apple will become dssoh. since we remembered this one-to-one relationship, we will be able to get back to apple by reverse this mapping.

Relationship to Natural Language Processing

What does this have anything to do with NLP? Well, suppose you don’t know the above mapping, how would we guess what the mapping is? It turns out we could take an iterative approach to tackle this problem — starting with a randomly initialized mapping and iteratively minimize the discrepancies between the decryption results generated from real mapping and the results generated from estimated mapping. This discrepancy metric is just the log-likelihood of how “likely” this decrypted word is going to appear in English. Building this evaluation metric would require us to generate all the likelihoods of all letter combinations of all 26 letters — aa, ab, ac… yz, zz. Some combinations are more likely than others, and this likelihood will be inferred from some giant text corpora. Oh wait, that is what we call bigrams.

Overall Structure of This Project

Now we understand the relationships, we can talk about how this project is constructed. We start by building this language model from some giant text corpora that contain real language words and generate all the frequencies of those bigrams. This model will be saved into a python module named LanguageModel for frequent access to evaluate the correctness of the encryption process. Then we will construct the real encryption map, encryption, and decryption methods, stored in a python module called Encoder . Finally, we will also build a module GeneticAlgorithm , which handles the iterative guessing of this encryption map and select the better, more correct encryption maps until the best map is selected.

You might have questions regarding the object-oriented approach of this project — as the readability and build difficulties would be increased significantly from a functional approach. However, using OOP will allow us to have a nice user interface that contains only a few lines of code.

The overall project structure regarding files is the following

/cipher_decryption |--/model

|--language_model.py

|--encoder.py

|--genetic_algorithm.py |--/notebooks

|--Cipher Decryption.ipynb |--cipher.py

By setting up the folder hierarchy above, we will be able to import the modules like the following — the way we are all very familiar with.

The Cipher Decryption.ipynb contains the exploration and evaluation result, which could be found here.

Think about what we have just accomplished — We essentially recreated what we are so used to with those NLP packages, except this time we build those packages on our own this time. In fact, this is normally how software projects are built in real industries — downstream users should be able to have a simple user interface without worrying too much about the details of the component he/she is trying to use. In the next section, let’s dive into some implementation details of these 3 modules we mentioned above.

The Language Model

Language model module contains the bigram probabilities that could be accessed with one line of code

LanguageModel().log_M

This log_M property contains the log probabilities of occurrences of any two-letter combinations in English. Since there are 26 letters, log_M has a dimension of 26 × 26. An illustrative example M is shown below (all hypothetical values),

Referencing the example above, we get the probability of seeing a combination of “aa” is 0.04 while seeing a combination of “be” is 0.07. But how do we generate these values?

These values are generated by counting. For each two-character group in any word in our text corpus, we just count the number of occurrences and divide the count with the total number of words in the text corpus — this will produce a number between 0 and 1, which satisfies the definition of a probability.

The following snippet generates this combination matrix and saves it as an attribute of LanguageModel object.

Why could this two-letter combination set up properly model English sentences? It is based on an assumption called Markov Property — it states that the next occurrence of the letter only depends on the previous letter, but not any letter before that. In mathematical terms,

P(letter n | letter n-1, letter n-2, ...) = P(letter n | letter n-1)

This property essentially states that we can appropriately model the probability of the entire word by probabilities of two-letter combinations in that word.

The LanguageModel also have two methods that make it easy to get the log property of the word and sentences, which are simply iterative loops that calculate log probabilities of all the individual combinations of letters in words and sum them up to get the total log probabilities.

To get the sentence log probabilities, we can simply call the above get_log_word_probability multiple times and sum up the result.

As you might already see, we have built our own bigram probabilities matrix using the method of counting the proportion of the bigrams occurring in the training text corpus. Albeit other variations, most of the bigrams — regardless of a character or word levels — are built in a similar fashion. Also, we will call the build method (shown above) upon instantiation of the LanguageModel class, which allows LanugageModel automatically calculate the property log_M and make it available for downstream usage The full implementation of this module is linked here.

The Encoder

The encoder module will generate a true mapping of the encryption by random shuffling the 26 letters and maintain the one-to-one relationship. This functionality is accomplished by the private method build_cipher_mapping

We will also build methods that will perform the task of encoding/decoding input sentences

The full implementation of Encoder is shown in the link here.

The Genetic Algorithm

The GeneticAlgorithm module contains the iterative process that guesses and improves the estimated encryption map, with the help from the utility methods of the LanguageModel and Encoder modules. The general process is the following,

Randomly guess the encryption map multiple times and save it to a set Use the decode method from the Encoder module and decrypt the encoded input message according to the encryption maps generated in step 1 Evaluate each decoded message by calculating the log-likelihood of the message using the LanguageModel module’s get_sentence_log_probability method. Rank all log-likelihood values of the decrypted message from highest to lowest Select the top 5 encryption maps that have the highest log-likelihood values. Randomly swap 1 pair of mappings of those 5 encryption maps and save the results to the set. Repeat the process starting from step 2

After several iterations, the more “correct” encryption maps will be preserved — which will allow us to get closer and closer to the true encryption map. This iterative process is implemented in a class method called train

The full implementation of GeneticAlgorithm module is shown in the link here.

Results

What would the decryption progress look like over iterations? As a reminder, here is our input message from the novel Sherlock Holmes,

“I then lounged down the street and found, as I expected, that there was a mews in a lane which runs down by one wall of the garden. I lent the ostlers a hand in rubbing down their horses, and received in exchange twopence, a glass of half-and-half, two fills of shag tobacco, and as much information as I could desire about Miss Adler, to say nothing of half a dozen other people in the neighbourhood in whom I was not in the least interested, but whose biographies I was compelled to listen to.”