Achieving Order From Randomness
By: Agni Kumar
Tags: Mathematics, Education, Data Science, Coding, Editors Pick

Encoding and decoding

We’ll evaluate a decryption method that attempts to unscramble a text encoded by a substitution cipher, by repeatedly (with some exchange probability) randomly switching all incidences of two letters at a time. We would like to know whether this method will in general result in a successful decoding, and if so, how long we can expect this process to take (i.e., how many steps before we reach the original text).

We model the problem as an absorbing Markov chain, which is a Markov chain containing states that are impossible to leave. We present both theoretical and experimental results for small text lengths, which differ for reasons to be explored in the future. In addition to quantifying the relationship between the time to decode and text length, we also examine how time to decode changes as the exchange probability varies, as well as when the difference between a scrambled text’s formulation and those of general English words increases.

Unscrambling text

We give a concrete application of Markov chains, specifically in the context of understanding randomness. We present a possible message decryption method based on random substitutions. Given a message encoded using a simple substitution cipher, we want to decode the ciphertext using the following (to simplify the problem, we restrict the set of characters for use in the substitution cipher to include only the characters already included in the original message):

We obtain an incidence matrix of the English language, where rows and columns are the letters of the alphabet and entries are the probabilities with which two symbols typically follow each other.

of the English language, where rows and columns are the letters of the alphabet and entries are the probabilities with which two symbols typically follow each other. We obtain an encoded version of the text using a simple substitution cipher, restricted to the characters included in the original text. For simplicity, we take the original text to be an English word with non-repeating characters.

Comparing the incidence matrix of the encoded text with the one for the English language (referred to as typical), we exchange the positions of two randomly chosen characters and then run the same comparison again. If the text has become more “typical’’ (closer by Euclidean distance to the typical incidence matrix), we accept the change and continue with the new text. However, if the text has become less typical, we accept the change only with some set probability (referred to as the exchange probability, or switch_prob ) and discard it otherwise. If on the n-th turn we obtain the original message, we stop the decoding process and record the number of steps.

Absorbing Markov chains

An absorbing Markov chain is a Markov chain such that there exists at least one state which is impossible to leave, called an absorbing state, and from any state it is possible to get to an absorbing state (not necessarily in one step).

Our message decryption method can be represented by a Markov chain since the probability distribution for the word we obtain on each subsequent turn is not dependent on any of the words that came before in the chain. It can be represented by an absorbing Markov chain because, once we have successfully decoded the text, we stop switching letters, which is represented by setting the probability of getting the decoded text on the next turn, given that the current state is the decoded text, to one (and consequently the probabilities of getting any other encoded versions on the next turn to zero).

Decryption method

Recall that we originally had two questions about this decryption method:

Will the method definitely result in a successful decoding?

If so, how long we can expect this process to take?

To answer the first question, we make use of the theorem below.

Theorem 1 — The probability that an absorbing Markov chain will eventually reach an absorbing state is 1.

Since our Markov chain has only one absorbing state, namely the original text, the probability that the chain will eventually reach the desired state is 1. Our decoding method is guaranteed to work.

To answer the second question, we write the transition matrix associated with our chain as the block matrix

Here, Q and R are t × t and t × 1 dimensional matrices, respectively, where t is the number of non-absorbing states, i.e., the number of possible encrypted versions of the text which are not the original text. The row {0, 0, …, 0, 1} represents the original text. We define the fundamental matrix N = (I−Q)⁻¹, if this exists.

Theorem 2 — The matrix N as defined above exists, and the expected number of steps from the state represented by the i-th row of Q is equal to the i-th entry of the vector Nc, where c = (1, 1, …, 1)ᵀ.

Proofs for the theorems above are contained here. Let’s use these to calculate the expected number of steps to decode various permutations of a three-letter English word, ‘yes’.

Experiments

Let’s compute the fundamental matrix and expected number of steps for each permutation when the original word is ‘yes’. Ordering the permutations, from least to most typical, we have: ‘sye’, ‘eys’, ‘sey’, ‘yse’, ‘esy’, and ‘yes’.

The transition matrix looks like so:

Then N = (I−Q)⁻¹ is the following, and Nc = (7.5, 5.5, 5.5, 6.5, 5.5)ᵀ. The expected numbers of steps to absorption from each of the states ‘esy’, ‘eys’, ‘sey’, ‘sye’, and ‘yse’ are 7.5, 5.5, 5.5, 6.5, and 5.5, respectively.

Simulations

We present simulations to evaluate whether the aforementioned method efficiently decodes texts. (In theoretical computer science, usually every polynomial time algorithm is considered efficient; an O(log n) algorithm is considered highly efficient, since as n increases, the ratio of the number of operations to the size of the input decreases and tends to zero.)

Text data

To begin, we constructed the English incidence matrix given depicted below using frequencies of bigrams (pairs of consecutive letters). Row labels are the first letters of the bigrams and column labels are the second. Cells are colored according to their corresponding bigrams’ probabilities, where warm colors indicate higher values and cool colors indicate lower ones.

English incidence matrix

For a given text, we constructed an incidence matrix for it, by examining its bigram frequencies. We also code the functions get_subpart_typical(text, typical) , to obtain a submatrix from the English incidence matrix by combining the rows and columns associated with a text’s characters, and get_dist_from_typical(text, typical) , to compute the Euclidean distance between a text’s incidence matrix and the typical one.

The decoding scheme was applied over texts of multiple lengths and over multiple exchange probabilities. The distribution of text lengths in the British National Corpus, from which texts were drawn, is displayed below. Given the time taken to run the simulations, ten random texts from each of seven length categories (lengths two through eight characters) were used.

Distribution of text lengths in characters

Decoding time by text length

To quantify the time taken to decode texts, we code the function decode(typical, text, encoded_text, switch_prob) , which computes the number of two-character exchanges required to obtain the original text. The function, which repeatedly randomly switches two letters of an encoded string with the goal of minimizing distance from a typical incidence matrix, was applied to several texts.

Below, switch_prob is taken to be 0.5. The plots depict distributions of the number of exchanges over various text lengths, where each point represents a different text upon which the experiment was run. The right plot is a zoomed-in version of the left. The box and whisker plots depict several measures of spread. For longer text lengths, the numbers of exchanges differ quite a bit between texts, but it is clear that over text length, the time to decode increases exponentially.

Exponential distributions of the number of exchanges over various text lengths

This trend is more clearly evident in the plots below, which depicts averages of the number of exchanges taken to decode the text over all experiments in each of the length classes. Error bars represent one standard deviation of uncertainty.

Average time to decide over various texts and text lengths

A log-log plot of the left panel of the figure above is useful for recognizing an exponential relationship between increasing text length and time to decode, given the linear trend observed here.

Exponential growth trend of decoding time

Decoding time by similarity to English

In addition, we examined how the number of exchanges changes as the distances between texts’ incidence matrices and their corresponding typical submatrices increase. A visualization is given below, where each point represents a text upon which the experiment was run. As expected, we see that longer texts are associated with smaller distances from their associated typical submatrices. That longer texts are associated with high decode times is also captured.

Number of exchanges as distances from typical English submatrices change

We also examined how the number of exchanges changes as the switch_prob is varied, as shown below. Each point represents a text upon which the experiment was run. For each exchange probability value, it is evident that the time to decode increases exponentially as text length increases linearly. For longer texts, the time to decode fluctuates wildly between text lengths; it is not completely clear that always accepting an exchange ( switch_prob = 1) yields a longer decode time, but this could be further investigated by experimenting over more texts with longer lengths.

Number of exchanges as exchange probabilities are varied

All code for this particular Markov chain application is given here. Through simulation, we examined trends for the number of transitions taken to reach a certain state (the original text) from a random start state (the encoded text). The experiments involved achieving order from randomness, and showed that the time taken to do so is exponential, but that varying the exchange probabilities on the transitions does not affect decode time much.