Pattern Search with the Knuth-Morris-Pratt (KMP) algorithm | by Max Lefarov | Towards Data Science

So here you are, sitting and being interviewed for the position in your dream company. The interviewer, your fellow developer, whose seniority and proficiency are beyond doubt and cannot be challenged by anyone, presents you with a problem that boils down to pattern matching. Being happy and excited you announce that you would like to use a naive approach with O(nk) complexity. However, the interviewer is frowning, he’s disappointed. You can see in his eyes, filled with wisdom and experience, that he’s quickly losing interest in this conversation. His attention is fading and his thoughts are gravitating towards a free lunch, which he’ll soon get in one of those company’s legendary food courts. The rumors is that the retired Michelin chefs themselves are forging the food of gods in those shrines of catering. So what should you do, how to regain the interviewer’s favor, how to make sure that you’ll also be able to taste that fast-food miracle in a few months? The only way is to give your interviewer what he wants, the O(n) solution. “But which O(n) solution?!” — you ask me. No worries. I got your back. Please welcome our today’s guest the KMP (Knuth-Morris-Pratt) Pattern Search algorithm.
A good overview of KMP and the reasoning behind it is provided in this article on Medium. TL;DR when doing naive pattern matching every time you’ve encountered a mismatch between the pattern and the target string you move only one step forward in the target and start to re-match the entire pattern from the very beginning. KMP, on contrary, can help to skip several unnecessary comparisons, which will, of course, positively impact your complexity. How positively, you ask me? O(n) positively is KMP’s answer. But how does this tricky guy do that? Well, he’s using the help from his little friend an array of Longest Proper Prefix which is Suffix (LPS), which is constructed for the pattern prior to the main phase of the algorithm. First things first, let’s understand what is LPS (the essential building block of KMP) and how it can be constructed, then write an algorithm to do so. Once we have it in our arsenal, KMP won’t seem so scary anymore, believe me.
As the name suggests, the LSP array contains the information about the “Longest Proper Prefix which is Suffix”. No worries, I was also quite confused when I’ve seen this formulation for the first time. What does make a prefix “Proper” after all? Well, believe me, or not, but it turned out that it’s just any prefix of a word that is not equal to the word itself. So let’s take an example. For the word ACA we have the following proper prefixes: , A, AC, and the suffixes: , A, CA, ACA. Thus, the longest proper prefix that is equal to the suffix is A.
So the LPS array of some string at any position i contains the length of such a prefix for the sub-string from 0 till i (pattern[:(i+1)] for my Python friends out there). In the .gif below, you can see how the LPS array is constructed manually step by step for the pattern string ACABACACD. No rush here, take your time and make sure that everything is clear for you so far.
Well, now, when we understand what is an LPS array and have seen how to construct it manually, let’s write an algorithm to do so for any given pattern. I have bad news for you though. This algorithm requires us to operate two pointers (indices). Yes, I know, I know... These kinds of algorithms are also among my least favorite, partially because it’s quite challenging to write them in a clean “Pythonic” way, and partially because my short-term memory gives up quite quickly on a task of tracking more than one pointer at once. So, in order to understand the algorithm for LPS array construction, I would propose for you to think about it in terms of “Phases” and a state-machine. That is my favorite way of approaching a new algorithm, and I hope it might be useful for you as well. Below you’ll see the picture summarizing the algorithm, followed by a detailed description in the next paragraph. If you are the kind of person who would rather read through the code than any lengthy description, please jump straight to the implementation section.
As you might infer from the picture above, we’re working on a single pattern string and using two pointers which for now carry the abstract names “top” and “bottom”. The bottom pointer bp iterates over our pattern string. The top pointer tp is used to determine the LPS of the current sub-string. Additionally, we have the LPS array where the lengths of LPSs are stored. The size of LPS is equal to the pattern string’s size and is initialized with 0s.
The algorithm begins with “Phase 1”, where we move the bottom pointer and trying to find a symbol in our pattern string that is equal to the first symbol (still tracked by the tp). This phase corresponds to the search of the sub-string with the LPS of length 1 (i.e., a single-letter prefix is equal to a single-letter suffix). Once found, we can write 1 into our LPS array at the position of the bp. Now we can shift to “Phase 2”. Here we move both pointers while they are pointing to the equal symbols and write tp+1, i.e. the length of LPS, to the LPS at the position of the bp. In this phase we’re continuously discovering the sub-string with the longer LPS with every subsequent step. But, it might very well be that we’ll eventually run out of luck and come to the situation where the pointers are pointing towards two different symbols. That indicates that we’ve found the sub-string whose LPS is shorter than the previous one.
“We are in the endgame now.”
In “Phase 3” we roll back the top pointer to find the new LPS (which is smaller than the LPS at the previous bottom index). “But where should I place the top pointer?” — you ask me. Let’s figure that out. First of all, let’s consider an example of such a situation to have something to look at. We’ll crop one frame from our .gif example from the previous section. Please see the picture below.
Well, we’ve discovered that the symbol at the top pointer (3 in our example) doesn’t match a symbol at the bottom pointer 7. Let’s consider the pattern’s sub-string till the bottom pointer exclusively (pattern[:bp] for those speaking Parseltongue). It’s guaranteed that its prefix from 0 to tp matches the suffix staring at (bp-tp), i.e.
pattern[:tp] == pattern[(bp-tp):bp] 
(shaded with light green in our example). This always holds due to the definition of “Phase 1” and “Phase 2”. Thus we can state that patterns[:tp]’s LPS, is equal to the ending part of the pattern[:bp]’s suffix (shaded in blue). That allows us to move the top pointer to the next index after the patterns[:tp]’s LPS, which is already stored in LPS[tp-1] (blue 1 in the example above). The described procedure is repeated until the symbol at the new top pointer position matches the symbol at the bottom pointer (transition to the second phase), or the top pointer has become 0 (back to “Phase 1”). The algorithm terminates when the iteration over the pattern string with the bottom pointer is completed.
Below you can find the “Pythonic” implementation of the algorithm for the LPS array construction. “Phases” are implemented in a reverse order here for the readability concerns. If you didn’t skip the previous section, please replace prefi and i with the “top” and “bottom” pointers in your mind.
Now, finally, we’re at the “main” part of KMP. But actually, there’s not too much to do for us in this main part. Once we have the LPS array constructed for our pattern string, KMP becomes quite similar to the naive pattern search. The only difference is that on a mismatch, instead of moving one symbol forward in a target text and starting from the very beginning of a pattern string, KMP uses LPS to continue from exactly the same place where the mismatch occurred. First, let’s look at the KMP’s main part “Phases” and the state-machine, similarly to what we did for the LPS construction. Then, we’ll describe in detail how exactly KMP manages to continue from the mismatch position.
As demonstrated in the picture above, we have a target string (black) and a pattern string (blue). We iterate over the target with the target pointer and over the pattern string with the pattern pointer. Similar to the LPS construction, the main part of KMP consists of three phases. “Phase 1” and “Phase 2” are identical to the naive pattern search. In the first phase, we’re moving in a target until we find a symbol that matches the first symbol in a pattern string. That is where the transition into the second phase happens. In “Phase 2” both, the target and the pattern pointers, are moved while the symbols, that they point to, match. If the end of the pattern string is reached, the solution is found. If the mismatch is encountered the algorithm transitions to the third phase. That is the only phase different from the naive approach. In the naive pattern search when a mismatch is found, you’re moving the target pointer one symbol forward from the place where you’ve started the comparison, and the pattern pointer (index) is reset to 0. On contrary, when running KMP, the target pointer remains at the same place, but the pattern pointer is rolled back using the information from the LPS array. In order to understand how it’s done, let’s consider an example in the picture below.
So we did encounter the mismatch at some point in the comparison between the pattern and the target text. E is not equal to C, what a bummer. However, we can be sure that the pattern sub-string till the pattern pointer matches the target sub-string till the target pointer (red dashed line). Otherwise, we wouldn’t reach this point according to our definition of the second phase. Additionally, our LPS array for the current pattern sub-string tells us that the prefix of length 3 is equal to the suffix (shaded in green). With this information in hand, we can safely move the pattern pointer back to the fourth symbol (the first symbol after LPS) and leave the target pointer at the same place, as demonstrated below.
It’s guaranteed that the part of the pattern before the new pattern pointer matched the target string due to the definition of the LPS. It also holds true that we didn’t miss any possible matches between the current target pointer and the index at which we’ve started to compare last time. This shift or roll-back operation is repeated until a symbol at the new pattern pointer position matches one at the target pointer (transition to the second phase), or the pattern pointer is 0 (transition to the first phase).
We’re at the home stretch. Below you can find the code implementing the main part of KMP. As for the LPS algorithm, the “Phases” are written in reverse order here for better readability. patterni is our patter pointer from the above description and the target pointer holds a rather boring name i. One minor detail implemented below that was not discussed above is the search for all occurrences. I’m leaving it as an exercise for the curious reader (I’ll be happy to answer any question in the comments).
So, you declare your KMP based solution to your interviewer and hope that you didn’t miss some minor detail. But apparently, it’s not important anymore. With each word spoken, his excitement is rising, he cannot believe his eyes. Finally, he managed to find the candidate who will be a good fit for the team’s most important mission of re-implementing pattern search for every standard library in every language. Maybe you’ll get that free lunch after all..
Good luck with the behavioral questions, and let me know if I can help you with anything else 😃.