algorithm to quickly find a pattern in a block - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: February 14, 2025, 11:10:16 am
content: Need to find an pattern, array of uint8_t, so it's "hex searching", in a large block.Looking for the fastest methodUmmm, tempted to reuse what GNU/hexedit does, but first, let's look around ...

author: tggzzz
author_rank: Super Contributor
timestamp: February 14, 2025, 12:06:43 pm
content: Quote from: DiTBho on February 14, 2025, 11:10:16 amSimple linear search of contiguous data, or regular expression pattern matching?Bit/byte/word/etc alignment?One way or another you will end up with a FSM, whether or not you realise it. Merely by thinking that way will suggest solutions.

author: artag
author_rank: Super Contributor
timestamp: February 14, 2025, 12:51:48 pm
content: Are you looking for a specific pattern in terms of a sequence of bytes, or trying to find something unknown but repetitive such as a musical note in a digitised audio sample ?The first might be a simple search but the second would have attributes of scaling in both amplitude and time and would probably be based on an autocorrelation.

author: DiTBho
author_rank: Super Contributor
timestamp: February 14, 2025, 12:59:20 pm
content: Quote from: artag on February 14, 2025, 12:51:48 pmsequence of bytes

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 14, 2025, 01:10:12 pm
content: If you don't care for performance, use a naive linear search.But since your said "quickly", I recommend using what's been used for searching text for a few decades: the Knuth-Morris-Pratt algorithm.https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm

author: tggzzz
author_rank: Super Contributor
timestamp: February 14, 2025, 01:21:01 pm
content: Quote from: DiTBho on February 14, 2025, 12:59:20 pmABCDE or A?CDE or A*E ?

author: DiTBho
author_rank: Super Contributor
timestamp: February 14, 2025, 01:51:44 pm
content: Quote from: tggzzz on February 14, 2025, 01:21:01 pmgood question, I need bothit can be implemented with two different algorithms, in two different modules.It will be part of a firmware I am writing, kind of debugging feature of a "monitor"It needs to search things into buffer of ram and device buffer.the bad thing is that they are not always stringsand that often, as you just pointed out, there can be small variations (*)ABCDE -> A*CDE(*) means whatever, 1 byte of freedom, covering 0x00 ... 0xff

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 14, 2025, 01:57:22 pm
content: KMP can be used on any series of integer values.Adding wildcards though will require some changes.

author: DiTBho
author_rank: Super Contributor
timestamp: February 14, 2025, 01:59:54 pm
content: Quote from: SiliconWizard on February 14, 2025, 01:10:12 pmuhm, I implemented the KMP-algo in my myC text editor myED.I can reuse some of its code for the monitor/mips(1),but this time it's not "substring search" but "hex search".Even worse, with 1 byte of freedom.(1) guess what for? for the damn rb532a routerhacking and reverse engineering is the only way ...

author: tggzzz
author_rank: Super Contributor
timestamp: February 14, 2025, 02:44:06 pm
content: Quote from: DiTBho on February 14, 2025, 01:51:44 pmIn many regexp libraries "?"=one, "*"=0 to many. The difference matters, hence the differences between grep, egrep and fgrep utilities.

author: spostma
author_rank: Regular Contributor
timestamp: February 14, 2025, 08:59:42 pm
content: the Boyer-Moore serach algorithm could be useful for you:https://www.geeksforgeeks.org/boyer-moore-algorithm-for-pattern-searching/

author: Analog Kid
author_rank: Super Contributor
timestamp: February 14, 2025, 09:53:07 pm
content: This is similar to a parsing problem.My go-to method for parsing is to implement a state machine (technically a finite-state automaton or FSA). Easily coded: you have a loop that fetches the next character from the stream, then a jump table that dispatches to code stubs depending on the character seen (which can either be a single character or a class of characters, like numeric digits).I use assembly language, but this can be done in any language, like C: it's driven by the jump table which has one row for each state, and columns corresponding to the character (or class of character) just seen.

author: DiTBho
author_rank: Super Contributor
timestamp: February 14, 2025, 10:30:17 pm
content: Quote from: spostma on February 14, 2025, 08:59:42 pmgoing to implement it, thanks!

author: benseno
author_rank: Regular Contributor
timestamp: February 14, 2025, 11:49:38 pm
content: May be, comparing the sum (add here any operation with lowest cost resulting in quasi unique result) of character values within sliding window substring (whose length is equal ot the length of patter) to the sum of characters in the pattern. Fast shifting of sliding window can be done by subtracting the oldest element and adding a newest one.Above can be applied to the strict patter check.Utilizing platform dependent asm instructions can also help. And another approach is to use multiple cores for paralleling the task.You can also check glogg - huge file viewer. It can efficiently search gigabytes big texts.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 15, 2025, 12:52:03 am
content: Quote from: DiTBho on February 14, 2025, 01:59:54 pmIt's just a sequence of integers. What exactly do you see as a difference? You may of course have to change the data types but the algorithm is the same.Quote from: DiTBho on February 14, 2025, 01:59:54 pmYes, as I mentioned, it's *the* difference: adding wildcards. But from what I get, your wildcards are restricted to 1 character only, so that should make it easier.?probably means "any value" (but there must be exactly one in that place)*usually means zero or more values (in your case, it would seem you only want 1 value of "freedom", so does that mean that the*is none, or any value?

author: Tation
author_rank: Regular Contributor
timestamp: February 15, 2025, 08:01:27 am
content: https://www.google.fr/url?sa=t&source=web&rct=j&opi=89978449&url=https://igm.univ-mlv.fr/~lecroq/articles/cl2008.pdf&ved=2ahUKEwjpn5PZlsWLAxWITqQEHXWzDAMQFnoECCEQAQ&usg=AOvVaw3UPBGG_I-mK6Th2mO18Z3yIt seems that, from Boyer-Moore up, the "best" algorithm depends on exactly what kind of search is needed (size of alphabet, size of pattern...). The paper does not include the Knuth-Morris-Pratt, though.In any case I wonder if there's much to gain implementing any of these against using any available search()/find() function in some common library.

author: DiTBho
author_rank: Super Contributor
timestamp: February 15, 2025, 12:00:52 pm
content: Quote from: SiliconWizard on February 15, 2025, 12:52:03 amstrings are managed byxx_string_compare(), which considers the special char '\0' has end of string.hex-arrays should usexx_mem_compare(), which considers a pair of uint8 within a window

author: Traceless
author_rank: Frequent Contributor
timestamp: February 15, 2025, 12:06:13 pm
content: Quote from: DiTBho on February 14, 2025, 10:30:17 pmLet me save you some time there just usestd::search.

author: DiTBho
author_rank: Super Contributor
timestamp: February 15, 2025, 12:23:57 pm
content: Quote from: Tation on February 15, 2025, 08:01:27 ameh, there are some constraints I didn't mention.I'm not using a standard C compiler, but rather myC, myown C-like compiler.This excludes C++ and existing standard-C libraries, unless they are "adapted".

author: DiTBho
author_rank: Super Contributor
timestamp: February 15, 2025, 12:26:22 pm
content: This algo needs to do search from 32kbyte (hw buffer) up to 4Mbyte (rom0)While the size of searching key is always less than 512byte.Edit:The CPU is a single core MIPS32 at 400Mhz, with 8Kbyte of D/I cache.

author: Tation
author_rank: Regular Contributor
timestamp: February 15, 2025, 03:39:51 pm
content: 512 byte patterns are "long" patterns. In this case Backward Oracle Matching (BOM) algorithms and derivatives are thought to be the most efficient.But I do not think that the benefit over Boyer-Moore (maybe around 50 % in selected cases, definitely not orders of magnitude) worths the effort.

author: radiolistener
author_rank: Super Contributor
timestamp: February 15, 2025, 09:51:22 pm
content: Here is AI ChatGPT-4-turbo version (a little bit modified and fixed for better readability)Code:#include <stdio.h>#include <stdint.h>int indexOf(uint8_t *buffer, size_t bufferSize, uint8_t *pattern, size_t patternSize) {if (patternSize == 0 || bufferSize < patternSize) {return -1;}size_t badChar;for (size_t i = 0; i < 256; i++) {badChar = patternSize;}for (size_t i = 0; i < patternSize - 1; i++) {badChar[pattern] = patternSize - 1 - i;}size_t i = 0;while (i <= bufferSize - patternSize) {size_t j = patternSize - 1;while (j < patternSize && buffer[i + j] == pattern) {if (j == 0) {return i; // Pattern found}j--;}i += badChar[buffer[i + patternSize - 1]]; // Shift using bad character rule}return -1; // Pattern not found}int main() {uint8_t buffer[] = {0x01, 0x02, 0x03, 0x04, 0x05, 0x00, 0x07, 0x08, 0x04, 0x05, 0x06, 0x07, 0x08};uint8_t pattern[] = {0x04, 0x05, 0x06};int result = indexOf(buffer, sizeof(buffer), pattern, sizeof(pattern));if (result < 0) {printf("Pattern not found\n");} else {printf("Pattern found at index: %d\n", result);}return 0;}

author: radiolistener
author_rank: Super Contributor
timestamp: February 17, 2025, 07:55:05 am
content: Here is AI Grok2 version, with no fixes:Code:int indexOf(uint8_t *buffer, size_t bufferSize, uint8_t *pattern, size_t patternSize) {if (bufferSize < patternSize) {return -1; // Pattern cannot fit in buffer}for (size_t i = 0; i <= bufferSize - patternSize; ++i) {if (memcmp(&buffer, pattern, patternSize) == 0) {return (int)i; // Match found, return index}}return -1; // Pattern not found}The same AI Grok2 with asking to not use memcmp:Code:int indexOf(uint8_t *buffer, size_t bufferSize, uint8_t *pattern, size_t patternSize) {if (bufferSize < patternSize) {return -1; // Pattern cannot fit in buffer}for (size_t i = 0; i <= bufferSize - patternSize; ++i) {size_t j;for (j = 0; j < patternSize; ++j) {if (buffer[i + j] != pattern) {break; // No match, continue searching}}if (j == patternSize) {return (int)i; // Match found, return index}}return -1; // Pattern not found}

author: tggzzz
author_rank: Super Contributor
timestamp: February 17, 2025, 10:58:21 am
content: Anybody whothinksfeels posting LLM responses to technical questions is useful should read these two summaries of the competence of LLMs.https://www.science.org/content/blog-post/evaluation-deep-research-performancehttps://www.bbc.com/news/articles/c0m17d8827ko

author: DiTBho
author_rank: Super Contributor
timestamp: February 17, 2025, 11:15:00 am
content: Not interested in answers from chatGPT&C.Don't post, please

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 17, 2025, 12:11:16 pm
content: Note that Boyer-Moore can be more efficient than KMP but its worst case complexity is much worse. So if you are dealing with long patterns that are not very likely to repeat (contrary to letter patterns in text), I think I would personally favor KMP.That said, the complexity of either may be significantly different when introducing wildcards. I haven't thought about it in details. I've implemented both wildcard match and KMP, but never both combined.

author: DiTBho
author_rank: Super Contributor
timestamp: February 17, 2025, 01:59:43 pm
content: Quote from: SiliconWizard on February 17, 2025, 12:11:16 pmhow did you implemented wildcards?

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 18, 2025, 05:53:56 am
content: I'll give you the base version, which I think is pretty much the common way of implementing it efficiently. It handles patterns that contain '?' and '*', with the usual definitions for both wildcards. This version is for zero-terminated strings, but it's easy to modify it for arrays of bytes associated with a length. Returns true if the string matches the pattern, false otherwise.Code:bool PatternMatch(const char *szText, const char *szPattern){if ((szText == NULL) || (szPattern == NULL))return false;if (*szPattern == '\0')return (*szText == '\0');const char *pText = NULL, *pPattern = NULL;while (*szText != '\0'){if ((*szText == *szPattern) || (*szPattern == '?')){szText++;szPattern++;}else if (*szPattern == '*'){pText = szText;pPattern = szPattern;szPattern++;}else if (pPattern != NULL){szText = pText + 1;szPattern = pPattern + 1;pText++;}else{return false;}}while (*szPattern == '*')szPattern++;return (*szPattern == '\0');}