Change:
It seems that for UTF-8 strings the find_str method can return the byte index of the first matching substring, but there is no matching substring in fact. Here's the code: Compiled with "rustc " And it returns: Which indicates that the string "hah" was found at the end of the m string, but it isn't there - the end is "jah". Rust version: rustc 0.12.0-pre-nightly ( 2014-08-28 23:56:20 +0000) 64bit OS: OS X 10.9.4
cc
Pretty weird! This also returns true: . Also, if you delete any of the numbers it gives the correct result. Also gives the correct result if you change the first 'a' or 'h'.
Is there value in switching to a simpler algorithm?
That was my thought as well. What's the fastest alternative to Two-Way thats simple to implement? Boyer-Moore-Horspool?
I don't know about fastest; would be worth researching. In any case, we currently have an algorithm that takes infinite time to get the right answer for some inputs, so even temporarily switching back to a naive search would be a speed-up (correctness trumps performance :) ).
In addition to being buggy, the current implementation of findstr seems to be very inefficient. The following Rust code: needs about x1.6 time more than it's C++ (Apple LLVM version 5.1 (clang-503.0.40) x8664-apple-darwin13.3.0) equivalent: I was thinking that it was because Rust is just much slower than C++, but what makes me think that it's find_str's fault is the fact that when I change the "contains" condition to something like this in both programs: the Rust version (surprisingly?) becomes faster - the C++ one needs x2.5 more time to execute. The average length of the "auction" string is 70 and the "t" string is 7 on average.
I'm far from an expert on string matching algorithms, but after a bit of googling I found which benchmarks 85 different algorithms. I'm currently attempting to implement and benchmark three of them since they seem both generally fast and simple to implement: Horspool's Quick Search Tuned Boyer-Moore
The algorithm python uses for string search is pretty simple and pretty fast:
It seems to me that the naive algorithm with some optimizations (loop unrolling, code inlining, etc) gives the best results. Here's the libc++ implementation that turned out to be much faster than the Rust's one in my tests: (look for __search)
the paper you referred to seems to compare algorithms that find ALL occurrences, but we most of the time need to find only one. I'm not sure if implementing any of the algorithms you proposed in the standard library makes sense. I'd suggest using the naive algorithm, but optimize the code (unroll loops, inline code, write some unsafe code to disable bounds checks if necessary, etc). However, the algorithm used in Python might be applicable.
Please don't switch a programming environment's string matcher to anything with a quadratic worst-case. It is better to debug your implementation of two-way matching until it works. If you really, really want to switch your two-way matcher for something "simpler", the simplest thing I know that has a linear worst-case time complexity and constant storage overhead works like this: linear time and constant space, find the lexicographically largest suffix of the pattern. Write the pattern as uv, where v is the lex-largest suffix and u is the rest of the string. occurrences of v, again in linear time and constant space. (This is possible because v is lex-largest.) you find a v and it's at least len(u) characters after the last occurrence of v, check, using memcmp, that u precedes it. If it does, return a match; if not, do nothing. Step 1 is nontrivial, but it's slightly easier than two-way matching. It is Algorithm MS in Crochemore and Rytter's free "Text Algorithms" book. Steps 2 and 3 look exactly the same as two-way matching. Again, two-way matching is the way to go. Please use two-way matching for your stock string matcher, not a hacky implementation of a naive matcher or something that requires an auxiliary table!
Good idea. Maybe it's slow because it's broken. Ideally, the performance of the implementation should be comparable to the C++ one for searching for short patterns. I can perform my tests again when you fix it.
Thanks for the pointer to the Text Algorithms book, this seems much more readable than the paper. I'm going to give debugging Two-Way a shot.
is another example that currently comes up true, for future reference.
maybe randomized testing using quick check could help here. If it can easily come up with examples for the this bug.

Discussion:
memory: uint } <del> // This is the Two-Way search algorithm, which was introduced in the paper: // Crochemore, M., Perrin, D., 1991, Two-way string-matching, Journal of the ACM 38(3):651-675. </del> <ins> /* This is the Two-Way search algorithm, which was introduced in the paper: Crochemore, M., Perrin, D., 1991, Two-way string-matching, Journal of the ACM 38(3):651-675. Here's some background information. A *word* is a string of symbols. The *length* of a word should be a familiar notion, and here we denote it for any word x by |x|. (We also allow for the possibility of the *empty word*, a word of length zero). If x is any non-empty word, then an integer p with 0 < p <= |x| is said to be a *period* for x iff for all i with 0 <= i <= |x| - p - 1, we have x[i] == x[i+p]. For example, both 1 and 2 are periods for the string "aa". As another example, the only period of the string "abcd" is 4. We denote by period(x) the *smallest* period of x (provided that x is non-empty). This is always well-defined since every non-empty word x has at least one period, |x|. We sometimes call this *the period* of x. If u, v and x are words such that x = uv, where uv is the concatenation of u and v, then we say that (u, v) is a *factorization* of x. Let (u, v) be a factorization for a word x. Then if w is a non-empty word such that both of the following hold - either w is a suffix of u or u is a suffix of w - either w is a prefix of v or v is a prefix of w then w is said to be a *repetition* for the factorization (u, v). Just to unpack this, there are four possibilities here. Let w = "abc". Then we might have: - w is a suffix of u and w is a prefix of v. ex: ("lolabc", "abcde") - w is a suffix of u and v is a prefix of w. ex: ("lolabc", "ab") - u is a suffix of w and w is a prefix of v. ex: ("bc", "abchi") - u is a suffix of w and v is a prefix of w. ex: ("bc", "a") Note that the word vu is a repetition for any factorization (u,v) of x = uv, so every factorization has at least one repetition. If x is a string and (u, v) is a factorization for x, then a *local period* for (u, v) is an integer r such that there is some word w such that |w| = r and w is a repetition for (u, v). We denote by local_period(u, v) the smallest local period of (u, v). We sometimes call this *the local period* of (u, v). Provided that x = uv is non-empty, this is well-defined (because each non-empty word has at least one factorization, as noted above). It can be proven that the following is an equivalent definition of a local period for a factorization (u, v): any positive integer r such that x[i] == x[i+r] for all i such that |u| - r <= i <= |u| - 1 and such that both x[i] and x[i+r] are defined. (i.e. i > 0 and i + r < |x|). Using the above reformulation, it is easy to prove that 1 <= local_period(u, v) <= period(uv) A factorization (u, v) of x such that local_period(u,v) = period(x) is called a *critical factorization*. The algorithm hinges on the following theorem, which is stated without proof: **Critical Factorization Theorem** Any word x has at least one critical factorization (u, v) such that |u| < period(x). The purpose of maximal_suffix is to find such a critical factorization. */ </ins> impl TwoWaySearcher { fn new(needle: &[u8]) -> TwoWaySearcher { let (crit_pos1, period1) = TwoWaySearcher::maximal_suffix(needle, false);

period = period2; } <ins> // This isn't in the original algorithm, as far as I'm aware. </ins> let byteset = needle.iter() .fold(0, |a, &b| (1 << ((b & 0x3f) as uint)) | a); <del> // The logic here (calculating crit_pos and period, the final if statement to see which // period to use for the TwoWaySearcher) is essentially an implementation of the // "small-period" function from the paper (p. 670) </del> <ins> // A particularly readable explanation of what's going on here can be found // in Crochemore and Rytter's book "Text Algorithms", ch 13. Specifically // see the code for "Algorithm CP" on p. 323. </ins> // <del> // In the paper they check whether `needle.slice_to(crit_pos)` is a suffix of // `needle.slice(crit_pos, crit_pos + period)`, which is precisely what this does </del> <ins> // What's going on is we have some critical factorization (u, v) of the // needle, and we want to determine whether u is a suffix of // v.slice_to(period). If it is, we use "Algorithm CP1". Otherwise we use // "Algorithm CP2", which is optimized for when the period of the needle // is large. </ins> if needle.slice_to(crit_pos) == needle.slice(period, period + crit_pos) { TwoWaySearcher { crit_pos: crit_pos,

} } <ins> // One of the main ideas of Two-Way is that we factorize the needle into // two halves, (u, v), and begin trying to find v in the haystack by scanning // left to right. If v matches, we try to match u by scanning right to left. // How far we can jump when we encounter a mismatch is all based on the fact // that (u, v) is a critical factorization for the needle. </ins> #[inline] fn next(&mut self, haystack: &[u8], needle: &[u8], long_period: bool) -> Option<(uint, uint)> { 'search: loop {

} } <del> // returns (i, p) where i is the "critical position", the starting index of // of maximal suffix, and p is the period of the suffix // see p. 668 of the paper </del> <ins> // Computes a critical factorization (u, v) of `arr`. // Specifically, returns (i, p), where i is the starting index of v in some // critical factorization (u, v) and p = period(v) </ins> #[inline] fn maximal_suffix(arr: &[u8], reversed: bool) -> (uint, uint) { let mut left = -1; // Corresponds to i in the paper