Repository: quickwit-oss/tantivy (language: rust)

Issue:
RegexQuery Exception in Parsing Regular Expressions.
During the integration of `Tantivy` into `ClickHouse`, I encountered a need to adapt SQL's `LIKE` syntax. This led me to consider using Tantivy's `RegexQuery`. The proposed solution involves converting the `LIKE` matching strings from SQL statements into standard regular expressions, and then using Tantivy's `RegexQuery` to search for these expressions.

However, I have come across an issue. While certain regular expressions run effortlessly in Python, they fail to execute in `Tantivy`. 

Here's a specific example: 

I aim to find all strings containing the percentage symbol (%). In Python, this task is straightforward, but it seems to encounter difficulties when attempted in Tantivy.

```python
import re

docs = [
"Ancient empires rise and fall, shaping history's course.",
"Artistic expressions reflect diverse cultural heritages.",
"Social movements transform societies, forging new paths.",
"Economies fluctuate, % reflecting the complex interplay of global forces.",
"Strategic military camp%aigns alter the balance of power.",
]
compiled_regex = re.compile(".*\\%.*")
for line in docs:
    if compiled_regex.match(line):
        print(line)

========== terminal ==========
Economies fluctuate, % reflecting the complex interplay of global forces.
Strategic military camp%aigns alter the balance of power.

Process finished with exit code 0

```
```rust
use tantivy::collector::Count;
use tantivy::query::RegexQuery;
use tantivy::schema::{Schema, INDEXED, TEXT, FAST};
use tantivy::{Index, Document};


fn main() {
    let mut schema_builder = Schema::builder();
    let row_id = schema_builder.add_u64_field("row_id", FAST|INDEXED);
    let text = schema_builder.add_text_field("text", TEXT);
    let schema = schema_builder.build();
    let index = Index::create_in_ram(schema);

    let mut index_writer = index.writer_with_num_threads(8, 1024 * 1024 * 1024).unwrap();

    let str_vec: Vec<String> = vec![
        "Ancient empires rise and fall, shaping history's course.".to_string(),
        "Artistic expressions reflect diverse cultural heritages.".to_string(),
        "Social movements transform societies, forging new paths.".to_string(),
        "Economies fluctuate, % reflecting the complex interplay of global forces.".to_string(),
        "Strategic military camp%aigns alter the balance of power.".to_string(),
        ];
    
    for i in 0..str_vec.len() {
        let mut temp = Document::default();
        temp.add_u64(row_id, i as u64);
        temp.add_text(text, &str_vec[i]);
        let _ = index_writer.add_document(temp);
    }
    index_writer.commit().unwrap();

    let reader = index.reader().unwrap();
    let searcher = reader.searcher();
    let regex_query = RegexQuery::from_pattern(".*\\%.*", text).unwrap();
    let regex_result = searcher.search(&regex_query, &Count).expect("failed to search");

    println!("regex query result count:{:?}", regex_result);
}

========== terminal ==========
thread 'main' panicked at tests/regex_search.rs:37:65:
called `Result::unwrap()` on an `Err` value: InvalidArgument(".*\\%.*")
stack backtrace:
   0: rust_begin_unwind
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/result.rs:1652:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/result.rs:1077:23
   4: regrex_search::main
             at ./tests/regrex_search.rs:37:23
   5: core::ops::function::FnOnce::call_once
             at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```

I would like to understand the similarities and differences between the regular expressions used in `RegexQuery` and standard regular expressions. Where can I find information on this topic?


Discussion / hints:
Tantivy uses [`tantivy_fst::Regex`](https://docs.rs/tantivy-fst/latest/tantivy_fst/struct.Regex.html), the docs of which contain some information on how syntax and matching differs from the usual [`regex`](https://docs.rs/regex/latest/regex/) semantics.
Using only 

```rust
fn main() {
    tantivy_fst::Regex::new(".*\\%.*").unwrap();
}
```

as a test program, I cannot reproduce the above error though.
It seems that the version of `tantivy` I am using is `0.21.1`, which depends on `tantivy-fst` version `0.4.0`. I have confirmed that `tantivy-fst` version `0.5.0` can correctly parse this regular expression, while version `0.4.0` throws an error. Could you tell me which versions of `tantivy` use `tantivy-fst` version `0.5.0`? I am considering upgrading my `tantivy` version to resolve this issue.
@adamreichold 
I don't think there is released version of `tantivy` which yet which uses `tantivy-fst@0.5.0`. That said, using `tantivy-fst@0.4.0`, I was able to reproduce your error, i.e.

```console
thread 'main' panicked at src/main.rs:2:40:
called `Result::unwrap()` on an `Err` value: Syntax(Parse(Error { kind: EscapeUnrecognized, pattern: ".*\\%.*", span: Span(Position(o: 2, l: 1, c: 3), Position(o: 4, l: 1, c: 5)) }))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```

and I think it should be fixed by using the pattern `".*%.*"` instead.

Fix patch:
diff --git a/src/query/regex_query.rs b/src/query/regex_query.rs
index 50527980a8..815832d31e 100644
--- a/src/query/regex_query.rs
+++ b/src/query/regex_query.rs
@@ -63,7 +63,7 @@ impl RegexQuery {
     /// Creates a new RegexQuery from a given pattern
     pub fn from_pattern(regex_pattern: &str, field: Field) -> crate::Result<Self> {
         let regex = Regex::new(regex_pattern)
-            .map_err(|_| TantivyError::InvalidArgument(regex_pattern.to_string()))?;
+            .map_err(|err| TantivyError::InvalidArgument(format!("RegexQueryError: {err}")))?;
         Ok(RegexQuery::from_regex(regex, field))
     }
 
@@ -176,4 +176,16 @@ mod test {
         verify_regex_query(matching_one, matching_zero, reader);
         Ok(())
     }
+
+    #[test]
+    pub fn test_pattern_error() {
+        let (_reader, field) = build_test_index().unwrap();
+
+        match RegexQuery::from_pattern(r"(foo", field) {
+            Err(crate::TantivyError::InvalidArgument(msg)) => {
+                assert!(msg.contains("error: unclosed group"))
+            }
+            res => panic!("unexpected result: {:?}", res),
+        }
+    }
 }


Test patch: