Project: talenddataquality
Issue TDQ-4989: Add an option to escape special characters and key words for the queries in SynonymIndexSearcher
lucene query parser accepts a set of complex syntax such as * for wildcard match. if this kind of characters exist in the original string to search, the result will be influenced. here is a list of special characters from http://lucene.apache.org/core/old_versioned_docs/versions/3_0_1/queryparsersyntax.html + - && || ! ( ) { } [ ] ^ " ~ * ? : \ this problem is mainly for tSynonymSearch, because we always search the entire string Ex: for lucene <9~17> means search <9> with 17.0 as the minimum similarity (which sould be between 0 and 1) lucene will throws an exception saying: {code} Cannot parse '9~17': Minimum similarity for a FuzzyQuery has to be between 0.0f and 1.0f ! {code} What we need to do here is sending <9\ ~17> to query parser instead of the original string <9~17>. With tStandardizeRow, we parse the string at first. most of the characters above as well as the strings containg them will be marked as UNDEFINED, which we will not search any more in the index. However, we still have a problem with some keywords: In tStandardizeRow, we incrementally match the words. That means if "TOM" matches, we continue to search "TOM OR", which will unfortunately violate the syntax of lucene query parser, because there is no right hand side of the keyword OR. {code} org.apache.lucene.queryParser.ParseException: Cannot parse 'TOM OR': Encountered "" at line 1, column 7. Was expecting one of: ... "+" ... "-" ... "(" ... "*" ... ... ... ... ... "[" ... "{" ... ... ... "*" ... {code} what we need to do here is to replace all the < OR> with < "OR"> to tell lucene that the here is just another word to search. As the replacement action will consume some extra resources, we should add it as an option but not the default behavior.
Story points: 8