Repository: getredash/redash

Issue:
Can't search query correctly with non-ASCII chars
### Issue Summary

Can't search query correctly with non-ASCII chars.

### Steps to Reproduce

1. Make query which has non-ASCII chars name or description
2. Search with non-ASCII chars

e.g.

There is a query which has non-ASCII chars `ユーザ`.

<img width="380" alt="all_queries" src="https://user-images.githubusercontent.com/3317191/41804093-925f57a8-76cb-11e8-9a9e-4e3bb068c306.png">

Search with `ユーザ`, then no quries in the result.

<img width="366" alt="search_query1" src="https://user-images.githubusercontent.com/3317191/41804094-92a2f710-76cb-11e8-9f63-c64a9564613f.png">

When I search with `ユー`, then hit correctly.

<img width="315" alt="search_query2" src="https://user-images.githubusercontent.com/3317191/41804095-92d418f4-76cb-11e8-9481-c6e2ff9af210.png">

I guess that `Query.search`(in #2041) has changed this behavior. But I have no idea what we should fix it with keeping full text search feature.

### Technical details:

* Redash Version: master
* Browser/OS: Version 67.0.3396.87 (Official Build) (64-bit)
* How did you install Redash: Docker



Discussion:
Possibly related to #2618.
Yeah, the recently udpated query full text search is based on Postgres' built-in [textsearch](https://www.postgresql.org/docs/9.5/static/textsearch-intro.html) extension which will use the "simple" configuration (parsers, templates, dictionaries) which only applies lower case and removes stop words from the content body while searching.

Unfortunately by default it only comes with support for [a few Indo-European languages](https://www.compose.com/articles/mastering-postgresql-tools-full-text-search-and-phrase-search/#languages) and misses others such as Korean, Japanese and Chinese (and more).

To add support for this, we'd need to add additional support for those languages, for example [PGroonga](https://pgroonga.github.io/), which supports all languages, but requires a 3rd party extension. The [tutorial](https://pgroonga.github.io/tutorial/) gives an idea how this would like, including for example the ability to just keep using `ILIKE` queries.

Alternatively we could move away the FTS from using Postgres altogether and switch to one of the many alternative search engines such as Elasticsearch, but that comes with a non-trivial amount of architectural changes.
> Alternatively we could move away the FTS from using Postgres altogether and switch to one of the many alternative search engines such as Elasticsearch, but that comes with a non-trivial amount of architectural changes.

I wouldn't want to have ES as a mandatory dependency in Redash as it will make deployments harder. But maybe we can make this functionality pluggable: 

1. Have a hook for "index new content" (dashboard / query / other in the future) and "index updated content".
2. Have an interface for performing a search.

By default the two will use Postgres, but will have additional implementation using ES, Algolia, other.
It would complicate the list views a bit since it's written right now to not differ between searching and just fetching the list of all items. I guess the API handlers can provide the interface to cater to that and ask the search backend to provide a list of item model IDs in the order of the search ranking and then fetch the appropriate date model items from the data base.

While I don't think it will be a huge deal, there is some overhead involved that we should probably be testing. E.g. support for pagination in the search backend would seem like a good idea.
BTW, would you consider making this something to be distributed in the Redash core, or as extensions?
The list view is the least of the complications this will create :) I'm more worried about permissions and similar concerns, data sync (between search engine and database) and other.

And yes, this can be an extension. 
Hi @jezdez,

Could we bring back the naive and slow `LIKE` search as an option? Maybe an ENV variable `LEGACY_FULL_TEXT_SEARCH` or something to switch between two ways of searching?

For me, being able to search in multi-byte is far more critical than having faster and more modern tsvector textsearch.
@deecay adding an option to enable simpler search sounds good to me. Considering the global usage of Redash, I expect this to be popular enough to put in `Organization Settings` UI.

Resolving patch:
diff --git a/redash/handlers/queries.py b/redash/handlers/queries.py
--- a/redash/handlers/queries.py
+++ b/redash/handlers/queries.py
@@ -112,6 +112,7 @@ def get_queries(self, search_term):
                 self.current_user.group_ids,
                 self.current_user.id,
                 include_drafts=True,
+                multi_byte_search=current_org.get_setting('multi_byte_search_enabled'),
             )
         else:
             results = models.Query.all_queries(
@@ -256,6 +257,7 @@ def get_queries(self, search_term):
                 self.current_user.id,
                 include_drafts=False,
                 include_archived=True,
+                multi_byte_search=current_org.get_setting('multi_byte_search_enabled'),
             )
         else:
             return models.Query.all_queries(
diff --git a/redash/models/__init__.py b/redash/models/__init__.py
--- a/redash/models/__init__.py
+++ b/redash/models/__init__.py
@@ -578,13 +578,24 @@ def outdated_queries(cls):
 
     @classmethod
     def search(cls, term, group_ids, user_id=None, include_drafts=False,
-               limit=None, include_archived=False):
+               limit=None, include_archived=False, multi_byte_search=False):
         all_queries = cls.all_queries(
             group_ids,
             user_id=user_id,
             include_drafts=include_drafts,
             include_archived=include_archived,
         )
+
+        if multi_byte_search:
+            # Since tsvector doesn't work well with CJK languages, use `ilike` too
+            pattern = u'%{}%'.format(term)
+            return all_queries.filter(
+                or_(
+                    cls.name.ilike(pattern),
+                    cls.description.ilike(pattern)
+                )
+            ).order_by(Query.id).limit(limit)
+
         # sort the result using the weight as defined in the search vector column
         return all_queries.search(term, sort=True).limit(limit)
 
diff --git a/redash/settings/organization.py b/redash/settings/organization.py
--- a/redash/settings/organization.py
+++ b/redash/settings/organization.py
@@ -20,6 +20,7 @@
 TIME_FORMAT = os.environ.get("REDASH_TIME_FORMAT", "HH:mm")
 INTEGER_FORMAT = os.environ.get("REDASH_INTEGER_FORMAT", "0,0")
 FLOAT_FORMAT = os.environ.get("REDASH_FLOAT_FORMAT", "0,0.00")
+MULTI_BYTE_SEARCH_ENABLED = parse_boolean(os.environ.get("MULTI_BYTE_SEARCH_ENABLED", "false"))
 
 JWT_LOGIN_ENABLED = parse_boolean(os.environ.get("REDASH_JWT_LOGIN_ENABLED", "false"))
 JWT_AUTH_ISSUER = os.environ.get("REDASH_JWT_AUTH_ISSUER", "")
@@ -41,6 +42,7 @@
     "time_format": TIME_FORMAT,
     "integer_format": INTEGER_FORMAT,
     "float_format": FLOAT_FORMAT,
+    "multi_byte_search_enabled": MULTI_BYTE_SEARCH_ENABLED,
     "auth_jwt_login_enabled": JWT_LOGIN_ENABLED,
     "auth_jwt_auth_issuer": JWT_AUTH_ISSUER,
     "auth_jwt_auth_public_certs_url": JWT_AUTH_PUBLIC_CERTS_URL,


Test patch:
diff --git a/tests/models/test_queries.py b/tests/models/test_queries.py
--- a/tests/models/test_queries.py
+++ b/tests/models/test_queries.py
@@ -52,6 +52,17 @@ def test_search_finds_in_description(self):
         self.assertIn(q2, queries)
         self.assertNotIn(q3, queries)
 
+    def test_search_finds_in_multi_byte_name_and_description(self):
+        q1 = self.factory.create_query(name="日本語の名前テスト")
+        q2 = self.factory.create_query(description=u"日本語の説明文テスト")
+        q3 = self.factory.create_query(description=u"Testing search")
+
+        queries = Query.search(u"テスト", [self.factory.default_group.id], multi_byte_search=True)
+
+        self.assertIn(q1, queries)
+        self.assertIn(q2, queries)
+        self.assertNotIn(q3, queries)
+
     def test_search_by_id_returns_query(self):
         q1 = self.factory.create_query(description=u"Testing search")
         q2 = self.factory.create_query(description=u"Testing searching")