Repository: wagtail/wagtail

Issue:
Hyphens in search query are normalized differently than in ElasticSearch
If I have the substring "fooo-baar" in the text of one of the indexed fields for a Page-derived model instance, I'd expect to be see that page when I search for "fooo-baar", but I don't.

This seems to be because `wagtailsearch` normalizes "fooo-baar" to "fooobaar", while ElasticSearch treats the hyphen as a whitespace character.

Failing test: add 

```
"Hello-world",
```

to `test_queries:45`.

Suggested fix: normalize to "fooo baar" instead.



Discussion:
Generally, it seems like `normalise_query_string` should probably be treating punctuation characters as whitespace rather than stripping them.

Yep, I agree. Thanks for spotting!

The punctuation stripping was actually implemented to work around another bug\* in the Elasticsearch backend which is now fixed. It may be safe now to just send the unnormalised query string to Elasticsearch and let it decide what do do with the punctuation.

*We were previously performing a `query` query instead of a `match` query in Elasticsearch which took punctuation to mean commands. This caused a few headaches/crashes.


Resolving patch:
diff --git a/wagtail/wagtailsearch/backends/base.py b/wagtail/wagtailsearch/backends/base.py
--- a/wagtail/wagtailsearch/backends/base.py
+++ b/wagtail/wagtailsearch/backends/base.py
@@ -7,7 +7,6 @@
 from django.core.exceptions import ImproperlyConfigured
 
 from wagtail.wagtailsearch.index import class_is_indexed
-from wagtail.wagtailsearch.utils import normalise_query_string
 
 
 class FilterError(Exception):
@@ -213,10 +212,6 @@ def search(self, query_string, model_or_queryset, fields=None, filters=None, pre
         if not class_is_indexed(model):
             return []
 
-        # Normalise query string
-        if query_string is not None:
-            query_string = normalise_query_string(query_string)
-
         # Check that theres still a query string after the clean up
         if query_string == "":
             return []


Test patch:
diff --git a/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py b/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py
--- a/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py
+++ b/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py
@@ -135,6 +135,34 @@ def test_query_analyser(self):
         # Even though they both start with the letter "H". This should not be considered a match
         self.assertEqual(len(results), 0)
 
+    def test_search_with_hyphen(self):
+        """
+        This tests that punctuation characters are treated the same
+        way in both indexing and querying.
+
+        See: https://github.com/torchbox/wagtail/issues/937
+        """
+        # Reset the index
+        self.backend.reset_index()
+        self.backend.add_type(models.SearchTest)
+        self.backend.add_type(models.SearchTestChild)
+
+        # Add some test data
+        obj = models.SearchTest()
+        obj.title = "Hello-World"
+        obj.live = True
+        obj.save()
+        self.backend.add(obj)
+
+        # Refresh the index
+        self.backend.refresh_index()
+
+        # Test search for "Hello-World"
+        results = self.backend.search("Hello-World", models.SearchTest.objects.all())
+
+        # Should find the result
+        self.assertEqual(len(results), 1)
+
 
 class TestElasticSearchQuery(TestCase):
     def assertDictEqual(self, a, b):