Repository: pypi/warehouse

Issue:
Support searching for an exact phrase
The search box does not appear to have a way to search for a specific exact phrase.

Based on how other search engines work, putting a phrase in quotes should require an exact phrase match.

However, currently searches with and without quotes produce the same result:

https://pypi.org/search/?q=%22Image+processing+routines+for+SciPy%22
and 
https://pypi.org/search/?q=Image+processing+routines+for+SciPy

I would have expected the first one with quotes to only produce the result containing the exact phrase.

---

**Good First Issue**: This issue is good for first time contributors. If there is not a corresponding pull request for this issue, it is up for grabs. For directions for getting set up, see our [Getting Started Guide](https://warehouse.pypa.io/development/getting-started/). If you are working on this issue and have questions, please feel free to ask them here, [`#pypa-dev` on Freenode](https://webchat.freenode.net/?channels=%23pypa-dev), or the [pypa-dev mailing list](https://groups.google.com/forum/#!forum/pypa-dev).


Discussion:
Thanks for your report, @pv, and sorry for the slow response!

The folks working on Warehouse have gotten [funding to concentrate on improving and deploying Warehouse](https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html), and have kicked off work towards [our development roadmap](https://wiki.python.org/psf/WarehouseRoadmap) -- the most urgent task is to improve Warehouse to the point where we can redirect pypi.python.org to pypi.org so the site is more sustainable and reliable.

We discussed this issue [in our meeting today](https://wiki.python.org/psf/PackagingWG/2018-02-12-Warehouse) to prioritize it. Since search in Warehouse is already much better than search on legacy PyPI, but users will probably expect search to work as you suggest, I've moved this issue to a future milestone that we'll work on in the next few months.

Thanks, @pv, and sorry again for the wait.

Note to people thinking about contributing to Warehouse: this would be a great first issue for a new contributor to tackle if the new contributor were already familiar with Elasticsearch.
I'm looking into this issue.

This doesn't look like a simple `match` to `phrase_match` change.
@waseem18 Yeah, I'm not _totally_ sure that this issue is actually a "good first issue" -- elasticsearch tuning has generally been pretty tricky in my experience. But since you're not a new contributor anymore, should be a good issue for you. 🙂 
Update:

After changing `match` to `match_phrase` at [this line](https://github.com/pypa/warehouse/blob/598963368bb71ea2310a6b0220327f7d90e63591/warehouse/views.py#L260) and searching Warehouse with string containing spaces (example: cli github), we get `Transport Error`.

`
elasticsearch.exceptions.TransportError: TransportError(500, 'search_phase_execution_exception', 'field "normalized_name" was indexed without position data; cannot run PhraseQuery (phrase=normalized_name:"cli github")')
`

As phrase queries require `index_options: positions`- I changed 
`normalized_name = Text(analyzer=NameAnalyzer, index_options="docs")` to `normalized_name = Text(analyzer=NameAnalyzer, index_options="positions")` [here](https://github.com/pypa/warehouse/blob/598963368bb71ea2310a6b0220327f7d90e63591/warehouse/packaging/search.py#L37) and then reindexed.

At this point the search functionality works but I found the results are not efficient.
@waseem18 what do you mean "not efficient"? I would be happy to help fine-tune the queries
@waseem18 Maybe you have done some profiling and you have some specific performance numbers to share?
@HonzaKral By not efficient I mean the search results are way better when `index_options=docs`.

For example when we search `image processing routines`, `index_options=docs` gives good results while `index_options=positions` doesn't give any result.

This is same with queries containing spaces.
@waseem18 it's going to be helpful if you give concrete numbers or results, I think. Saying "good results" might mean that there are a lot of them, or they're relevant, or they're well ordered... feel free to cut and paste or take screenshots to show what you mean. :)
Sure @brainwane 
In this particular case there should be no difference between which documents match with various `index_options` as long as the query remains the same. Only ordering (`_score`) of hits would be different where `docs` doesn't take into account frequency (how many times the word occurs in the document). `positions` is the default and should be used here
Thanks for the information @HonzaKral I'll get back with some examples and screenshots.
@HonzaKral Thanks for the help. 

- Tweaked `index_options`  to `positions` for Package index and respective `match_phrase` change. 
- Changed `SEARCH_BOOST` value of `description` from 5 to 10 which I believe improved phrase queries.
- Attached some screenshots which show how relevant the results are without and with double quotes respectively. 

1.
![1](https://user-images.githubusercontent.com/5843007/36833208-a376ffa8-1d54-11e8-88da-e7e22aa40cc0.png)
![1](https://user-images.githubusercontent.com/5843007/36833213-a70c689c-1d54-11e8-96e8-dfae5f7ec37c.png)

2.
![2](https://user-images.githubusercontent.com/5843007/36833239-c9c8db2c-1d54-11e8-8cde-21c7f806692d.png)
![2](https://user-images.githubusercontent.com/5843007/36833245-ccf86588-1d54-11e8-920e-176fea86abcd.png)

3.
![3](https://user-images.githubusercontent.com/5843007/36833425-4ec42c0a-1d55-11e8-9afc-92832df7989f.png)
![3](https://user-images.githubusercontent.com/5843007/36833428-52b524c2-1d55-11e8-90d3-e4a32e0078a8.png)


Note that users expect that it's also possible to do queries containing 
both exact and inexact matches (probably should be ANDed), along the 
lines of:

"image processing" geo

good use case @pv 

Will have that in mind

Resolving patch:
diff --git a/warehouse/packaging/search.py b/warehouse/packaging/search.py
--- a/warehouse/packaging/search.py
+++ b/warehouse/packaging/search.py
@@ -34,7 +34,7 @@
 class Project(DocType):
 
     name = Text()
-    normalized_name = Text(analyzer=NameAnalyzer, index_options="docs")
+    normalized_name = Text(analyzer=NameAnalyzer)
     version = Keyword(multi=True)
     latest_version = Keyword()
     summary = Text(analyzer="snowball")
diff --git a/warehouse/views.py b/warehouse/views.py
--- a/warehouse/views.py
+++ b/warehouse/views.py
@@ -11,6 +11,7 @@
 # limitations under the License.
 
 import collections
+import re
 
 from pyramid.httpexceptions import (
     HTTPException, HTTPSeeOther, HTTPMovedPermanently, HTTPNotFound,
@@ -248,20 +249,13 @@ def classifiers(request):
 def search(request):
 
     q = request.params.get("q", '')
+    q = q.replace("'", '"')
 
     if q:
-        should = []
-        for field in SEARCH_FIELDS:
-            kw = {"query": q}
-            if field in SEARCH_BOOSTS:
-                kw["boost"] = SEARCH_BOOSTS[field]
-            should.append(Q("match", **{field: kw}))
-
-        # Add a prefix query if ``q`` is longer than one character.
-        if len(q) > 1:
-            should.append(Q('prefix', normalized_name=q))
-
-        query = request.es.query("dis_max", queries=should)
+        bool_query = gather_es_queries(q)
+
+        query = request.es.query(bool_query)
+
         query = query.suggest("name_suggestion", q, term={"field": "name"})
     else:
         query = request.es.query()
@@ -372,3 +366,45 @@ def force_status(request):
         raise exception_response(int(request.matchdict["status"]))
     except KeyError:
         raise exception_response(404) from None
+
+
+def filter_query(s):
+    """
+    Filters given query with the below regex
+    and returns lists of quoted and unquoted strings
+    """
+    matches = re.findall(r'(?:"([^"]*)")|([^"]*)', s)
+    result_quoted = [t[0].strip() for t in matches if t[0]]
+    result_unquoted = [t[1].strip() for t in matches if t[1]]
+    return result_quoted, result_unquoted
+
+
+def form_query(query_type, query):
+    """
+    Returns a multi match query
+    """
+    fields = [
+        field + "^" + str(SEARCH_BOOSTS[field])
+        if field in SEARCH_BOOSTS else field
+        for field in SEARCH_FIELDS
+    ]
+    return Q('multi_match', fields=fields,
+             query=query, type=query_type
+             )
+
+
+def gather_es_queries(q):
+    quoted_string, unquoted_string = filter_query(q)
+    must = [
+        form_query("phrase", i) for i in quoted_string
+    ] + [
+        form_query("best_fields", i) for i in unquoted_string
+    ]
+
+    bool_query = Q('bool', must=must)
+
+    # Allow to optionally match on prefix
+    # if ``q`` is longer than one character.
+    if len(q) > 1:
+        bool_query = bool_query | Q("prefix", normalized_name=q)
+    return bool_query


Test patch:
diff --git a/tests/unit/test_views.py b/tests/unit/test_views.py
--- a/tests/unit/test_views.py
+++ b/tests/unit/test_views.py
@@ -14,7 +14,6 @@
 
 import pretend
 import pytest
-from elasticsearch_dsl import Q
 from webob.multidict import MultiDict
 
 from pyramid.httpexceptions import (
@@ -23,9 +22,9 @@
 
 from warehouse import views
 from warehouse.views import (
-    SEARCH_BOOSTS, SEARCH_FIELDS, classifiers, current_user_indicator,
-    forbidden, health, httpexception_view, index, robotstxt, opensearchxml,
-    search, force_status, flash_messages, forbidden_include
+    classifiers, current_user_indicator, forbidden, health,
+    httpexception_view, index, robotstxt, opensearchxml, search, force_status,
+    flash_messages, forbidden_include
 )
 
 from ..common.db.accounts import UserFactory
@@ -207,17 +206,6 @@ def test_esi_flash_messages():
 
 class TestSearch:
 
-    def _gather_es_queries(self, q):
-        queries = []
-        for field in SEARCH_FIELDS:
-            kw = {"query": q}
-            if field in SEARCH_BOOSTS:
-                kw["boost"] = SEARCH_BOOSTS[field]
-            queries.append(Q("match", **{field: kw}))
-        if len(q) > 1:
-            queries.append(Q("prefix", normalized_name=q))
-        return queries
-
     @pytest.mark.parametrize("page", [None, 1, 5])
     def test_with_a_query(self, monkeypatch, db_request, page):
         params = MultiDict({"q": "foo bar"})
@@ -257,8 +245,59 @@ def test_with_a_query(self, monkeypatch, db_request, page):
         assert url_maker_factory.calls == [pretend.call(db_request)]
         assert db_request.es.query.calls == [
             pretend.call(
-                "dis_max",
-                queries=self._gather_es_queries(params["q"])
+                views.gather_es_queries(params["q"])
+            )
+        ]
+        assert es_query.suggest.calls == [
+            pretend.call(
+                "name_suggestion",
+                params["q"],
+                term={"field": "name"},
+            ),
+        ]
+
+    @pytest.mark.parametrize("page", [None, 1, 5])
+    def test_with_exact_phrase_query(self, monkeypatch, db_request, page):
+        params = MultiDict({"q": '"foo bar"'})
+        if page is not None:
+            params["page"] = page
+        db_request.params = params
+
+        sort = pretend.stub()
+        suggest = pretend.stub(
+            sort=pretend.call_recorder(lambda *a, **kw: sort),
+        )
+        es_query = pretend.stub(
+            suggest=pretend.call_recorder(lambda *a, **kw: suggest),
+        )
+        db_request.es = pretend.stub(
+            query=pretend.call_recorder(lambda *a, **kw: es_query)
+        )
+
+        page_obj = pretend.stub(page_count=(page or 1) + 10,
+                                item_count=(page or 1) + 10
+                                )
+        page_cls = pretend.call_recorder(lambda *a, **kw: page_obj)
+        monkeypatch.setattr(views, "ElasticsearchPage", page_cls)
+
+        url_maker = pretend.stub()
+        url_maker_factory = pretend.call_recorder(lambda request: url_maker)
+        monkeypatch.setattr(views, "paginate_url_factory", url_maker_factory)
+
+        assert search(db_request) == {
+            "page": page_obj,
+            "term": params.get("q", ''),
+            "order": params.get("o", ''),
+            "applied_filters": [],
+            "available_filters": [],
+        }
+        assert page_cls.calls == [
+            pretend.call(suggest, url_maker=url_maker, page=page or 1),
+        ]
+        assert url_maker_factory.calls == [pretend.call(db_request)]
+        assert db_request.es.query.calls == [
+            pretend.call(
+                views.gather_es_queries(params["q"])
             )
         ]
         assert es_query.suggest.calls == [
@@ -308,8 +347,7 @@ def test_with_a_single_char_query(self, monkeypatch, db_request, page):
         assert url_maker_factory.calls == [pretend.call(db_request)]
         assert db_request.es.query.calls == [
             pretend.call(
-                "dis_max",
-                queries=self._gather_es_queries(params["q"])
+                views.gather_es_queries(params["q"])
             )
         ]
         assert es_query.suggest.calls == [
@@ -380,8 +418,7 @@ def test_with_an_ordering(self, monkeypatch, db_request, page, order,
         assert url_maker_factory.calls == [pretend.call(db_request)]
         assert db_request.es.query.calls == [
             pretend.call(
-                "dis_max",
-                queries=self._gather_es_queries(params["q"])
+                views.gather_es_queries(params["q"])
             )
         ]
         assert es_query.suggest.calls == [
@@ -455,8 +492,7 @@ def test_with_classifiers(self, monkeypatch, db_request, page):
         assert url_maker_factory.calls == [pretend.call(db_request)]
         assert db_request.es.query.calls == [
             pretend.call(
-                "dis_max",
-                queries=self._gather_es_queries(params["q"])
+                views.gather_es_queries(params["q"])
             )
         ]
         assert es_query.suggest.calls == [