Repository: pypi/warehouse

Issue:
Search method in XML-RPC API treats {"summary": "fix code"} as {"summary": ["fix", "code"]}
As stated in the title, the `search` method of [PyPI/Warehouse's XML-RPC API](https://warehouse.readthedocs.io/api-reference/xml-rpc/) splits the values of its `spec` argument on whitespace, resulting in the method returning (for the example in the title) packages with either "fix" or "code" in their summary, not just those with both "fix" and "code" (let alone just those that contain the exact string "fix code").  This happens even if the `spec` value is passed as a one-element list (e.g., `{"summary": ["fix code"]}`), and while I haven't tested every field, it is not limited to just "summary".

On the off chance that this is the intended behavior, the documentation should at least be updated.


Discussion:
We discussed this in [today's Warehouse bug triage meeting](https://wiki.python.org/psf/PackagingWG/2018-01-29-Warehouse) and decided this is probably a small fix, we need to compare it to behavior on legacy PyPI, and we'll address it in the End User MVP milestone (probably in March). Thanks, @jwodder!
@waseem18 I saw you're working on this? Thanks! Hope you'll join us [in Freenode IRC in `#pypa-dev`](https://webchat.freenode.net/?channels=%23pypa-dev) if you have any questions.
@brainwane Yeah I've started working on this an will put a WIP PR soon.
**Cause of this bug** : 

In `xmlrpc.py` line number [86](https://github.com/pypa/warehouse/blob/747b054c798b25dd43d86a75a5120a3245e153ac/warehouse/legacy/api/xmlrpc.py#L86) and 88 we're using `match query` to query elasticsearch for the given value.

In `match query` what happens is when the value of the given field has space(s) it tokenizes the given value thereby searching for each of the split tokens.


`"query": {
        "match" : {
            "summary" : "fix code"
        }
    }`

Above is a `match query` with space in the value of the field `summary`. So it tokenizes the value to `["fix", "code"]` and searches for `fix` or `code`.

**Resolution** : 

Use [Term query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#query-dsl-term-query) when the value of the given field has space(s) and `match query` when the value doesn't have any space(s). 

#2850 is similar to this issue. Using Term query / `match_phrase` when the entered value has double quotes.

@di @brainwane  Will be happy to get your feedback on this.


@waseem18  Using the `term` query makes sense, especially considering the API allows for multiple strings to be provided.

Any reason why we wouldn't just use it for queries without whitespace as well? Seems like it wouldn't make much difference if the user is just searching for a single word.

For #2850, it seems like we might actually want to do [Phrase Matching](https://www.elastic.co/guide/en/elasticsearch/guide/current/phrase-matching.html) instead, but let's just focus on this issue for now.
Good catch @di 

Using `term` query for values without space wouldn't make much difference. I'll go with it :+1: 

Resolving patch:
diff --git a/warehouse/legacy/api/xmlrpc.py b/warehouse/legacy/api/xmlrpc.py
--- a/warehouse/legacy/api/xmlrpc.py
+++ b/warehouse/legacy/api/xmlrpc.py
@@ -83,9 +83,9 @@ def search(request, spec, operator="and"):
         q = None
         for item in value:
             if q is None:
-                q = Q("match", **{field: item})
+                q = Q("term", **{field: item})
             else:
-                q |= Q("match", **{field: item})
+                q |= Q("term", **{field: item})
         queries.append(q)
 
     if operator == "and":


Test patch:
diff --git a/tests/unit/legacy/api/test_xmlrpc.py b/tests/unit/legacy/api/test_xmlrpc.py
--- a/tests/unit/legacy/api/test_xmlrpc.py
+++ b/tests/unit/legacy/api/test_xmlrpc.py
@@ -57,12 +57,12 @@ def __getitem__(self, name):
             def execute(self):
                 assert self.type == "bool"
                 assert [q.to_dict() for q in self.must] == [
-                    {"match": {"name": "foo"}},
+                    {"term": {"name": "foo"}},
                     {
                         "bool": {
                             "should": [
-                                {"match": {"summary": "one"}},
-                                {"match": {"summary": "two"}},
+                                {"term": {"summary": "one"}},
+                                {"term": {"summary": "two"}},
                             ],
                         },
                     },
@@ -94,6 +94,53 @@ def execute(self):
             {"name": "foo-bar", "summary": "other summary", "version": "1.0"},
         ]
 
+    def test_default_search_operator_with_spaces_in_values(self):
+        class FakeQuery:
+            def __init__(self, type, must):
+                self.type = type
+                self.must = must
+
+            def __getitem__(self, name):
+                self.offset = name.start
+                self.limit = name.stop
+                self.step = name.step
+                return self
+
+            def execute(self):
+                assert self.type == "bool"
+                assert [q.to_dict() for q in self.must] == [
+                    {'bool': {'should': [
+                        {'term': {'summary': 'fix code'}},
+                        {'term': {'summary': 'like this'}}
+                    ]}}
+                ]
+                assert self.offset is None
+                assert self.limit == 1000
+                assert self.step is None
+                return [
+                    pretend.stub(
+                        name="foo",
+                        summary="fix code",
+                        version=["1.0"],
+                    ),
+                    pretend.stub(
+                        name="foo-bar",
+                        summary="like this",
+                        version=["2.0", "1.0"],
+                    ),
+                ]
+
+        request = pretend.stub(es=pretend.stub(query=FakeQuery))
+        results = xmlrpc.search(
+            request,
+            {"summary": ["fix code", "like this"]},
+        )
+        assert results == [
+            {"name": "foo", "summary": "fix code", "version": "1.0"},
+            {"name": "foo-bar", "summary": "like this", "version": "2.0"},
+            {"name": "foo-bar", "summary": "like this", "version": "1.0"},
+        ]
+
     def test_searches_with_and(self):
         class FakeQuery:
             def __init__(self, type, must):
@@ -109,12 +156,12 @@ def __getitem__(self, name):
             def execute(self):
                 assert self.type == "bool"
                 assert [q.to_dict() for q in self.must] == [
-                    {"match": {"name": "foo"}},
+                    {"term": {"name": "foo"}},
                     {
                         "bool": {
                             "should": [
-                                {"match": {"summary": "one"}},
-                                {"match": {"summary": "two"}},
+                                {"term": {"summary": "one"}},
+                                {"term": {"summary": "two"}},
                             ],
                         },
                     },
@@ -162,12 +209,12 @@ def __getitem__(self, name):
             def execute(self):
                 assert self.type == "bool"
                 assert [q.to_dict() for q in self.should] == [
-                    {"match": {"name": "foo"}},
+                    {"term": {"name": "foo"}},
                     {
                         "bool": {
                             "should": [
-                                {"match": {"summary": "one"}},
-                                {"match": {"summary": "two"}},
+                                {"term": {"summary": "one"}},
+                                {"term": {"summary": "two"}},
                             ],
                         },
                     },