Repository: wagtail/wagtail

Issue:
WagtailSearch ElasticSearch backend setting to set results size
### Issue Summary

There is no way to specify results count (`'size'`), returned by `elasticsearch2` backend which results into always getting the defaults (10) results.

### Steps to Reproduce

1. Set up WagtailSearch with `elasticsearch2` backend.
2. Run `.search('QUERY').results()` always returns 10 results

There is nothing in documentation about specifying the results count. Should be a django setting?
Here is what I am doing:

```
In [7]: query = 'shoes'

In [8]: TenantPage.objects.count()
Out[8]: 88

In [9]: TenantPage.objects.search(query).count()
Out[9]: 80

In [10]: len(TenantPage.objects.search(query).results())
Out[10]: 10
```

I also tried to add 'size' to `OPTIONS` in Django settings as below:

```
WAGTAILSEARCH_BACKENDS = {
    'default': {
        'BACKEND': 'wagtail.wagtailsearch.backends.elasticsearch2',
        'INDEX': 'wagtail',
        'INDEX_SETTINGS': {},
        'OPTIONS': {'from': 0, 'size': 100},  # has no effect
        'TIMEOUT': 5,
        'URLS': ['http://localhost:9200']
    }
}
```

If you trace the `Elasticsearch2SearchBackend`s result class (`Elasticsearch2SearchResults`) to it's parent, you end up [here](https://github.com/wagtail/wagtail/blob/master/wagtail/wagtailsearch/backends/elasticsearch.py#L418):

```
class ElasticsearchSearchResults(BaseSearchResults):
    fields_param_name = 'fields'
    
    ...

    def _do_search(self):
        # Params for elasticsearch query
        params = dict(
            index=self.backend.get_index_for_model(self.query.queryset.model).name,
            body=self._get_es_body(),
            _source=False,
            from_=self.start,
        )

        params[self.fields_param_name] = 'pk'

        # Add size if set
        if self.stop is not None:
            params['size'] = self.stop - self.start
    ...
```
If I hardcode `params['size']=100` it works. I see some logic to "Add size if set" but nothing about how to actually set it. Is it documented? But I assume it's being used for pagination and offsetting the results.

### Technical details

* Python version: 2.7.3
* Django version: 1.9.6
* Wagtail version: 1.9


Discussion:
There could potentially be many 1000s of returned results and ``len(TenantPage.objects.search(query))`` would have to fetch them all in order to behave correctly.

I'm thinking that, if the search limit is unbounded, we should initially try to fetch 1000 results. If there were more, we should perform another search to get the next 1000, and so on until we've got all the results.

Obviously, if this does actually happen in the wild, something's probably wrong with the code and the memory usage is probably going to blow up. But that would be the expected behaviour...
Just spoke to @gasman about this and he reminded me that you can just do ``len(TenantPage.objects.search(query)[:100])`` and this would return all of the results.

(I know, it's weird that slicing something makes it bigger. We should fix that)
@kaedroho I agree, but that should probably be left to developers as a judgement call. The bottom line is that there is a default value for size of 10 and there should be an option to override it.

py-elasticsearch supports this, you can see `.search()` accepts many params including `size`, here is the [link](https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.Elasticsearch.search).

Currently I think it is odd that I cannot see more than 10 results. Here is the live example of a project I'm working on: [Americana Manhasset](https://www.americanamanhasset.com/search?q=shoes)
Whoa, thanks @kaedroho @gasman. That solves my issue for now. Can you document this solution?

Do you think the more explicit setting (say on the `WAGTAILSEARCH_BACKENDS` level) is something you would implement? I think slicing can get expensive depending on the amount of results etc.
Slicing is implemented internally by passing the relevant `from` / `size` parameters to Elasticsearch (just as Django querysets implement slicing by adding a LIMIT clause to the SQL), so there shouldn't be any performance implication there.

I think it would make sense for our ES backend to set a default `size` of e.g. 100 when no slice has been given - rather than ES's own default of 10 - so that people are less likely to run into this "taking a slice gives you more results" weirdness. Perhaps we should also have it throw a warning when we run a query with no explicit range specified, similarly to how Django now throws a warning when paginating an unordered queryset?
@gasman thanks for clarification. I think that's a good approach guys but of course up to you. Do you mind adding this to documentation though? That would be helpful.
Aside: the `[:100]` workaround strips the annotation added by `.annotate_score()`, as does retrieving a result by its index.

    >>> results = MyObject.search('foo').annotate_score('_score')
    >>> results[0]._score
    AttributeError: 'MyObject' object has no attribute '_score'
    >>> [x._score for x in results[:100]]
    AttributeError: 'MyObject' object has no attribute '_score'
    >>> [x._score for x in results]
    [0.38437524, 0.24610822, 0.24095446, 0.23447372, 0.22940964, 0.21996033, 0.21083516, 0.1894834, 0.18731217, 0.1755507]

Resolving patch:
diff --git a/wagtail/wagtailsearch/backends/base.py b/wagtail/wagtailsearch/backends/base.py
--- a/wagtail/wagtailsearch/backends/base.py
+++ b/wagtail/wagtailsearch/backends/base.py
@@ -135,7 +135,7 @@ def _do_count(self):
 
     def results(self):
         if self._results_cache is None:
-            self._results_cache = self._do_search()
+            self._results_cache = list(self._do_search())
         return self._results_cache
 
     def count(self):
diff --git a/wagtail/wagtailsearch/backends/db.py b/wagtail/wagtailsearch/backends/db.py
--- a/wagtail/wagtailsearch/backends/db.py
+++ b/wagtail/wagtailsearch/backends/db.py
@@ -79,7 +79,7 @@ def _do_search(self):
         if self._score_field:
             queryset = queryset.annotate(**{self._score_field: Value(None, output_field=models.FloatField())})
 
-        return queryset
+        return queryset.iterator()
 
     def _do_count(self):
         return self.get_queryset().count()
diff --git a/wagtail/wagtailsearch/backends/elasticsearch.py b/wagtail/wagtailsearch/backends/elasticsearch.py
--- a/wagtail/wagtailsearch/backends/elasticsearch.py
+++ b/wagtail/wagtailsearch/backends/elasticsearch.py
@@ -439,41 +439,106 @@ def _get_es_body(self, for_count=False):
 
         return body
 
-    def _do_search(self):
-        # Params for elasticsearch query
-        params = dict(
-            index=self.backend.get_index_for_model(self.query.queryset.model).name,
-            body=self._get_es_body(),
-            _source=False,
-            from_=self.start,
-        )
-
-        params[self.fields_param_name] = 'pk'
-
-        # Add size if set
-        if self.stop is not None:
-            params['size'] = self.stop - self.start
-
-        # Send to Elasticsearch
-        hits = self.backend.es.search(**params)
-
+    def _get_results_from_hits(self, hits):
+        """
+        Yields Django model instances from a page of hits returned by Elasticsearch
+        """
         # Get pks from results
-        pks = [hit['fields']['pk'][0] for hit in hits['hits']['hits']]
-        scores = {str(hit['fields']['pk'][0]): hit['_score'] for hit in hits['hits']['hits']}
+        pks = [hit['fields']['pk'][0] for hit in hits]
+        scores = {str(hit['fields']['pk'][0]): hit['_score'] for hit in hits}
 
         # Initialise results dictionary
-        results = dict((str(pk), None) for pk in pks)
+        results = {str(pk): None for pk in pks}
 
         # Find objects in database and add them to dict
-        queryset = self.query.queryset.filter(pk__in=pks)
-        for obj in queryset:
+        for obj in self.query.queryset.filter(pk__in=pks):
             results[str(obj.pk)] = obj
 
             if self._score_field:
                 setattr(obj, self._score_field, scores.get(str(obj.pk)))
 
-        # Return results in order given by Elasticsearch
-        return [results[str(pk)] for pk in pks if results[str(pk)]]
+        # Yield results in order given by Elasticsearch
+        for pk in pks:
+            result = results[str(pk)]
+            if result:
+                yield result
+
+    def _do_search(self):
+        PAGE_SIZE = 100
+
+        if self.stop is not None:
+            limit = self.stop - self.start
+        else:
+            limit = None
+
+        use_scroll = limit is None or limit > PAGE_SIZE
+
+        params = {
+            'index': self.backend.get_index_for_model(self.query.queryset.model).name,
+            'body': self._get_es_body(),
+            '_source': False,
+            self.fields_param_name: 'pk',
+        }
+
+        if use_scroll:
+            params.update({
+                'scroll': '2m',
+                'size': PAGE_SIZE,
+            })
+
+            # The scroll API doesn't support offset, manually skip the first results
+            skip = self.start
+
+            # Send to Elasticsearch
+            page = self.backend.es.search(**params)
+
+            while True:
+                hits = page['hits']['hits']
+
+                if len(hits) == 0:
+                    break
+
+                # Get results
+                if skip < len(hits):
+                    for result in self._get_results_from_hits(hits):
+                        if limit is not None and limit == 0:
+                            break
+
+                        if skip == 0:
+                            yield result
+
+                            if limit is not None:
+                                limit -= 1
+                        else:
+                            skip -= 1
+
+                    if limit is not None and limit == 0:
+                        break
+                else:
+                    # Skip whole page
+                    skip -= len(hits)
+
+                # Fetch next page of results
+                if '_scroll_id' not in page:
+                    break
+
+                page = self.backend.es.scroll(scroll_id=page['_scroll_id'], scroll='2m')
+
+            # Clear the scroll
+            if '_scroll_id' in page:
+                self.backend.es.clear_scroll(scroll_id=page['_scroll_id'])
+        else:
+            params.update({
+                'from_': self.start,
+                'size': limit or PAGE_SIZE,
+            })
+
+            # Send to Elasticsearch
+            hits = self.backend.es.search(**params)['hits']['hits']
+
+            # Get results
+            for result in self._get_results_from_hits(hits):
+                yield result
 
     def _do_count(self):
         # Get count


Test patch:
diff --git a/wagtail/wagtailsearch/tests/elasticsearch_common_tests.py b/wagtail/wagtailsearch/tests/elasticsearch_common_tests.py
--- a/wagtail/wagtailsearch/tests/elasticsearch_common_tests.py
+++ b/wagtail/wagtailsearch/tests/elasticsearch_common_tests.py
@@ -126,3 +126,48 @@ def test_annotate_score_with_slice(self):
 
         for result in results:
             self.assertIsInstance(result._score, float)
+
+    def test_more_than_ten_results(self):
+        # #3431 reported that Elasticsearch only sends back 10 results if the results set is not sliced
+        results = self.backend.search(None, models.Book)
+
+        self.assertEqual(len(results), 13)
+
+    def test_more_than_one_hundred_results(self):
+        # Tests that fetching more than 100 results uses the scroll API
+        books = []
+        for i in range(150):
+            books.append(models.Book.objects.create(title="Book {}".format(i), publication_date=date(2017, 10, 21), number_of_pages=i))
+
+        index = self.backend.get_index_for_model(models.Book)
+        index.add_items(models.Book, books)
+        index.refresh()
+
+        results = self.backend.search(None, models.Book)
+        self.assertEqual(len(results), 163)
+
+    def test_slice_more_than_one_hundred_results(self):
+        books = []
+        for i in range(150):
+            books.append(models.Book.objects.create(title="Book {}".format(i), publication_date=date(2017, 10, 21), number_of_pages=i))
+
+        index = self.backend.get_index_for_model(models.Book)
+        index.add_items(models.Book, books)
+        index.refresh()
+
+        results = self.backend.search(None, models.Book)[10:120]
+        self.assertEqual(len(results), 110)
+
+    def test_slice_to_next_page(self):
+        # ES scroll API doesn't support offset. The implementation has an optimisation
+        # which will skip the first page if the first result is on the second page
+        books = []
+        for i in range(150):
+            books.append(models.Book.objects.create(title="Book {}".format(i), publication_date=date(2017, 10, 21), number_of_pages=i))
+
+        index = self.backend.get_index_for_model(models.Book)
+        index.add_items(models.Book, books)
+        index.refresh()
+
+        results = self.backend.search(None, models.Book)[110:]
+        self.assertEqual(len(results), 53)
diff --git a/wagtail/wagtailsearch/tests/test_elasticsearch2_backend.py b/wagtail/wagtailsearch/tests/test_elasticsearch2_backend.py
--- a/wagtail/wagtailsearch/tests/test_elasticsearch2_backend.py
+++ b/wagtail/wagtailsearch/tests/test_elasticsearch2_backend.py
@@ -21,12 +21,6 @@
 class TestElasticsearch2SearchBackend(BackendTests, ElasticsearchCommonSearchBackendTests, TestCase):
     backend_path = 'wagtail.wagtailsearch.backends.elasticsearch2'
 
-    # TODO: we have to put [:100] on the end of the query due to issue #3431
-    def test_search_all(self):
-        # Searches on None should return everything in the index
-        results = self.backend.search(None, models.Book)[:100]
-        self.assertSetEqual(set(results), set(models.Book.objects.all()))
-
     # Broken
     @unittest.expectedFailure
     def test_filter_in_values_list_subquery(self):
@@ -390,11 +384,12 @@ def test_basic_search(self, search):
         list(results)  # Performs search
 
         search.assert_any_call(
-            from_=0,
             body={'query': 'QUERY'},
             _source=False,
             fields='pk',
-            index='wagtail__searchtests_book'
+            index='wagtail__searchtests_book',
+            scroll='2m',
+            size=100
         )
 
     @mock.patch('elasticsearch.Elasticsearch.search')
diff --git a/wagtail/wagtailsearch/tests/test_elasticsearch5_backend.py b/wagtail/wagtailsearch/tests/test_elasticsearch5_backend.py
--- a/wagtail/wagtailsearch/tests/test_elasticsearch5_backend.py
+++ b/wagtail/wagtailsearch/tests/test_elasticsearch5_backend.py
@@ -20,12 +20,6 @@
 class TestElasticsearch5SearchBackend(BackendTests, ElasticsearchCommonSearchBackendTests, TestCase):
     backend_path = 'wagtail.wagtailsearch.backends.elasticsearch5'
 
-    # TODO: we have to put [:100] on the end of the query due to issue #3431
-    def test_search_all(self):
-        # Searches on None should return everything in the index
-        results = self.backend.search(None, models.Book)[:100]
-        self.assertSetEqual(set(results), set(models.Book.objects.all()))
-
     # Broken
     @unittest.expectedFailure
     def test_filter_isnull_true(self):
@@ -394,11 +388,12 @@ def test_basic_search(self, search):
         list(results)  # Performs search
 
         search.assert_any_call(
-            from_=0,
             body={'query': 'QUERY'},
             _source=False,
             stored_fields='pk',
-            index='wagtail__searchtests_book'
+            index='wagtail__searchtests_book',
+            scroll='2m',
+            size=100
         )
 
     @mock.patch('elasticsearch.Elasticsearch.search')
diff --git a/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py b/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py
--- a/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py
+++ b/wagtail/wagtailsearch/tests/test_elasticsearch_backend.py
@@ -22,12 +22,6 @@
 class TestElasticsearchSearchBackend(BackendTests, ElasticsearchCommonSearchBackendTests, TestCase):
     backend_path = 'wagtail.wagtailsearch.backends.elasticsearch'
 
-    # TODO: we have to put [:100] on the end of the query due to issue #3431
-    def test_search_all(self):
-        # Searches on None should return everything in the index
-        results = self.backend.search(None, models.Book)[:100]
-        self.assertSetEqual(set(results), set(models.Book.objects.all()))
-
     # Broken
     @unittest.expectedFailure
     def test_filter_in_values_list_subquery(self):
@@ -391,11 +385,12 @@ def test_basic_search(self, search):
         list(results)  # Performs search
 
         search.assert_any_call(
-            from_=0,
             body={'query': 'QUERY'},
             _source=False,
             fields='pk',
-            index='wagtail'
+            index='wagtail',
+            scroll='2m',
+            size=100
         )
 
     @mock.patch('elasticsearch.Elasticsearch.search')