Repository: django-haystack/django-haystack

Issue:
Search (ala SearchView) doesn't work with UUID pk's(Non Int pk's in general)
* [x] Tested with the latest Haystack release
* [x] Tested with the current Haystack master branch

## Expected behaviour
You could use UUID pk's for models in django.  Setup a SearchView and it all works.

## Actual behaviour
You can see the number of results with sqs.all().count() in for example your SearchForm Class.  But the moment you try to instantiate and get all the results...it evaporates.  No results available when you try to iterate over the list.

## Steps to reproduce the behaviour

1.Use UUID's as primary keys on django models that are indexed
2.Create a SearchView and SearchForm that use the Model.
3.Try to use the sqs in the html template through object_list OR a paginated result set (page_obj.object_list)

## Configuration

* Operating system version:Ubuntu 16.04
* Search engine version:6.5
* Python version:3.5.2
* Django version:1.11.4
* Haystack version:master branch and earlier

## Root Cause Analysis
Still in process but it looks like post_process_results of query.py:SearchQuerySet tries to force an int() conversion on the pk causing a problem...or it's involved somehow.  resulting in every result being discarded as an ignored result silently

Still working out the exact nature.

This took me forever to figure out.  I thought I had it working before...I don't know what changed to make it stop.


Discussion:
Going to work on a PR to fix as soon as I finish detailed root cause
I am correct.  The problem lies in assuming all the keys can be coerced to ints (in otherwords assuming that all primary keys of indexed models are ints -- if they are not strings).

I am looking into the proper way to handle this.

Initial thought was to initialize the class type from the first key we can pull out of the database (2.x and 3.x python compatible):
```
for result in results:
            if self._load_all:
                result_klass = None
                # We have to deal with semi-arbitrary keys being cast from strings (UUID, int, etc)
                # devine type on first try.
                model_objects = loaded_objects.get(result.model, {})
                if result.pk not in model_objects:
                    if result_klass is None:
                        result_klass = type(next(iter(model_objects)))

                    try:
                        result.pk = result_klass(result.pk)
```

The problem is that you can have multiple indexed models.  I need to map the indexed model type to the class and do a lookup.  Thinking about the best way to go about this.

I am incredibly surprised that the rest of django-haystack worked and regular search did not when I was using UUID's.
@acdha I'd appreciate any guidance or insight about how you might think to solve this.  I'll be hacking away over the coming spare time snippets I get to fix this.
At second look...That looks like it will solve it properly.  My fix only adds a O(constant) performance overhead with a couple lines.  I just need to add some tests to prevent UUID's breaking again.

This brings up a design decision discussion.  What primary keys should django-haystack allow?  Django allows a pretty wide range of primary id's.  I don't know about the full swath of our supported db's.  I'm happy enough with UUID+str+int.  And I probably wont invest a lot of work beyond getting the UUID's working...but perhaps it's something to think about.

Now that I've looked through and breadcrumb debugged so much of the code I feel like there's performance improvements to be made.  There is a lot of mapping and stuff going on that I feel could be avoided.

Resolving patch:
diff --git a/haystack/query.py b/haystack/query.py
--- a/haystack/query.py
+++ b/haystack/query.py
@@ -172,23 +172,32 @@ def post_process_results(self, results):
 
         for result in results:
             if self._load_all:
-                # We have to deal with integer keys being cast from strings
+
                 model_objects = loaded_objects.get(result.model, {})
-                if result.pk not in model_objects:
+                # Try to coerce a primary key object that matches the models pk
+                # We have to deal with semi-arbitrary keys being cast from strings (UUID, int, etc)
+                if model_objects:
+                    result_klass = type(next(iter(model_objects)))
+                    result.pk = result_klass(result.pk)
+
                     try:
-                        result.pk = int(result.pk)
-                    except ValueError:
-                        pass
-                try:
-                    result._object = model_objects[result.pk]
-                except KeyError:
-                    # The object was either deleted since we indexed or should
-                    # be ignored; fail silently.
-                    self._ignored_result_count += 1
-
-                    # avoid an unfilled None at the end of the result cache
-                    self._result_cache.pop()
-                    continue
+                        result._object = model_objects[result.pk]
+                    except KeyError:
+                        # The object was either deleted since we indexed or should
+                        # be ignored for other reasons such as an overriden 'load_all_queryset';
+                        # fail silently.
+                        self._ignored_result_count += 1
+
+                        # avoid an unfilled None at the end of the result cache
+                        self._result_cache.pop()
+                        continue
+                else:
+                    # No objects were returned -- possible due to SQS nesting such as
+                    # XYZ.objects.filter(id__gt=10) where the amount ignored are
+                    # exactly equal to the ITERATOR_LOAD_PER_QUERY
+                    del self._result_cache[:len(results)]
+                    self._ignored_result_count += len(results)
+                    break
 
             to_cache.append(result)
 


Test patch:
diff --git a/test_haystack/core/fixtures/base_data.json b/test_haystack/core/fixtures/base_data.json
--- a/test_haystack/core/fixtures/base_data.json
+++ b/test_haystack/core/fixtures/base_data.json
@@ -70,5 +70,19 @@
       "author": "sam2",
       "deleted": true
     }
+  },
+  {
+    "pk": "53554c58-7051-4350-bcc9-dad75eb248a9",
+    "model": "core.uuidmockmodel",
+    "fields": {
+      "characteristics": "some text that was indexed"
+    }
+  },
+  {
+    "pk": "77554c58-7051-4350-bcc9-dad75eb24888",
+    "model": "core.uuidmockmodel",
+    "fields": {
+      "characteristics": "more text that was indexed"
+    }
   }
 ]
diff --git a/test_haystack/core/models.py b/test_haystack/core/models.py
--- a/test_haystack/core/models.py
+++ b/test_haystack/core/models.py
@@ -25,7 +25,7 @@ def __unicode__(self):
     def hello(self):
         return 'World!'
 
-class MockUUIDModel(models.Model):
+class UUIDMockModel(models.Model):
     id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
     characteristics = models.TextField()
 
diff --git a/test_haystack/mocks.py b/test_haystack/mocks.py
--- a/test_haystack/mocks.py
+++ b/test_haystack/mocks.py
@@ -108,6 +108,11 @@ class CharPKMockSearchBackend(MockSearchBackend):
     mock_search_results = [MockSearchResult('core', 'CharPKMockModel', 'sometext', 0.5),
                            MockSearchResult('core', 'CharPKMockModel', '1234', 0.3)]
 
+class UUIDMockSearchBackend(MockSearchBackend):
+    model_name = 'uuidmockmodel'
+    mock_search_results = [MockSearchResult('core', 'UUIDMockModel', '53554c58-7051-4350-bcc9-dad75eb248a9', 0.5),
+                           MockSearchResult('core', 'UUIDMockModel', '77554c58-7051-4350-bcc9-dad75eb24888', 0.5)]
+
 
 class ReadQuerySetMockSearchBackend(MockSearchBackend):
     model_name = 'afifthmockmodel'
diff --git a/test_haystack/test_query.py b/test_haystack/test_query.py
--- a/test_haystack/test_query.py
+++ b/test_haystack/test_query.py
@@ -6,7 +6,7 @@
 
 from django.test import TestCase
 from django.test.utils import override_settings
-from test_haystack.core.models import AnotherMockModel, CharPKMockModel, MockModel
+from test_haystack.core.models import AnotherMockModel, CharPKMockModel, MockModel, UUIDMockModel
 
 from haystack import connections, indexes, reset_search_queries
 from haystack.backends import SQ, BaseSearchQuery
@@ -16,7 +16,7 @@
 from haystack.utils.loading import UnifiedIndex
 
 from .mocks import (MOCK_SEARCH_RESULTS, CharPKMockSearchBackend, MockSearchBackend, MockSearchQuery,
-                    ReadQuerySetMockSearchBackend)
+                    ReadQuerySetMockSearchBackend, UUIDMockSearchBackend)
 from .test_indexes import GhettoAFifthMockModelSearchIndex, TextReadQuerySetTestSearchIndex
 from .test_views import BasicAnotherMockModelSearchIndex, BasicMockModelSearchIndex
 
@@ -323,6 +323,11 @@ class CharPKMockModelSearchIndex(indexes.SearchIndex, indexes.Indexable):
     def get_model(self):
         return CharPKMockModel
 
+class SimpleMockUUIDModelIndex(indexes.SearchIndex, indexes.Indexable):
+    text = indexes.CharField(document=True, model_attr="characteristics")
+
+    def get_model(self):
+        return UUIDMockModel
 
 @override_settings(DEBUG=True)
 class SearchQuerySetTestCase(TestCase):
@@ -336,7 +341,8 @@ def setUp(self):
         self.ui = UnifiedIndex()
         self.bmmsi = BasicMockModelSearchIndex()
         self.cpkmmsi = CharPKMockModelSearchIndex()
-        self.ui.build(indexes=[self.bmmsi, self.cpkmmsi])
+        self.uuidmmsi = SimpleMockUUIDModelIndex()
+        self.ui.build(indexes=[self.bmmsi, self.cpkmmsi, self.uuidmmsi])
         connections['default']._index = self.ui
 
         # Update the "index".
@@ -402,6 +408,8 @@ def test_manual_iter(self):
         # Test to ensure we properly fill the cache, even if we get fewer
         # results back (not a handled model) than the hit count indicates.
         # This will hang indefinitely if broken.
+
+        # CharPK testing
         old_ui = self.ui
         self.ui.build(indexes=[self.cpkmmsi])
         connections['default']._index = self.ui
@@ -412,6 +420,15 @@ def test_manual_iter(self):
         self.assertEqual(loaded, [u'sometext', u'1234'])
         self.assertEqual(len(connections['default'].queries), 1)
 
+        #UUID testing
+        self.ui.build(indexes=[self.uuidmmsi])
+        connections['default']._index = self.ui
+        self.uuidmmsi.update()
+
+        results = self.msqs.all()
+        loaded = [result.pk for result in results._manual_iter()]
+        self.assertEqual(loaded, [u'53554c58-7051-4350-bcc9-dad75eb248a9', u'77554c58-7051-4350-bcc9-dad75eb24888'])
+
         connections['default']._index = old_ui
 
     def test_cache_is_full(self):
@@ -524,6 +541,14 @@ def test_load_all(self):
         results._fill_cache(0, 2)
         self.assertEqual(len([result for result in results._result_cache if result is not None]), 2)
 
+        # Models with uuid primary keys.
+        sqs = SearchQuerySet()
+        sqs.query.backend = UUIDMockSearchBackend('uuid')
+        results = sqs.load_all().all()
+        self.assertEqual(len(results._result_cache), 0)
+        results._fill_cache(0, 2)
+        self.assertEqual(len([result for result in results._result_cache if result is not None]), 2)
+
         # If nothing is handled, you get nothing.
         old_ui = connections['default']._index
         ui = UnifiedIndex()