Repository: encode/django-rest-framework

Issue:
Optimize pk-based hyperlinked relationships.
Maybe I'm doing something wrong here. When I ask my application for a list of objects (e.g. /my-objects/) it makes 2 initial database queries (one for the count and one for the list). Then during serialization - using hyperlinked serialization - for each related field on each object another query is fired. So I easily end up with more than 200 database queries for a simple list - that's just unacceptable.

Looking into the code it's clear what causes this: in order to create the hyperlinks to the related objects, those objects must be loaded (often the same object is loaded 100 times). So my assumption is that the described behavior is not caused by misconfiguration but by design.

I created a kind of hacky solution which uses the _my_related_object_id_ property to generate the url for _my_related_object_ without loading the object (of which only the id is used anyway). Is that valid? Shouldn't this be something that's done by drf? It seems to work, haven't had any problems so far.

Even when this is solved the combination with nested routes is a whole new adventure, but that's out of the scope of this project / ticket :)



Discussion:
Let me start off by saying that this is a known issue with Django REST Framework, but it's also _very_ difficult to solve for the general use case.

> Shouldn't this be something that's done by drf?

While this sounds like a really easy fix that should work for everyone, not everyone links by the `id`, and that is why we don't do this by default. That doesn't make it a bad move though, as there really isn't a general use case here.

I have had success working around this issue with combinations of `select_related` and `prefetch_related`  when getting the queryset in `get_queryset`. It's not something which DRF can detect and add in automatically, as there are tons of cases to consider.

Seconding @kevin-brown, tho it would be great if we could come up with a sensible answer to this. Going to re-review as part of the 3.0 release.

Thanks for the explanation, sounds plausible to me. I feel encouraged to go with the solution I have so far - we'll see how far it carries. Also, I'm using `select_related` for the nested routes because they surely need the related objects in order to create the urls for those. In the end the number of models / serializers is limited and the code is not too hard to write. So there is always the option to modify the queryset per model as you suggested.

I'll close this issue and check again in version 3.0. In case I come up with something extremely intelligent and generic I'll let you know, but that's kinda unlikely because we only link via id.

I haven't checked whether this is fixed in 3.0, but FWIW in 2.x `PrimaryKeyRelatedField` already has some workarounds to avoid unnecessarily querying related objects when all that's needed is their PK, but `HyperlinkedRelatedField` has no such workarounds, even though it also needs only the PK. For now I'm using this workaround in a subclass of `HyperlinkedRelatedField`:

```
+    def field_to_native(self, obj, field_name):
+        """Avoid querying for related objects when all we need is the pk."""
+        source = self.source or field_name
+        if obj is not None and '.' not in source and not self.many:
+            stub = HyperlinkedRelatedStub(
+                **{self.lookup_field: getattr(obj, source + '_id')})
+            return self.to_native(stub)
+        return super(HyperlinkedRelatedField, self).field_to_native(
+            obj, field_name)
```

where `HyperlinkedRelatedStub` is just:

```
class HyperlinkedRelatedStub(object):
    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)
```

It's not the prettiest solution, due to the need for that stub object, but the stub avoids the need to change the signature of `HyperlinkedRelatedField.to_native` or `HyperlinkedRelatedField.get_url`, and I haven't dug in to see if those might be called from elsewhere.

I'm sure that it would be possible to apply a nicer fix-up to `HyperlinkedRelatedField` that would avoid needless queries on related objects (presuming it hasn't already been done in 3.0).

Not yet done for 3.0, but yes planned, and won't be much work at all. Re opening this to ensure it stays tracked.


Resolving patch:
diff --git a/rest_framework/relations.py b/rest_framework/relations.py
--- a/rest_framework/relations.py
+++ b/rest_framework/relations.py
@@ -84,9 +84,20 @@ def get_queryset(self):
             queryset = queryset.all()
         return queryset
 
-    def get_iterable(self, instance, source_attrs):
-        relationship = get_attribute(instance, source_attrs)
-        return relationship.all() if (hasattr(relationship, 'all')) else relationship
+    def use_pk_only_optimization(self):
+        return False
+
+    def get_attribute(self, instance):
+        if self.use_pk_only_optimization() and self.source_attrs:
+            # Optimized case, return a mock object only containing the pk attribute.
+            try:
+                instance = get_attribute(instance, self.source_attrs[:-1])
+                return PKOnlyObject(pk=instance.serializable_value(self.source_attrs[-1]))
+            except AttributeError:
+                pass
+
+        # Standard case, return the object instance.
+        return get_attribute(instance, self.source_attrs)
 
     @property
     def choices(self):
@@ -120,6 +131,9 @@ class PrimaryKeyRelatedField(RelatedField):
         'incorrect_type': _('Incorrect type. Expected pk value, received {data_type}.'),
     }
 
+    def use_pk_only_optimization(self):
+        return True
+
     def to_internal_value(self, data):
         try:
             return self.get_queryset().get(pk=data)
@@ -128,32 +142,6 @@ def to_internal_value(self, data):
         except (TypeError, ValueError):
             self.fail('incorrect_type', data_type=type(data).__name__)
 
-    def get_attribute(self, instance):
-        # We customize `get_attribute` here for performance reasons.
-        # For relationships the instance will already have the pk of
-        # the related object. We return this directly instead of returning the
-        # object itself, which would require a database lookup.
-        try:
-            instance = get_attribute(instance, self.source_attrs[:-1])
-            return PKOnlyObject(pk=instance.serializable_value(self.source_attrs[-1]))
-        except AttributeError:
-            return get_attribute(instance, self.source_attrs)
-
-    def get_iterable(self, instance, source_attrs):
-        # For consistency with `get_attribute` we're using `serializable_value()`
-        # here. Typically there won't be any difference, but some custom field
-        # types might return a non-primitive value for the pk otherwise.
-        #
-        # We could try to get smart with `values_list('pk', flat=True)`, which
-        # would be better in some case, but would actually end up with *more*
-        # queries if the developer is using `prefetch_related` across the
-        # relationship.
-        relationship = super(PrimaryKeyRelatedField, self).get_iterable(instance, source_attrs)
-        return [
-            PKOnlyObject(pk=item.serializable_value('pk'))
-            for item in relationship
-        ]
-
     def to_representation(self, value):
         return value.pk
 
@@ -184,6 +172,9 @@ def __init__(self, view_name=None, **kwargs):
 
         super(HyperlinkedRelatedField, self).__init__(**kwargs)
 
+    def use_pk_only_optimization(self):
+        return self.lookup_field == 'pk'
+
     def get_object(self, view_name, view_args, view_kwargs):
         """
         Return the object corresponding to a matched URL.
@@ -285,6 +276,11 @@ def __init__(self, view_name=None, **kwargs):
         kwargs['source'] = '*'
         super(HyperlinkedIdentityField, self).__init__(view_name, **kwargs)
 
+    def use_pk_only_optimization(self):
+        # We have the complete object instance already. We don't need
+        # to run the 'only get the pk for this relationship' code.
+        return False
+
 
 class SlugRelatedField(RelatedField):
     """
@@ -349,7 +345,8 @@ def to_internal_value(self, data):
         ]
 
     def get_attribute(self, instance):
-        return self.child_relation.get_iterable(instance, self.source_attrs)
+        relationship = get_attribute(instance, self.source_attrs)
+        return relationship.all() if (hasattr(relationship, 'all')) else relationship
 
     def to_representation(self, iterable):
         return [


Test patch:
diff --git a/tests/test_relations_hyperlink.py b/tests/test_relations_hyperlink.py
--- a/tests/test_relations_hyperlink.py
+++ b/tests/test_relations_hyperlink.py
@@ -89,7 +89,14 @@ def test_many_to_many_retrieve(self):
             {'url': 'http://testserver/manytomanysource/2/', 'name': 'source-2', 'targets': ['http://testserver/manytomanytarget/1/', 'http://testserver/manytomanytarget/2/']},
             {'url': 'http://testserver/manytomanysource/3/', 'name': 'source-3', 'targets': ['http://testserver/manytomanytarget/1/', 'http://testserver/manytomanytarget/2/', 'http://testserver/manytomanytarget/3/']}
         ]
-        self.assertEqual(serializer.data, expected)
+        with self.assertNumQueries(4):
+            self.assertEqual(serializer.data, expected)
+
+    def test_many_to_many_retrieve_prefetch_related(self):
+        queryset = ManyToManySource.objects.all().prefetch_related('targets')
+        serializer = ManyToManySourceSerializer(queryset, many=True, context={'request': request})
+        with self.assertNumQueries(2):
+            serializer.data
 
     def test_reverse_many_to_many_retrieve(self):
         queryset = ManyToManyTarget.objects.all()
@@ -99,7 +106,8 @@ def test_reverse_many_to_many_retrieve(self):
             {'url': 'http://testserver/manytomanytarget/2/', 'name': 'target-2', 'sources': ['http://testserver/manytomanysource/2/', 'http://testserver/manytomanysource/3/']},
             {'url': 'http://testserver/manytomanytarget/3/', 'name': 'target-3', 'sources': ['http://testserver/manytomanysource/3/']}
         ]
-        self.assertEqual(serializer.data, expected)
+        with self.assertNumQueries(4):
+            self.assertEqual(serializer.data, expected)
 
     def test_many_to_many_update(self):
         data = {'url': 'http://testserver/manytomanysource/1/', 'name': 'source-1', 'targets': ['http://testserver/manytomanytarget/1/', 'http://testserver/manytomanytarget/2/', 'http://testserver/manytomanytarget/3/']}
@@ -197,7 +205,8 @@ def test_foreign_key_retrieve(self):
             {'url': 'http://testserver/foreignkeysource/2/', 'name': 'source-2', 'target': 'http://testserver/foreignkeytarget/1/'},
             {'url': 'http://testserver/foreignkeysource/3/', 'name': 'source-3', 'target': 'http://testserver/foreignkeytarget/1/'}
         ]
-        self.assertEqual(serializer.data, expected)
+        with self.assertNumQueries(1):
+            self.assertEqual(serializer.data, expected)
 
     def test_reverse_foreign_key_retrieve(self):
         queryset = ForeignKeyTarget.objects.all()
@@ -206,7 +215,8 @@ def test_reverse_foreign_key_retrieve(self):
             {'url': 'http://testserver/foreignkeytarget/1/', 'name': 'target-1', 'sources': ['http://testserver/foreignkeysource/1/', 'http://testserver/foreignkeysource/2/', 'http://testserver/foreignkeysource/3/']},
             {'url': 'http://testserver/foreignkeytarget/2/', 'name': 'target-2', 'sources': []},
         ]
-        self.assertEqual(serializer.data, expected)
+        with self.assertNumQueries(3):
+            self.assertEqual(serializer.data, expected)
 
     def test_foreign_key_update(self):
         data = {'url': 'http://testserver/foreignkeysource/1/', 'name': 'source-1', 'target': 'http://testserver/foreignkeytarget/2/'}
diff --git a/tests/test_relations_pk.py b/tests/test_relations_pk.py
--- a/tests/test_relations_pk.py
+++ b/tests/test_relations_pk.py
@@ -71,6 +71,12 @@ def test_many_to_many_retrieve(self):
         with self.assertNumQueries(4):
             self.assertEqual(serializer.data, expected)
 
+    def test_many_to_many_retrieve_prefetch_related(self):
+        queryset = ManyToManySource.objects.all().prefetch_related('targets')
+        serializer = ManyToManySourceSerializer(queryset, many=True)
+        with self.assertNumQueries(2):
+            serializer.data
+
     def test_reverse_many_to_many_retrieve(self):
         queryset = ManyToManyTarget.objects.all()
         serializer = ManyToManyTargetSerializer(queryset, many=True)
@@ -188,6 +194,12 @@ def test_reverse_foreign_key_retrieve(self):
         with self.assertNumQueries(3):
             self.assertEqual(serializer.data, expected)
 
+    def test_reverse_foreign_key_retrieve_prefetch_related(self):
+        queryset = ForeignKeyTarget.objects.all().prefetch_related('sources')
+        serializer = ForeignKeyTargetSerializer(queryset, many=True)
+        with self.assertNumQueries(2):
+            serializer.data
+
     def test_foreign_key_update(self):
         data = {'id': 1, 'name': 'source-1', 'target': 2}
         instance = ForeignKeySource.objects.get(pk=1)
diff --git a/tests/test_relations_slug.py b/tests/test_relations_slug.py
--- a/tests/test_relations_slug.py
+++ b/tests/test_relations_slug.py
@@ -54,7 +54,14 @@ def test_foreign_key_retrieve(self):
             {'id': 2, 'name': 'source-2', 'target': 'target-1'},
             {'id': 3, 'name': 'source-3', 'target': 'target-1'}
         ]
-        self.assertEqual(serializer.data, expected)
+        with self.assertNumQueries(4):
+            self.assertEqual(serializer.data, expected)
+
+    def test_foreign_key_retrieve_select_related(self):
+        queryset = ForeignKeySource.objects.all().select_related('target')
+        serializer = ForeignKeySourceSerializer(queryset, many=True)
+        with self.assertNumQueries(1):
+            serializer.data
 
     def test_reverse_foreign_key_retrieve(self):
         queryset = ForeignKeyTarget.objects.all()
@@ -65,6 +72,12 @@ def test_reverse_foreign_key_retrieve(self):
         ]
         self.assertEqual(serializer.data, expected)
 
+    def test_reverse_foreign_key_retrieve_prefetch_related(self):
+        queryset = ForeignKeyTarget.objects.all().prefetch_related('sources')
+        serializer = ForeignKeyTargetSerializer(queryset, many=True)
+        with self.assertNumQueries(2):
+            serializer.data
+
     def test_foreign_key_update(self):
         data = {'id': 1, 'name': 'source-1', 'target': 'target-2'}
         instance = ForeignKeySource.objects.get(pk=1)