Repository: django-oscar/django-oscar

Issue:
ProductQuerySet.base_queryset() should not perform annotations
The annotations performed in [`base_queryset()`](https://github.com/django-oscar/django-oscar/blob/a5096da38a265925f768ec5ce5258e9b7b1a2d72/src/oscar/apps/catalogue/managers.py#L97-L98) are very expensive. 

We have a catalogue of 250,000 products, where loading the dashboard products list view takes about 6-7 seconds because of these annotations - despite the fact that they are not used anywhere in that view. Removing the annotation drops this to less than 0.5 seconds.

I propose that the base queryset should be just that - a base queryset without any annotations. The views that need to use options should add annotations when preparing their context. The same argument could actually apply for the prefetching being done here. I think we need to review all the places that `base_queryset()` is used, and whether it is more optimal to perform the prefetching at point of use.


Discussion:
Hi solarissmoke, I have been doing a lot with the querysets these days and in this case I believe that the performance problem could be caused by your queryset not being sliced. It could be that your entire 250000 products are being returned annotated, which could cause the annotation to be very slow indeed because it performs a count. If done on a single page of objects, that would be harmless. Could you please turn on db logging and make sure that there is no query being performed that would return all the objects in your database?

The paginators are tricky and sometimes do not slice the queryset.
@specialunderwear the query is being sliced. Here is the generated SQL for the problematic query - note the `LIMIT` clause at the end:

```
SELECT ••• FROM "catalogue_product" 
LEFT OUTER JOIN "catalogue_productclass" ON ("catalogue_product"."product_class_id" = "catalogue_productclass"."id") 
LEFT OUTER JOIN "catalogue_productclass_options" ON ("catalogue_productclass"."id" = "catalogue_productclass_options"."productclass_id") 
LEFT OUTER JOIN "catalogue_product_product_options" ON ("catalogue_product"."id" = "catalogue_product_product_options"."product_id") 
WHERE "catalogue_product"."parent_id" IS NULL 
GROUP BY "catalogue_product"."id", "catalogue_productclass"."id" 
ORDER BY "catalogue_product"."date_updated" DESC LIMIT 20
```

This query takes 2,000ms. If the annotation is removed it takes 1.9ms and is reduced to:

```
SELECT ••• FROM "catalogue_product" 
LEFT OUTER JOIN "catalogue_productclass" ON ("catalogue_product"."product_class_id" = "catalogue_productclass"."id") 
WHERE "catalogue_product"."parent_id" IS NULL 
ORDER BY "catalogue_product"."date_updated" DESC LIMIT 20
```

I'm fairly certain that the annotation is problematic. The annotation appears to be happening before the `LIMIT` is applied in the database.
Thanks solarissmoke, I think the group by makes that query slow. has_options is the only method that uses that info. I don't think has_options would be very hard to implement efficiently in a different way.
Something like this would work, but not that this always returns all the options of product class and product in one go. That is not what the current options method does. At least it returns a queryset!
```
    @property
    def options(self):
        Option = self._meta.get_field("product_options").related_model
        return Option.objects.filter(
            models.Q(product=self) | models.Q(productclass=self.product_class_id or -1),
        ).distinct()

    @property
    def has_options(self):
        return self.options.exists()
```

That might need a bit of work for parent child products.
It is really fun to use the newest query features in django !!!

Resolving patch:
diff --git a/src/oscar/apps/catalogue/abstract_models.py b/src/oscar/apps/catalogue/abstract_models.py
--- a/src/oscar/apps/catalogue/abstract_models.py
+++ b/src/oscar/apps/catalogue/abstract_models.py
@@ -499,10 +499,10 @@ def options(self):
     def has_options(self):
         # Extracting annotated value with number of product class options
         # from product list queryset.
-        num_product_class_options = getattr(self, 'num_product_class_options', None)
-        num_product_options = getattr(self, 'num_product_options', None)
-        if num_product_class_options is not None and num_product_options is not None:
-            return num_product_class_options > 0 or num_product_options > 0
+        has_product_class_options = getattr(self, 'has_product_class_options', None)
+        has_product_options = getattr(self, 'has_product_options', None)
+        if has_product_class_options is not None and has_product_options is not None:
+            return has_product_class_options or has_product_options
         return self.get_product_class().options.exists() or self.product_options.exists()
 
     @property
diff --git a/src/oscar/apps/catalogue/managers.py b/src/oscar/apps/catalogue/managers.py
--- a/src/oscar/apps/catalogue/managers.py
+++ b/src/oscar/apps/catalogue/managers.py
@@ -1,10 +1,11 @@
 from collections import defaultdict
 
 from django.db import models
-from django.db.models import Count
+from django.db.models import OuterRef, Exists
 from django.db.models.constants import LOOKUP_SEP
 
 from oscar.core.decorators import deprecated
+from oscar.core.loading import get_model
 
 
 class AttributeFilter(dict):
@@ -92,10 +93,13 @@ def base_queryset(self):
         Applies select_related and prefetch_related for commonly related
         models to save on queries
         """
+        Option = get_model('catalogue', 'Option')
+        product_class_options = Option.objects.filter(productclass=OuterRef('product_class'))
+        product_options = Option.objects.filter(product=OuterRef('pk'))
         return self.select_related('product_class')\
             .prefetch_related('children', 'product_options', 'product_class__options', 'stockrecords', 'images') \
-            .annotate(num_product_class_options=Count('product_class__options'),
-                      num_product_options=Count('product_options'))
+            .annotate(has_product_class_options=Exists(product_class_options),
+                      has_product_options=Exists(product_options))
 
     def browsable(self):
         """


Test patch:
diff --git a/tests/integration/catalogue/test_options.py b/tests/integration/catalogue/test_options.py
--- a/tests/integration/catalogue/test_options.py
+++ b/tests/integration/catalogue/test_options.py
@@ -26,14 +26,14 @@ def test_queryset_per_product_class(self):
         qs = Product.objects.browsable().base_queryset().filter(id=self.product.id)
         product = qs.first()
         self.assertTrue(product.has_options)
-        self.assertEquals(product.num_product_class_options, 1)
+        self.assertTrue(product.has_product_class_options)
 
     def test_queryset_per_product(self):
         self.product.product_options.add(self.option)
         qs = Product.objects.browsable().base_queryset().filter(id=self.product.id)
         product = qs.first()
         self.assertTrue(product.has_options)
-        self.assertEquals(product.num_product_options, 1)
+        self.assertTrue(product.has_product_options, 1)
 
     def test_queryset_both(self):
         "The options attribute on a product should return a queryset containing "
@@ -49,13 +49,13 @@ def test_queryset_both(self):
         )
         qs = Product.objects.browsable().base_queryset().filter(id=self.product.id)
         product = qs.first()
-        self.assertEquals(
-            product.num_product_class_options, 1,
-            "num_product_class_options should indicate the product_class option"
+        self.assertTrue(
+            product.has_product_class_options,
+            "has_product_class_options should indicate the product_class option"
         )
-        self.assertEquals(
-            product.num_product_options, 1,
-            "num_product_options should indicate the number of product options"
+        self.assertTrue(
+            product.has_product_options,
+            "has_product_options should indicate the number of product options"
         )
         self.product_class.options.add(factories.OptionFactory(code="henk"))
         self.assertEquals(