Repository: django-import-export/django-import-export

Issue:
CHUNK_SIZE defaults to 1, so unless explicitly set, prefetching won't work
**Describe the bug**
Not setting `CHUNK_SIZE` in your resource will cause it to default to being set to 1. Since we need to paginate the queryset to enable prefetching, this means we iterate over the queryset row by row, which prevents prefetching from working.

Setting the `CHUNK_SIZE` explicitly against the resource fixes the issue and lets the prefetch work, but there's no signal that this problem could be happening to the user, which can be very confusing if you're new to prefetch and can't see why it's not working (not that I know anyone who fits that description, ahem)

**To Reproduce**
Steps to reproduce the behavior:
1. Remove the `CHUNK_SIZE` setting from your resource
2. Make a resource for a Django model that has a valid prefetch option, e.g. a `m2m_thing = ManyToManyField(…)`
3. Export the resource using `MyResource().export(queryset.prefetch_related('m2m_thing')`
4. See in the query log that one query for `m2m_thing` is run for each row of the model

**Versions:**
 - Django Import Export: 2.3.0
 - Python: 3.6.9
 - Django: 3.0.8 

**Expected behavior**
* Setting the `CHUNK_SIZE` default value to something higher, e.g. 5000?
* Possibly a warning output if `CHUNK_SIZE` isn't set but prefetches are being used on the queryset?
* A note added to the docs against `CHUNK_SIZE`?



Discussion:
I assume this is similar to what was reported here? https://github.com/django-import-export/django-import-export/pull/1050#issuecomment-664473338
> I assume this is similar to what was reported here? [#1050 (comment)](https://github.com/django-import-export/django-import-export/pull/1050#issuecomment-664473338)

It absolutely is - describes it perfectly. 

Happy to close this issue as a duplicate, but since #1050 is about the fundamental fix for `prefetch_related`, I think it makes sense to keep this issue open to track the simple "the default value doesn't work as expected" angle that @pawelzar points out?
I think it makes sense to keep this one as it's a new issue (regression). I should've created one, but I figured I'll just post a comment in the original issue, where the `CHUNK_SIZE` was added and wait for some feedback.

@jdewells I'd only change the title, because theoretically the `prefetching` works, but since queryset is evaluated for each item due to `CHUNK_SIZE = 1` it just doesn't make much sense.

I agree that the solution should be to set the default value much higher as it shouldn't harm anyone. There is no point in trying to keep "the old behavior" when the performance could just be improved for everyone.
> @jdewells I'd only change the title, because theoretically the `prefetching` works, but since queryset is evaluated for each item due to `CHUNK_SIZE = 1` it just doesn't make much sense.

Any suggestions? "CHUNK_SIZE defaults to 1, forcing queryset to evaluate for each row & preventing optimisation"?

Works for me @jdewells :+1:

Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -24,14 +24,6 @@
 from .signals import post_export, post_import
 from .tmp_storages import TempFolderStorage
 
-SKIP_ADMIN_LOG = getattr(settings, 'IMPORT_EXPORT_SKIP_ADMIN_LOG', False)
-TMP_STORAGE_CLASS = getattr(settings, 'IMPORT_EXPORT_TMP_STORAGE_CLASS',
-                            TempFolderStorage)
-
-
-if isinstance(TMP_STORAGE_CLASS, str):
-    TMP_STORAGE_CLASS = import_string(TMP_STORAGE_CLASS)
-
 
 class ImportExportMixinBase:
     def get_model_info(self):
@@ -63,15 +55,21 @@ class ImportMixin(ImportExportMixinBase):
 
     def get_skip_admin_log(self):
         if self.skip_admin_log is None:
-            return SKIP_ADMIN_LOG
+            return getattr(settings, 'IMPORT_EXPORT_SKIP_ADMIN_LOG', False)
         else:
             return self.skip_admin_log
 
     def get_tmp_storage_class(self):
         if self.tmp_storage_class is None:
-            return TMP_STORAGE_CLASS
+            tmp_storage_class = getattr(
+                settings, 'IMPORT_EXPORT_TMP_STORAGE_CLASS', TempFolderStorage,
+            )
         else:
-            return self.tmp_storage_class
+            tmp_storage_class = self.tmp_storage_class
+
+        if isinstance(tmp_storage_class, str):
+            tmp_storage_class = import_string(tmp_storage_class)
+        return tmp_storage_class
 
     def has_import_permission(self, request):
         """
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -41,9 +41,6 @@
 # Set default logging handler to avoid "No handler found" warnings.
 logger.addHandler(logging.NullHandler())
 
-USE_TRANSACTIONS = getattr(settings, 'IMPORT_EXPORT_USE_TRANSACTIONS', True)
-CHUNK_SIZE = getattr(settings, 'IMPORT_EXPORT_CHUNK_SIZE', 1)
-
 
 def get_related_model(field):
     if hasattr(field, 'related_model'):
@@ -126,10 +123,10 @@ class ResourceOptions:
 
     chunk_size = None
     """
-    Controls the chunk_size argument of Queryset.iterator or, 
+    Controls the chunk_size argument of Queryset.iterator or,
     if prefetch_related is used, the per_page attribute of Paginator.
     """
-    
+
     skip_diff = False
     """
     Controls whether or not an instance should be diffed following import.
@@ -159,8 +156,8 @@ class ResourceOptions:
     force_init_instance = False
     """
     If True, this parameter will prevent imports from checking the database for existing instances.
-    Enabling this parameter is a performance enhancement if your import dataset is guaranteed to 
-    contain new instances. 
+    Enabling this parameter is a performance enhancement if your import dataset is guaranteed to
+    contain new instances.
     """
 
 
@@ -278,13 +275,13 @@ def get_diff_class(self):
 
     def get_use_transactions(self):
         if self._meta.use_transactions is None:
-            return USE_TRANSACTIONS
+            return getattr(settings, 'IMPORT_EXPORT_USE_TRANSACTIONS', True)
         else:
             return self._meta.use_transactions
 
     def get_chunk_size(self):
         if self._meta.chunk_size is None:
-            return CHUNK_SIZE
+            return getattr(settings, 'IMPORT_EXPORT_CHUNK_SIZE', 100)
         else:
             return self._meta.chunk_size
 


Test patch:
diff --git a/tests/core/tests/test_resources.py b/tests/core/tests/test_resources.py
--- a/tests/core/tests/test_resources.py
+++ b/tests/core/tests/test_resources.py
@@ -344,7 +344,7 @@ def test_iter_queryset(self):
         qs = Book.objects.all()
         with mock.patch.object(qs, "iterator") as mocked_method:
             list(self.resource.iter_queryset(qs))
-            mocked_method.assert_called_once_with(chunk_size=1)
+            mocked_method.assert_called_once_with(chunk_size=100)
 
     def test_iter_queryset_prefetch_unordered(self):
         qsu = Book.objects.prefetch_related("categories").all()
@@ -357,9 +357,9 @@ def test_iter_queryset_prefetch_unordered(self):
     def test_iter_queryset_prefetch_ordered(self):
         qs = Book.objects.prefetch_related("categories").order_by('pk').all()
         with mock.patch("import_export.resources.Paginator", autospec=True) as p:
-            p.return_value = Paginator(qs, 1)
+            p.return_value = Paginator(qs, 100)
             list(self.resource.iter_queryset(qs))
-            p.assert_called_once_with(qs, 1)
+            p.assert_called_once_with(qs, 100)
 
     def test_iter_queryset_prefetch_chunk_size(self):
         class B(BookResource):