Repository: django-import-export/django-import-export

Issue:
`use_natural_foreign_keys` in ModelResource.Meta not working in V4
**Describe the bug**

`ModelResource`  with `use_natural_foreign_keys=True`  generates `ForeignKeyWidget` with `key_is_id=True` and `use_natural_foreign_keys=True`. User expect using natural keys like it represented in [documentation ]( https://django-import-export.readthedocs.io/en/4.0.0/advanced_usage.html#django-natural-keys) , but `key_is_id` has more priority then `use_natural_foreign_keys` 

**To Reproduce**
Steps to reproduce the behavior:
1. Create `ModelResource` with `Meta.use_natural_foreign_keys = True`
2. Run export with this resource
3. In exported data will be ids of objects

Test to reproduce problem
```python

class BookUsingNaturalKeys(resources.ModelResource):
    class Meta:
        model = Book
        fields = ["name", "author"]
        use_natural_foreign_keys = True


class BookUsingAuthorNaturalKey(resources.ModelResource):
    class Meta:
        model = Book
        fields = ["name", "author"]

    author = fields.Field(
        attribute="author",
        column_name="author",
        widget=widgets.ForeignKeyWidget(
            Author,
            use_natural_foreign_keys=True,
        )
    )

class TestNaturalKeys(TestCase):
    def setUp(self) -> None:
        author = Author.objects.create(name="J. R. R. Tolkien")
        book = Book.objects.create(author=author, name="The Hobbit")
        self.expected_dataset = tablib.Dataset(headers=["name", 'author'])
        row = ["The Hobbit", '["J. R. R. Tolkien"]']
        self.expected_dataset.append(row)

    def test_resource_use_natural_keys(self):
        """
        test with ModelResource.Meta.use_natural_foreign_keys=True
        Reproduces this problem
        """
        resource = BookUsingNaturalKeys()
        exported_dataset = resource.export(Book.objects.all())
        self.assertDatasetEqual(self.expected_dataset, exported_dataset)

    def test_field_use_natural_keys(self):
        """
        test with ModelResource.field.widget.use_natural_foreign_keys=True
        Example of correct behaviour
        """
        resource = BookUsingAuthorNaturalKey()
        exported_dataset = resource.export(Book.objects.all())
        self.assertDatasetEqual(self.expected_dataset, exported_dataset)

    def assertDatasetEqual(self, expected_dataset, actual_dataset, message=None):
        """
        Util for comparing datasets
        """
        self.assertEqual(len(expected_dataset), len(actual_dataset), message)
        for expected_row, actual_row in zip(expected_dataset, actual_dataset, strict=True):
            self.assertEqual(expected_row, actual_row, message)


```

**Versions (please complete the following information):**
 - Django Import Export:  4.0
 - Python 3.12
 - Django 5.0

**Expected behavior**
Natural keys in exported data for autogenerated fields


**Additional context**
I think `use_natural_foreign_keys` and `key_is_id` contradict each other and `ForeignKeyWidget(use_natural_foreign_keys=True, key_is_id=True)` should raise Exception

Performance: slow export with ForeignKeys id
I tried to investigate slow export on my model. I found out, that if I write `dehydrate` function on `ModelResource` like this:
```python
    def dehydrate_user_profile(self, search_report):
        return search_report.user_profile_id
```

The export rapidly speeds up. I think, that `django-import-export` has bad method of getting the `id` values for ForeignKeys.

The bad performance is similar to this (notice the `.` instead of `_`)
```python
    def dehydrate_user_profile(self, search_report):
        return search_report.user_profile.id
```


Discussion:

Hey @PetrDlouhy ! Thanks for the investigation. How much of an increase would you say? Would you like to submit a PR with some of these changes? 
@andrewgy8 In my case the increase was extreme - when I try with just 1300 records it is almost instant vs. 20 seconds.

I investigated a little bit further. The problem is, that on line `90` in `fields.py` in  `get_value()` function the value of foreign key is fetched by:
```
value = getattr(value, attr, None)
```
If the value is not already cached or not fetch by the queryset with `select_related`, it will trigger new DB query for every row of the export.
That value is further processed by `Widget.render()` where the value of certain key is fetched from the related object. This is unnecessary, if the key we want is primary key of the ForeignKey relation.

I am not sure, how to fix this properly.
One method might be to detect, that we want to fetch `pk`/`id` key and bypass the `get_value()->Widget.render()` mechanism.
Downside is that it might change behaviour if one of these function was overridden by user.

Second method is to automatically include `select_related` to the queryset.
Upside is that it can greatly improve performance not only for primary keys, but also for related fields. 
Downside is that it might not work for some user modified querysets properly. Also in some cases it can lead to slowdown, because the `select_related` would increase the query complexity and/or fetch unwanted data (which might be solved by using `only`).
Any ideas how to improve this?
Antoher option, if you are doing special things inside a derived custom ForeignKeyWidget is to inject a chached queryset when the before_import method fires for the rousource.

`
def before_import(self, dataset, using_transactions, dry_run, **kwargs):
    self.fields['my_column_name'].widget.cached_dataset1 = SomeModel1.objects.all()
    self.fields['my_column_name'].widget.cached_dataset2 = SomeModel2.objects.all()
`

This is also not an optimal solution as it pulls in all data for the cached dataset, which migth be inefficient when they are large models.
A hotfix for now, overriding [`get_export_queryset`](https://django-import-export.readthedocs.io/en/latest/api_admin.html#import_export.admin.ExportMixin.get_export_queryset):

```python
  def get_export_queryset(self, request):
      return (
          super(YourModelAdmin, self)
          .get_export_queryset(request)
          .select_related()
      )
```
@jfilter 's answer works great when you're exporting from the admin. If you'll be exporting from the shell or your views as well, you could override `get_queryset` directly on your `Resource`, which should cover all cases:

```python
def get_queryset(self):
    return super().get_queryset().selected_related('RELATED_MODEL1', 'RELATED_MODEL2', ...)
```

Maybe amending the Getting started section of the docs to suggest users do this if they're doing related lookups could be all that's needed?

Alternatively, an idea could be to mirror ModelAdmin's `list_select_related` in `Resource`, maybe as follows:

```python
class ModelResource(Resource, metaclass=ModelDeclarativeMetaclass):
    ...
    def get_queryset(self):
        """
        Returns a queryset of all objects for this model. Override this if you
        want to limit the returned queryset.
        """
        qs = self._meta.model.objects.all()
        select_related = getattr(self, 'export_select_related`, None)
        if select_related:
            qs = qs.select_related(*select_related)
        return qs

class MyResource(resources.ModelResource):
    export_selected_related = ('RELATED_MODEL1', 'RELATED_MODEL2', ...)
```

Can do a proper PR later if this looks sensible.
@mikaraunio Thanks for this update.  I feel that updating the 'Getting started' docs would be a great way to move this forward.  If you are able to submit a PR for this documentation update that would be appreciated.
See also #1715 
@PetrDlouhy Is now a good time to resolve this issue?
Some steps to reproduce with test app.

### Create a sample dataset (books with authors)

1.  Add the following to `core/admin.py` (`BookResource`)

```python
def before_import_row(self, row, **kwargs):
    author_name = row["author_name"]
    obj, created = Author.objects.get_or_create(name=author_name, defaults={"name": author_name})
    row["author"] = obj.id
```

2.  Import the attached file (1294-books.tsv) via Admin UI

### Test Export

Add the following to `core/settings.py`. (this will log SQL output)

```python
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {"level": "DEBUG", "class": "logging.StreamHandler"},
    },
    "loggers": {
        "django.db.backends": {"level": "DEBUG", "handlers": ["console"]},
        "django.server": {
            "handlers": ["console"],
            "level": "WARNING",
            "propagate": False,
        },
    }
}
```

### Export books

Now run an export of books and note that the SQL output contains a SELECT for each author (and category).  See attached.


[1294-books.tsv.txt](https://github.com/django-import-export/django-import-export/files/13727233/1294-books.tsv.txt)


[sql_output.txt](https://github.com/django-import-export/django-import-export/files/13727239/sql_output.txt)

When the fix in #1717 is applied, the log does not show the queries for Author select.

Adding the following to `BookResource` also prevents the additional Author SELECTs being run:

```python
    def filter_export(self, queryset, **kwargs):
        return queryset.select_related("author")
```


Resolving patch:
diff --git a/import_export/exceptions.py b/import_export/exceptions.py
--- a/import_export/exceptions.py
+++ b/import_export/exceptions.py
@@ -10,6 +10,12 @@ class FieldError(ImportExportError):
     pass
 
 
+class WidgetError(ImportExportError):
+    """Raised when there is a misconfiguration with a Widget."""
+
+    pass
+
+
 class ImportError(ImportExportError):
     def __init__(self, error, number=None, row=None):
         """A wrapper for errors thrown from the import process.
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -1306,7 +1306,11 @@ def field_from_django_field(cls, field_name, django_field, readonly):
         attribute = field_name
         column_name = field_name
         # To solve #974
-        if isinstance(django_field, ForeignKey) and "__" not in column_name:
+        if (
+            isinstance(django_field, ForeignKey)
+            and "__" not in column_name
+            and not cls._meta.use_natural_foreign_keys
+        ):
             attribute += "_id"
             widget_kwargs["key_is_id"] = True
 
diff --git a/import_export/widgets.py b/import_export/widgets.py
--- a/import_export/widgets.py
+++ b/import_export/widgets.py
@@ -14,6 +14,8 @@
 from django.utils.formats import number_format
 from django.utils.translation import gettext_lazy as _
 
+from import_export.exceptions import WidgetError
+
 logger = logging.getLogger(__name__)
 
 
@@ -497,6 +499,10 @@ def __init__(
         self.field = field
         self.key_is_id = key_is_id
         self.use_natural_foreign_keys = use_natural_foreign_keys
+        if use_natural_foreign_keys is True and key_is_id is True:
+            raise WidgetError(
+                _("use_natural_foreign_keys and key_is_id cannot both be True")
+            )
         super().__init__(**kwargs)
 
     def get_queryset(self, value, row, *args, **kwargs):


Test patch:
diff --git a/tests/core/tests/test_resources/test_natural_foreign_key.py b/tests/core/tests/test_resources/test_natural_foreign_key.py
new file mode 100644
--- /dev/null
+++ b/tests/core/tests/test_resources/test_natural_foreign_key.py
@@ -0,0 +1,64 @@
+import tablib
+from core.models import Author, Book
+from django.test import TestCase
+
+from import_export import fields, resources, widgets
+
+
+class BookUsingNaturalKeys(resources.ModelResource):
+    class Meta:
+        model = Book
+        fields = ["name", "author"]
+        use_natural_foreign_keys = True
+
+
+class BookUsingAuthorNaturalKey(resources.ModelResource):
+    class Meta:
+        model = Book
+        fields = ["name", "author"]
+
+    author = fields.Field(
+        attribute="author",
+        column_name="author",
+        widget=widgets.ForeignKeyWidget(
+            Author,
+            use_natural_foreign_keys=True,
+        ),
+    )
+
+
+class TestNaturalKeys(TestCase):
+    """Tests for issue 1816."""
+
+    def setUp(self) -> None:
+        author = Author.objects.create(name="J. R. R. Tolkien")
+        Book.objects.create(author=author, name="The Hobbit")
+        self.expected_dataset = tablib.Dataset(headers=["name", "author"])
+        row = ["The Hobbit", '["J. R. R. Tolkien"]']
+        self.expected_dataset.append(row)
+
+    def test_resource_use_natural_keys(self):
+        """
+        test with ModelResource.Meta.use_natural_foreign_keys=True
+        Reproduces this problem
+        """
+        resource = BookUsingNaturalKeys()
+        exported_dataset = resource.export(Book.objects.all())
+        self.assertDatasetEqual(self.expected_dataset, exported_dataset)
+
+    def test_field_use_natural_keys(self):
+        """
+        test with ModelResource.field.widget.use_natural_foreign_keys=True
+        Example of correct behaviour
+        """
+        resource = BookUsingAuthorNaturalKey()
+        exported_dataset = resource.export(Book.objects.all())
+        self.assertDatasetEqual(self.expected_dataset, exported_dataset)
+
+    def assertDatasetEqual(self, expected_dataset, actual_dataset, message=None):
+        """
+        Util for comparing datasets
+        """
+        self.assertEqual(len(expected_dataset), len(actual_dataset), message)
+        for expected_row, actual_row in zip(expected_dataset, actual_dataset):
+            self.assertEqual(expected_row, actual_row, message)
diff --git a/tests/core/tests/test_widgets.py b/tests/core/tests/test_widgets.py
--- a/tests/core/tests/test_widgets.py
+++ b/tests/core/tests/test_widgets.py
@@ -12,6 +12,7 @@
 from django.utils import timezone
 
 from import_export import widgets
+from import_export.exceptions import WidgetError
 
 
 class WidgetTest(TestCase):
@@ -582,6 +583,16 @@ def test_book_natural_key_render(self):
             json.dumps(self.book.natural_key()),
         )
 
+    def test_natural_foreign_key_with_key_is_id(self):
+        with self.assertRaises(WidgetError) as e:
+            widgets.ForeignKeyWidget(
+                Author, use_natural_foreign_keys=True, key_is_id=True
+            )
+        self.assertEqual(
+            "use_natural_foreign_keys and key_is_id " "cannot both be True",
+            str(e.exception),
+        )
+
 
 class ManyToManyWidget(TestCase, RowDeprecationTestMixin):
     def setUp(self):