Repository: django-import-export/django-import-export

Issue:
NOT NULL constraint failed for non-nullable Charfields in xlsx
When using the `.xslx` import, non-nullable CharFields will fail with an exception when their value is left empty. Presumably because the cell has a value of `None` in that case.

[As seen on StackOverflow](https://stackoverflow.com/questions/61987773/django-import-export-how-to-handle-blank-charfield) where a workaround was provided. IMHO this should simply be fixed instead.

**To Reproduce**
Steps to reproduce the behavior:

1. Start the example app
2. Edit a book, make the field "author_email" empty.
3. Export all books as .xlsx
4. Import this .xlsx
5. `sqlite3.IntegrityError: NOT NULL constraint failed: core_book.author_email`

**Versions (please complete the following information):**
 - Django Import Export: [2.8.0]

**Expected behavior**
I expect no error. In particular: I expect that the output of an export is a valid input for an import.



Discussion:
I wonder what the proper fix is for this--how do we know whether an empty cell in Excel is None or an empty string?  I ran into this problem a lot when I was messing with CSV/Excel and for my work project wound up using only YAML and JSON because of this.

Is it fundamentally a problem with how empty strings are sent *to* Excel? Could we maybe have a standard of quoting all strings in Excel, and escaping quotes that are meant to be explicit (e.g. \\") like json does?

I *think* I remember that the issue is that Excel saver interprets "" as an empty cell, which the importer interprets as None? 

Alternatively, maybe we could explicitly write None to empty cells, kinda like the json 'null' keyword? That's a bit hinky though and it would prevent you from having a string of 'None' unless it were quoted, which brings us back to quoting? 

The probelm I think is that we need to have a concept of None / null and empty cells are natural for that.

--

Maybe we could add a Charwidget/Textwidget that are intelligent and introspect the field parameters a little?  Say,

- on export, maintain current behavior
- on import, if the field is Nullabe, treat empty cell as None
- on import, if the field is not Nullable, treat empty cell as ""

I think that might be the most logical approach.  It would leave you with some weird behavior if you had both nulls and blanks. 
For exporting, we have a related discussion [here](https://github.com/django-import-export/django-import-export/pull/1501), namely that `FloatWidget` exports the value, whereas when you need localization support, it needs to export a formatted string.  I'm reticent about changing in that PR because exporting as string not value could affect existing implementations.

Looking at Django Rest Framework, they have a [`coerce_to_string` arg](https://www.django-rest-framework.org/api-guide/fields/#floatfield).  Maybe that is an option here?  It would give the user control on how exports are formatted for Excel.

There is a solution for [this issue on import](https://stackoverflow.com/a/62000383/39296), so maybe it's just a case of improving the documentation to state the issues.  Like you say, how do we know if an empty field is None or ''.
After having a think, I believe it'd be a sensible default to treat an incoming empty field as "" if the field is not nullable and is a char/text field,  and document that (for formats that do not have the concept of null, like csv and excel). 

At least that way django import export would be internally consistent, where we export a null=True field with empty strings to output an empty cell in Excel or an empty cell in Csv (,,), and when it came back in it'd know that those things should be "" if the char/textfield does not allow nulls. 

It's generally considered poor practice to mix nulls and empty strings in Django fields anyway, so that could be considered an edge case ("if you export both nulls and blanks, Excel can't distinguish between these"). 

Moving to a quoting strategy *could* work but it would be superbly tedious, whereas at least handling them consistently would be a step up. 

--

edited to say I double rethought it and there are some risks associated with doing it this way which is that I think we'd be taking away the ability of json and yaml to support text fields that mix nulls and blanks. Don't know if we want to remove functionality. 

bleh. 


Thanks for working through this issue.  

> I think I remember that the issue is that Excel saver interprets "" as an empty cell, which the importer interprets as None?

I tested and in this case the `author_email` is exported as empty string "".  In fact, 'None' values are [always forced to strings](https://github.com/django-import-export/django-import-export/blob/225524546bc46fa49b407bc0a58fe5173e7b5e6d/import_export/fields.py#L123) on export - this has been in the library from the start.

So in the case of Excel, we are exporting as empty string.  In Excel, I couldn't find a way to explicitly set a cell as being an empty string.  I tried setting the cell as type 'Text' and importing, but it is always interpreted as a 'None' (null) value by tablib when importing.  So we have the situation where we always export as empty string, but always get back a None value.

To fix this (rather than having to rely on the users using the workaround) I think we would have to include the logic you describe [here](https://github.com/django-import-export/django-import-export/issues/1485#issuecomment-1295859788).  Namely that the import process has to be more explicit when dealing with non-nullable fields.

Huh, that's very interesting. I thought that the YAML / Json libraries would have nulls, but I think my memory of that must be because I made a nullable field for our internal stuff, and used json/Yaml which support nulls. Which was probably necessary because we have some of those nullable-blankable fields you shouldn't really have. 

I think it would be fairly straightforward to have charfields and textfields check for nullability and interpret Nones as "" if the field is not-nullable. I'll try to pick that up if I get some time here soon, but if not someone's more than welcome to experiment. 

I think the logic for that would need to live in Field.save, since the widgets don't seem to have knowledge of their django fields. 

Interestingly, there is already an explicit setting in Fields for 'saves_null_values' that is checked, so we could just piggy back on that and say "if saves null values is false, then None should be interpreted as an empty string" -- and we might be able to add some autodetection to saves_null_values (maybe in `Resource.import_field`)

Do you think it should start from 3.x? 


If you export to JSON the empty field is rendered as empty string.  If you try to import JSON with a null field it fails with the same issue as above (at least in this context).  So there are probably other undiscovered bugs here.

> I'll try to pick that up if I get some time here soon, but if not someone's more than welcome to experiment.

That would be great - feel free to assign it to yourself if you think you can pick it up.  I think this is OK to go into v3 because it is a fix rather than explicitly changing behaviour.  If it looks like it might have more impact to existing implementations we can always defer to v4.

The Field / Widget logic is maybe a bit tangled.  We don't want CharField specific logic in Field, but also we don't want to couple the widgets module to Django Field implementations (as you say).  If the correct setting of `saves_null_values` was set on the Field instance, that may well solve it.  So that implies that the underlying Django model field would have to be inspected when the import_export Field is instantiated.  I think that is mostly done [here](https://github.com/django-import-export/django-import-export/blob/225524546bc46fa49b407bc0a58fe5173e7b5e6d/import_export/resources.py#L1164), and this method is passed the django field, so it could be fairly simple to add this logic, by using the django field `null` attribute to set `saves_null_values`.

The other issue is that users will often declare the Field themselves in the resource, so they would need to know the implications of this.  They may well have used 'default' kwarg to workaround this issue already.  Probably that's where the improved documentation could come in.  They would maybe need to set `saves_null_values` correctly themselves.

For info, #611 is where `saves_null_values` was added.

Yeah, I think we essentially need to inspect both Null and Blank setting on the django field and then set those values appropriately. Then when saving, we need to interpret that correctly. And we can allow users to explicitly override that at the field level (The way they can say saves_null_values, they could also have saves_blank_values)

My thinking is:

-if blank and null are true, empty cells are ""
-if blank is true and null is false, empty cells are ""
-if blank is false and null is false, empty cells are None, and will fail
-if blank is false and null is true, empty cells are None

I think the things we need to do are:
1. update field constructor to take saves_blank_values with a default of False
2. update field.save to know what to do based on saves null / saves blank
3. update field detection to correctly inspect the django field for saves_blank and saves_null

The problem here is that this reduces a little bit of functionality, which is that dialects that have a clear distinction between "" and null (json, yaml) won't be able to have both blanks and nulls. 

I can't think of any way to get around that except to move things even further up the chain to the importer, but the importer is not aware of the django field. so problematic there. 

It might make some sense long term to think about the widget getting information about the blank/null status tbh. I'll have to think about this some more. 

Resolving patch:
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -15,6 +15,7 @@
 from django.core.management.color import no_style
 from django.core.paginator import Paginator
 from django.db import connections, router
+from django.db.models import fields
 from django.db.models.fields.related import ForeignObjectRel
 from django.db.models.query import QuerySet
 from django.db.transaction import TransactionManagementError, set_rollback
@@ -1292,13 +1293,20 @@ class ArrayField:
         return result
 
     @classmethod
-    def widget_kwargs_for_field(self, field_name):
+    def widget_kwargs_for_field(cls, field_name, django_field):
         """
         Returns widget kwargs for given field_name.
         """
-        if self._meta.widgets:
-            return self._meta.widgets.get(field_name, {})
-        return {}
+        widget_kwargs = {}
+        if cls._meta.widgets:
+            cls_kwargs = cls._meta.widgets.get(field_name, {})
+            widget_kwargs.update(cls_kwargs)
+        if (
+            issubclass(django_field.__class__, fields.CharField)
+            and django_field.blank is True
+        ):
+            widget_kwargs.update({"coerce_to_string": True, "allow_blank": True})
+        return widget_kwargs
 
     @classmethod
     def field_from_django_field(cls, field_name, django_field, readonly):
@@ -1307,7 +1315,7 @@ def field_from_django_field(cls, field_name, django_field, readonly):
         """
 
         FieldWidget = cls.widget_from_django_field(django_field)
-        widget_kwargs = cls.widget_kwargs_for_field(field_name)
+        widget_kwargs = cls.widget_kwargs_for_field(field_name, django_field)
         field = cls.DEFAULT_RESOURCE_FIELD(
             attribute=field_name,
             column_name=field_name,


Test patch:
diff --git a/tests/core/exports/books.json b/tests/core/exports/books.json
new file mode 100644
--- /dev/null
+++ b/tests/core/exports/books.json
@@ -0,0 +1,38 @@
+[
+  {
+    "id": 11,
+    "name": "A Game of Thrones",
+    "author": 11,
+    "author_email": "martin@got.com",
+    "imported": "0",
+    "published": "1996-08-01",
+    "published_time": "21:00:00",
+    "price": "25.00",
+    "added": "",
+    "categories": "1"
+  },
+  {
+    "id": 6,
+    "name": "Circles",
+    "author": 11,
+    "author_email": "geo@met.ry",
+    "imported": "0",
+    "published": "2020-08-01",
+    "published_time": "21:00:00",
+    "price": "15.00",
+    "added": "",
+    "categories": "2"
+  },
+  {
+    "id": 5,
+    "name": "Squares",
+    "author": 5,
+    "author_email": "geo@met.ry",
+    "imported": "0",
+    "published": "1999-08-01",
+    "published_time": "21:00:00",
+    "price": "5.00",
+    "added": "",
+    "categories": "2"
+  }
+]
\ No newline at end of file
diff --git a/tests/core/tests/test_admin_integration.py b/tests/core/tests/test_admin_integration.py
--- a/tests/core/tests/test_admin_integration.py
+++ b/tests/core/tests/test_admin_integration.py
@@ -46,10 +46,10 @@ class AdminTestMixin(object):
 
     def setUp(self):
         super().setUp()
-        user = User.objects.create_user("admin", "admin@example.com", "password")
-        user.is_staff = True
-        user.is_superuser = True
-        user.save()
+        self.user = User.objects.create_user("admin", "admin@example.com", "password")
+        self.user.is_staff = True
+        self.user.is_superuser = True
+        self.user.save()
         self.client.login(username="admin", password="password")
 
     def _do_import_post(
@@ -1119,6 +1119,8 @@ def get_export_filename(self, request, queryset, file_format):
 
 @override_settings(IMPORT_EXPORT_SKIP_ADMIN_CONFIRM=True)
 class TestImportSkipConfirm(AdminTestMixin, TransactionTestCase):
+    fixtures = ["author"]
+
     def _is_str_in_response(
         self,
         filename,
@@ -1170,8 +1172,9 @@ def test_import_action_create(self):
 
     def test_import_action_invalid_date(self):
         # test that a row with an invalid date redirects to errors page
+        index = self._get_input_format_index("csv")
         response = self._do_import_post(
-            self.book_import_url, "books-invalid-date.csv", "0"
+            self.book_import_url, "books-invalid-date.csv", index
         )
         result = response.context["result"]
         # there should be a single invalid row
@@ -1182,15 +1185,11 @@ def test_import_action_invalid_date(self):
         # no rows should be imported because we rollback on validation errors
         self.assertEqual(0, Book.objects.count())
 
-    def test_import_action_empty_author_email(self):
-        xlsx_index = self._get_input_format_index("xlsx")
-        # sqlite / MySQL / Postgres have different error messages
-        self._is_regex_in_response(
-            "books-empty-author-email.xlsx",
-            xlsx_index,
-            follow=True,
-            regex_in_response=r"(NOT NULL|null value in column|cannot be null)",
-        )
+    def test_import_action_error_on_save(self):
+        with mock.patch("core.models.Book.save") as mock_save:
+            mock_save.side_effect = ValueError("some unknown error")
+            response = self._do_import_post(self.book_import_url, "books.csv")
+        self.assertIn("some unknown error", str(response.content))
 
     @override_settings(IMPORT_EXPORT_USE_TRANSACTIONS=True)
     def test_import_transaction_enabled_validation_error(self):
@@ -1208,22 +1207,26 @@ def test_import_transaction_disabled_validation_error(self):
 
     @override_settings(IMPORT_EXPORT_USE_TRANSACTIONS=True)
     def test_import_transaction_enabled_core_error(self):
-        # with transactions enabled, a core error should cause the entire import to fail
-        xlsx_index = self._get_input_format_index("xlsx")
-        self._do_import_post(
-            self.book_import_url, "books-empty-author-email.xlsx", xlsx_index
-        )
+        # test that if we send a file with multiple rows,
+        # and transactions is enabled, a core error means that
+        # no instances are persisted
+        index = self._get_input_format_index("json")
+        with mock.patch("core.admin.BookResource.skip_row") as mock_skip:
+            mock_skip.side_effect = [None, ValueError("some unknown error"), None]
+            response = self._do_import_post(self.book_import_url, "books.json", index)
+        self.assertIn("some unknown error", str(response.content))
         self.assertEqual(0, Book.objects.count())
 
     @override_settings(IMPORT_EXPORT_USE_TRANSACTIONS=False)
     def test_import_transaction_disabled_core_error(self):
-        # with transactions disabled, a core (db contraint) error should not cause the
+        # with transactions disabled, a core (db constraint) error should not cause the
         # entire import to fail
-        xlsx_index = self._get_input_format_index("xlsx")
-        self._do_import_post(
-            self.book_import_url, "books-empty-author-email.xlsx", xlsx_index
-        )
-        self.assertEqual(1, Book.objects.count())
+        index = self._get_input_format_index("json")
+        with mock.patch("core.admin.BookResource.skip_row") as mock_skip:
+            mock_skip.side_effect = [None, ValueError("some unknown error"), None]
+            response = self._do_import_post(self.book_import_url, "books.json", index)
+        self.assertIn("some unknown error", str(response.content))
+        self.assertEqual(2, Book.objects.count())
 
     def test_import_action_mac(self):
         self._is_str_in_response(
@@ -1267,12 +1270,10 @@ class MockModelAdmin(ImportExportMixinBase, ModelAdmin):
     change_list_template = "admin/import_export/change_list.html"
 
 
-class TestChangeListView(TestCase):
+class TestChangeListView(AdminTestMixin, TestCase):
     def setUp(self):
+        super().setUp()
         self.factory = RequestFactory()
-        self.user = User.objects.create_superuser(
-            username="testuser", password="password", email="test@example.com"
-        )
         self.model_admin = MockModelAdmin(User, admin.site)
 
     def test_changelist_view_context(self):
diff --git a/tests/core/tests/test_resources.py b/tests/core/tests/test_resources.py
--- a/tests/core/tests/test_resources.py
+++ b/tests/core/tests/test_resources.py
@@ -1108,6 +1108,29 @@ class Meta:
         self.assertIn(cat1, book.categories.all())
         self.assertIn(cat2, book.categories.all())
 
+    def test_import_null_django_CharField_saved_as_empty_string(self):
+        # issue 1485
+        resource = BookResource()
+        self.assertTrue(resource._meta.model.author_email.field.blank)
+        self.assertFalse(resource._meta.model.author_email.field.null)
+        headers = ["id", "author_email"]
+        row = [1, None]
+        dataset = tablib.Dataset(row, headers=headers)
+        resource.import_data(dataset, raise_errors=True)
+        book = Book.objects.get(id=1)
+        self.assertEqual("", book.author_email)
+
+    def test_import_empty_django_CharField_saved_as_empty_string(self):
+        resource = BookResource()
+        self.assertTrue(resource._meta.model.author_email.field.blank)
+        self.assertFalse(resource._meta.model.author_email.field.null)
+        headers = ["id", "author_email"]
+        row = [1, ""]
+        dataset = tablib.Dataset(row, headers=headers)
+        resource.import_data(dataset, raise_errors=True)
+        book = Book.objects.get(id=1)
+        self.assertEqual("", book.author_email)
+
     def test_m2m_add(self):
         cat1 = Category.objects.create(name="Cat 1")
         cat2 = Category.objects.create(name="Cat 2")