internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: django-import-export/django-import-export

Issue:
can't compare offset-naive and offset-aware datetimes
I tried to import a json file that was just exported.

settings.py contains USE_TZ=True

Line number: 1 - can't compare offset-naive and offset-aware datetimes
Traceback (most recent call last):
File "/apps/local/lib/python2.7/site-packages/import_export/resources.py", line 361, in import_data
if self.skip_row(instance, original):
File "/apps/local/lib/python2.7/site-packages/import_export/resources.py", line 266, in skip_row
if field.get_value(instance) != field.get_value(original):
TypeError: can't compare offset-naive and offset-aware datetimes

Same with CSV



Discussion:

Resolving patch:
diff --git a/import_export/widgets.py b/import_export/widgets.py
--- a/import_export/widgets.py
+++ b/import_export/widgets.py
@@ -2,7 +2,7 @@
 
 from decimal import Decimal
 from datetime import datetime
-from django.utils import datetime_safe
+from django.utils import datetime_safe, timezone
 from django.utils.encoding import smart_text
 from django.conf import settings
 
@@ -149,7 +149,13 @@ def clean(self, value):
             return None
         for format in self.formats:
             try:
-                return datetime.strptime(value, format)
+                dt = datetime.strptime(value, format)
+                if settings.USE_TZ:
+                    # make datetime timezone aware so we don't compare
+                    # naive datetime to an aware one
+                    dt = timezone.make_aware(dt,
+                                             timezone.get_default_timezone())
+                return dt
             except (ValueError, TypeError):
                 continue
         raise ValueError("Enter a valid date/time.")


Test patch:
diff --git a/tests/core/tests/widgets_tests.py b/tests/core/tests/widgets_tests.py
--- a/tests/core/tests/widgets_tests.py
+++ b/tests/core/tests/widgets_tests.py
@@ -4,7 +4,9 @@
 from decimal import Decimal
 from datetime import date, datetime
 
+from django.test.utils import override_settings
 from django.test import TestCase
+from django.utils import timezone
 
 from import_export import widgets
 
@@ -43,6 +45,11 @@ def test_render_none(self):
     def test_clean(self):
         self.assertEqual(self.widget.clean("13.08.2012"), self.date)
 
+    @override_settings(USE_TZ=True)
+    def test_use_tz(self):
+        self.assertEqual(self.widget.render(self.date), "13.08.2012")
+        self.assertEqual(self.widget.clean("13.08.2012"), self.date)
+
 
 class DateTimeWidgetTest(TestCase):
 
@@ -61,6 +68,15 @@ def test_clean(self):
         self.assertEqual(self.widget.clean("13.08.2012 18:00:00"),
                          self.datetime)
 
+    @override_settings(USE_TZ=True)
+    def test_use_tz(self):
+        self.assertEqual(self.widget.render(self.datetime),
+                         "13.08.2012 18:00:00")
+        aware_dt = timezone.make_aware(self.datetime,
+                                       timezone.get_default_timezone())
+        self.assertEqual(self.widget.clean("13.08.2012 18:00:00"),
+                         aware_dt)
+
 
 class DateWidgetBefore1900Test(TestCase):
Repository: django-import-export/django-import-export

Issue:
BigIntegerField doesn't get recongnized as IntegerWidget
[`ModelResource.widget_from_django_field`](https://github.com/django-import-export/django-import-export/blob/95722b924741dd7b2bfc4a461066cd662d3b209a/import_export/resources.py#L548-L550) only reconizes `IntegerField`, `PositiveIntegerField`, `PositiveSmallIntegerField`, `SmallIntegerField` and `AutoField` as integers.
This means that djangos [`BigIntegerField`](https://docs.djangoproject.com/en/1.8/ref/models/fields/#bigintegerfield) doesn't the `IntegerWidget` automatically, so it has to be manually.

Is there anything wrong with just adding `BigIntegerField` to the list?



Discussion:
Hi @luto, I don't see any reason why this should not be added. Do you want to open a pull request?


Resolving patch:
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -545,7 +545,7 @@ def widget_from_django_field(cls, f, default=widgets.Widget):
             result = widgets.DateTimeWidget
         elif internal_type in ('DateField', ):
             result = widgets.DateWidget
-        elif internal_type in ('IntegerField', 'PositiveIntegerField',
+        elif internal_type in ('IntegerField', 'PositiveIntegerField', 'BigIntegerField',
                 'PositiveSmallIntegerField', 'SmallIntegerField', 'AutoField'):
             result = widgets.IntegerWidget
         elif internal_type in ('BooleanField', 'NullBooleanField'):


Test patch:
Repository: django-import-export/django-import-export

Issue:
Show errors on bad file format user friendly
If I import XLSX file as XLS or CP1250 text file as CSV the application shows general server error (in non-debug mode). The import-export should inform the user more specifically, what went wrong (at least show, that the file format is wrong).



Discussion:

Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -216,10 +216,15 @@ def import_action(self, request, *args, **kwargs):
 
             # then read the file, using the proper format-specific mode
             # warning, big files may exceed memory
-            data = tmp_storage.read(input_format.get_read_mode())
-            if not input_format.is_binary() and self.from_encoding:
-                data = force_text(data, self.from_encoding)
-            dataset = input_format.create_dataset(data)
+            try:
+                data = tmp_storage.read(input_format.get_read_mode())
+                if not input_format.is_binary() and self.from_encoding:
+                    data = force_text(data, self.from_encoding)
+                dataset = input_format.create_dataset(data)
+            except UnicodeDecodeError as e:
+                return HttpResponse(_(u"<h1>Imported file is not in unicode: %s</h1>" % e))
+            except Exception as e:
+                return HttpResponse(_(u"<h1>%s encountred while trying to read file: %s</h1>" % (type(e).__name__, e)))
             result = resource.import_data(dataset, dry_run=True,
                                           raise_errors=False,
                                           file_name=import_file.name,


Test patch:
Repository: django-import-export/django-import-export

Issue:
RemovedInDjango110Warning
I am getting this error:

WARNING:py.warnings:/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/import_export/admin.py:247: RemovedInDjango110Warning: The current_app argument of TemplateResponse is deprecated. Set the current_app attribute of its request instead.
  context, current_app=self.admin_site.name)



Discussion:

Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -249,8 +249,9 @@ def import_action(self, request, *args, **kwargs):
         context['opts'] = self.model._meta
         context['fields'] = [f.column_name for f in resource.get_fields()]
 
+        request.current_app = self.admin_site.name
         return TemplateResponse(request, [self.import_template_name],
-                                context, current_app=self.admin_site.name)
+                                context)
 
 
 class ExportMixin(ImportExportMixinBase):
@@ -365,8 +366,9 @@ def export_action(self, request, *args, **kwargs):
 
         context['form'] = form
         context['opts'] = self.model._meta
+        request.current_app = self.admin_site.name
         return TemplateResponse(request, [self.export_template_name],
-                                context, current_app=self.admin_site.name)
+                                context)
 
 
 class ImportExportMixin(ImportMixin, ExportMixin):
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -489,7 +489,10 @@ def __new__(cls, name, bases, attrs):
                         verbose_path = ".".join([opts.model.__name__] + attrs[0:i+1])
 
                         try:
-                            f = model._meta.get_field_by_name(attr)[0]
+                            if VERSION >= (1, 8):
+                                f = model._meta.get_field(attr)
+                            else:
+                                f = model._meta.get_field_by_name(attr)[0]
                         except FieldDoesNotExist as e:
                             logging.exception(e)
                             raise FieldDoesNotExist(


Test patch:
diff --git a/tests/settings.py b/tests/settings.py
--- a/tests/settings.py
+++ b/tests/settings.py
@@ -33,6 +33,17 @@
     'django.middleware.clickjacking.XFrameOptionsMiddleware',
 )
 
+TEMPLATES = [
+    {
+        'BACKEND': 'django.template.backends.django.DjangoTemplates',
+        'DIRS': [],
+        'APP_DIRS': True,
+        'OPTIONS': {
+            'context_processors': {'django.contrib.messages.context_processors.messages',},
+        },
+    },
+]
+
 if os.environ.get('IMPORT_EXPORT_TEST_TYPE') == 'mysql-innodb':
     IMPORT_EXPORT_USE_TRANSACTIONS = True
     DATABASES = {
Repository: django-import-export/django-import-export

Issue:
Better result reporting
At the moment `import_data()` is a monolithic function with no easy customization points. In order to provide better feedback to users, I would like to extend it to report the total number of rows in each result type (NEW, UPDATE, SKIP, DELETE), including the total number of skipped rows even when `report_skipped == False`

There is no easy way to do this without copying the whole `import_data()` function into the subclass because there is no way to override Result class used, or the handling of individual rows.

I could achieve my requirements by making a slight change to import_data() to count the total number of each result type.

If it is desirable, I could also:
- replace the `result = Result()` with `result = self.get_result_class()()` to allow people to customize the Result if necessary
- move the inner part of the `for row in dataset.dict:` to a new `import_row()` method that would allow people to customize what happens before and after the row import is attempted
- add a `report_error()` and a `report_row_result()` to hold the code that appends the results to the result, once again providing an additional extension point for people that need additional reporting



Discussion:
@rhunwicks that looks sensible to me,  it wouldn't break anything and it would be easier to customise importing process


Resolving patch:
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -502,7 +502,7 @@ def import_data(self, dataset, dry_run=False, raise_errors=False,
                 if raise_errors:
                     if use_transactions:
                         savepoint_rollback(sp1)
-                    six.reraise(*sys.exc_info())
+                    raise row_result.errors[-1].error
             else:
                 result.totals[row_result.import_type] += 1
             if (row_result.import_type != RowResult.IMPORT_TYPE_SKIP or


Test patch:
Repository: django-import-export/django-import-export

Issue:
User made available for import but not export?
I've noticed that a `user` kwarg is populated (from the Django `request`) when `import_data()` gets called, but this is not done for exports. Is there any particular reason why? I am doing an export and need the user details so I can trim some fields out, and I'm happy to add this functionality, but I wanted to check first that there wasn't a reason. Thanks!



Discussion:
No, there is no particular reason. I would like to have this functionality.


Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -339,12 +339,12 @@ def get_export_queryset(self, request):
         except AttributeError:
             return cl.query_set
 
-    def get_export_data(self, file_format, queryset):
+    def get_export_data(self, file_format, queryset, *args, **kwargs):
         """
         Returns file_format representation for given queryset.
         """
         resource_class = self.get_export_resource_class()
-        data = resource_class().export(queryset)
+        data = resource_class().export(queryset, *args, **kwargs)
         export_data = file_format.export_data(data)
         return export_data
 
@@ -357,7 +357,7 @@ def export_action(self, request, *args, **kwargs):
             ]()
 
             queryset = self.get_export_queryset(request)
-            export_data = self.get_export_data(file_format, queryset)
+            export_data = self.get_export_data(file_format, queryset, user=request.user)
             content_type = file_format.get_content_type()
             # Django 1.7 uses the content_type kwarg instead of mimetype
             try:
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -549,7 +549,7 @@ def get_export_headers(self):
             force_text(field.column_name) for field in self.get_fields()]
         return headers
 
-    def export(self, queryset=None):
+    def export(self, queryset=None, *args, **kwargs):
         """
         Exports a resource.
         """


Test patch:
Repository: django-import-export/django-import-export

Issue:
ImportMixin.process_import does not handle exceptions and can return None
Most easily reproduced by not using POST method when the whole point of that view is POSTing data.

This is obviously not the solution, but to prevent regular Exceptions, a quick workaround in import_export/admin.py is possible:

```
+from django.views.decorators.http import require_POST

@@ -110,7 +111,7 @@ def get_urls(self):
         my_urls = [
             url(r'^process_import/$',
-                self.admin_site.admin_view(self.process_import),
+                require_POST(self.admin_site.admin_view(self.process_import)),
                 name='%s_%s_process_import' % info),
```



Discussion:
Nice catch, would you like to write a PR for require_POST?

I guess not handling exception in  `process_import` is ok, as exception would be logged/mailed to admin.


Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -16,6 +16,8 @@
 from django.core.urlresolvers import reverse
 from django.conf import settings
 from django.template.defaultfilters import pluralize
+from django.utils.decorators import method_decorator
+from django.views.decorators.http import require_POST
 
 from .forms import (
     ImportForm,
@@ -142,6 +144,7 @@ def get_import_formats(self):
         """
         return [f for f in self.formats if f().can_import()]
 
+    @method_decorator(require_POST)
     def process_import(self, request, *args, **kwargs):
         '''
         Perform the actual import action (after the user has confirmed he
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -352,7 +352,7 @@ def get_diff(self, original_fields, new, current_fields, dry_run=False):
         """
         data = []
         dmp = diff_match_patch()
-        for v1, v2 in itertools.izip(original_fields, current_fields):
+        for v1, v2 in six.moves.zip(original_fields, current_fields):
             if v1 != v2 and new:
                 v1 = ""
             diff = dmp.diff_main(force_text(v1), force_text(v2))


Test patch:
diff --git a/tests/core/tests/admin_integration_tests.py b/tests/core/tests/admin_integration_tests.py
--- a/tests/core/tests/admin_integration_tests.py
+++ b/tests/core/tests/admin_integration_tests.py
@@ -147,6 +147,10 @@ def test_import_file_name_in_tempdir(self):
         with self.assertRaises(IOError):
             self.client.post('/admin/core/book/process_import/', data)
 
+    def test_csrf(self):
+        response = self.client.get('/admin/core/book/process_import/')
+        self.assertEqual(response.status_code, 405)
+
     def test_import_log_entry(self):
         input_format = '0'
         filename = os.path.join(
Repository: django-import-export/django-import-export

Issue:
Does not set {{ title }} context variable
The default `django/contrib/admin/templates/admin/base_site.html`, is used by the page that allows you to select the page format - looks like this is `export.html`. It has:

```
{% block title %}{{ title }} | {{ site_title|default:_('Django site admin') }}{% endblock %}
```

Unfortunately, this project does not supply a value of title to this template, resulting in incorrect output or (in my case) fatal errors as I have configured Django to generate a fatal error if a template variable is not defined.



Discussion:
Hi @brianmay! Thanks for noticing. Would you like to add a PR for this?

My time is restricted, however I should be able to do something. First, just to confirm, what branch should I be working from? Master?


Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -266,6 +266,7 @@ def import_action(self, request, *args, **kwargs):
         elif django.VERSION >= (1, 7, 0):
             context.update(self.admin_site.each_context())
 
+        context['title'] = _("Import")
         context['form'] = form
         context['opts'] = self.model._meta
         context['fields'] = [f.column_name for f in resource.get_user_visible_fields()]
@@ -394,6 +395,7 @@ def export_action(self, request, *args, **kwargs):
         elif django.VERSION >= (1, 7, 0):
             context.update(self.admin_site.each_context())
 
+        context['title'] = _("Export")
         context['form'] = form
         context['opts'] = self.model._meta
         request.current_app = self.admin_site.name


Test patch:
Repository: django-import-export/django-import-export

Issue:
Can no longer customize diffs
Prior to 64db01cf85f85f002a0ae80a5af4f6e2e6cc0abe it was possible to customize the way that the diff was presented for a `Resource` by overriding `get_diff()`. That commit replaced `get_diff()` with a hard-coded `Diff()` class.

One particular problem we have with `Diff()` is that because it forces an export representation of all the fields it can cause Django to issue additional queries to create the diff when the importing foreign keys, and that ruins the import performance.

We'd like to be able to disable the diff in those situations.

I propose that we replace the hard-coded `Diff` with a `get_diff_class` which defaults to returning `Diff`. Then we can customize as required.

I will make a PR.


Discussion:

Resolving patch:
diff --git a/import_export/resources.py b/import_export/resources.py
--- a/import_export/resources.py
+++ b/import_export/resources.py
@@ -213,6 +213,13 @@ def get_error_result_class(self):
         """
         return Error
 
+    @classmethod
+    def get_diff_class(self):
+        """
+        Returns the class used to display the diff for an imported instance.
+        """
+        return Diff
+
     def get_use_transactions(self):
         if self._meta.use_transactions is None:
             return USE_TRANSACTIONS
@@ -444,7 +451,7 @@ def import_row(self, row, instance_loader, using_transactions=True, dry_run=Fals
                 row_result.import_type = RowResult.IMPORT_TYPE_UPDATE
             row_result.new_record = new
             original = deepcopy(instance)
-            diff = Diff(self, original, new)
+            diff = self.get_diff_class()(self, original, new)
             if self.for_delete(row, instance):
                 if new:
                     row_result.import_type = RowResult.IMPORT_TYPE_SKIP


Test patch:
Repository: django-import-export/django-import-export

Issue:
Last day isn't included in results queryset for export
Hi.
In case of export when one of filters is a datetime, behavior of export will not be compatible with data displayed inside admin. As I said in subject - last day is always missing in the queryset.

PR link: https://github.com/django-import-export/django-import-export/pull/714
  


Discussion:

Resolving patch:
diff --git a/import_export/admin.py b/import_export/admin.py
--- a/import_export/admin.py
+++ b/import_export/admin.py
@@ -360,19 +360,19 @@ def get_export_queryset(self, request):
 
         Default implementation respects applied search and filters.
         """
-        # copied from django/contrib/admin/options.py
         list_display = self.get_list_display(request)
         list_display_links = self.get_list_display_links(request, list_display)
+        list_filter = self.get_list_filter(request)
+        search_fields = self.get_search_fields(request)
 
         ChangeList = self.get_changelist(request)
         cl = ChangeList(request, self.model, list_display,
-                        list_display_links, self.list_filter,
-                        self.date_hierarchy, self.search_fields,
-                        self.list_select_related, self.list_per_page,
-                        self.list_max_show_all, self.list_editable,
-                        self)
+            list_display_links, list_filter, self.date_hierarchy,
+            search_fields, self.list_select_related, self.list_per_page,
+            self.list_max_show_all, self.list_editable, self
+        )
 
-        return cl.queryset
+        return cl.get_queryset(request)
 
     def get_export_data(self, file_format, queryset, *args, **kwargs):
         """


Test patch:
Repository: django-import-export/django-import-export

Issue:
DecimalWidget should be initialized from text
As [per doc](https://docs.python.org/3/library/decimal.html):

```python
>>> Decimal('3.14')
Decimal('3.14')
>>> Decimal(3.14)
Decimal('3.140000000000000124344978758017532527446746826171875')
```

When I've changed this line:
https://github.com/django-import-export/django-import-export/blob/3cf5e3f9796a5caf7c5ea3928119af7ce4706c0d/import_export/widgets.py#L88

to

```python
        return Decimal(force_text(value))
```

the import of value `1.4` changed from:

1.~~40~~399999999999999911182158029987476766109466552734375

to:

1.4~~0~~


Discussion:

Resolving patch:
diff --git a/import_export/widgets.py b/import_export/widgets.py
--- a/import_export/widgets.py
+++ b/import_export/widgets.py
@@ -85,7 +85,7 @@ class DecimalWidget(NumberWidget):
     def clean(self, value, row=None, *args, **kwargs):
         if self.is_empty(value):
             return None
-        return Decimal(value)
+        return Decimal(force_text(value))
 
 
 class CharWidget(Widget):


Test patch:
diff --git a/tests/core/tests/test_widgets.py b/tests/core/tests/test_widgets.py
--- a/tests/core/tests/test_widgets.py
+++ b/tests/core/tests/test_widgets.py
@@ -145,6 +145,7 @@ def setUp(self):
 
     def test_clean(self):
         self.assertEqual(self.widget.clean("11.111"), self.value)
+        self.assertEqual(self.widget.clean(11.111), self.value)
 
     def test_render(self):
         self.assertEqual(self.widget.render(self.value), self.value)