Repository: django-import-export/django-import-export

Issue:
Csv files should be opened with universal new line mode
When attempting to open a file saved on a windows computer import-export throw an error:

Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

It looks like U needs to be added to the open call - see http://stackoverflow.com/questions/17315635/csv-new-line-character-seen-in-unquoted-field-error



Discussion:
Thanks for report. We do open csv as 'rU' but only for PY3. Here is a link to code: 

https://github.com/bmihelac/django-import-export/blob/master/import_export/formats/base_formats.py#L110

To be honest I am not sure why is binary used for python 2.X here. This should be inspected carefully so we do not fix one issue and open another.

Is there a work around for 2.x? Any recommendations?

Hi bmihelac, has any progress been made with this issue?

Many thanks,
a python 2.x user

Any word on this?

I'm using Python 2.7.5 & cannot work with Excel files due to this error. I essentially have to have clients email me their CSV, save as CSV in Open Office & then do their import for them.

Yes there should be an option to support universal (and frankly, I'd have thought that should've been the default option)

@litchfield thanks for contribution.

I am thinking of treating CSV as text and not binary python2. I am not sure why it is opened as binary and cannot find any documentation regarding this.

Also I do not think there is needed additional setting as this should be a bugfix. Do you agree?

This would be achieved by removing following lines:

https://github.com/django-import-export/django-import-export/blob/master/import_export/formats/base_formats.py#L128-L133

I just need to make csv with office on windows to make sure it does not break anything.

Cool, yes a fix would be even better @bmihelac 


Resolving patch:
diff --git a/import_export/formats/base_formats.py b/import_export/formats/base_formats.py
--- a/import_export/formats/base_formats.py
+++ b/import_export/formats/base_formats.py
@@ -119,19 +119,10 @@ def is_binary(self):
         return False
 
 
-class CSV(TablibFormat):
-    """
-    CSV is treated as binary in Python 2.
-    """
+class CSV(TextFormat):
     TABLIB_MODULE = 'tablib.formats._csv'
     CONTENT_TYPE = 'text/csv'
 
-    def get_read_mode(self):
-        return 'rU' if six.PY3 else 'rb'
-
-    def is_binary(self):
-        return False if six.PY3 else True
-
 
 class JSON(TextFormat):
     TABLIB_MODULE = 'tablib.formats._json'


Test patch:
diff --git a/tests/core/exports/books-dos.csv b/tests/core/exports/books-dos.csv
new file mode 100644
--- /dev/null
+++ b/tests/core/exports/books-dos.csv
@@ -0,0 +1,2 @@
+id,name,author_email
+1,Some book,test@example.com
diff --git a/tests/core/tests/base_formats_tests.py b/tests/core/tests/base_formats_tests.py
--- a/tests/core/tests/base_formats_tests.py
+++ b/tests/core/tests/base_formats_tests.py
@@ -1,7 +1,8 @@
 from __future__ import unicode_literals
 
+import os
+
 from django.test import TestCase
-from django.utils import six
 
 from import_export.formats import base_formats
 
@@ -14,5 +15,15 @@ def test_binary_format(self):
 
 class CSVTest(TestCase):
 
-    def test_binary_format(self):
-        self.assertEqual(base_formats.CSV().is_binary(), not six.PY3)
+    def setUp(self):
+        self.format = base_formats.CSV()
+
+    def test_import_dos(self):
+        filename = os.path.join(
+            os.path.dirname(__file__),
+            os.path.pardir,
+            'exports',
+            'books-dos.csv')
+        in_stream = open(filename, self.format.get_read_mode()).read()
+        expected = 'id,name,author_email\n1,Some book,test@example.com\n'
+        self.assertEqual(in_stream, expected)