Repository: biolab/orange3

Issue:
Strip empty columns in .xlsx files
<!--
This is an issue template. Please fill in the relevant details in the
sections below.
-->

Hi,
I am new to orange and I get the error while importing my data. Hope you could help me.
##### Orange version

<!-- From menu _Help→About→Version_ or code `Orange.version.full_version` -->

3.3.6 (on windows 7)
##### Expected bahavior

Dataset has 61 columns and 338 rows. It is read as a XLSX file but not as a CSV file
##### Actual behavior

Gives the error: "Cannot parse dataset libro1.csv: too many indices for array".
##### Steps to reproduce the behavior
##### Additional info (worksheets, data, screenshots, ...)

[Libro1.xlsx](https://github.com/biolab/orange3/files/371787/Libro1.xlsx)



Discussion:
In Orange 3.3.7~dev, I can read this xlsx, either as-is or transformed into CSV, just fine. Can you attach the exact CSV file that fails to load?

Here is the file. I tried again and failed either from its location in a file server or from the desktop. Thanks in advance.
[AllData_20160719_pca.zip](https://github.com/biolab/orange3/files/373052/AllData_20160719_pca.zip)

@mlopezheredia Your .csv file is a bit 'corrupted'. It has extra blank columns with no headers and this is what's causing the problem. If you import your file in Excel as text, you will see the issue. Once I've copied only the actual data and saved it as a .csv file, the data was loaded.

However, I think Orange should somehow strip empty columns automatically. What do you think, @kernc? 

@ajdapretnar  Thanks for the information and help. I have not seen them. It would be a good idea to strip empty columns automatically.

Removing the extra blank columns with no headers from the .csv file solved the issue. Thanks @ajdapretnar for the tip.

Reading the file works in the pandas implementation. However, it leaves empty columns in the data. The Purge Domain widget is useful in thhis case.

I vote for fixing this before pandas and striping empty columns in pandas, too. Thus a gsoc label.

Resolving patch:
diff --git a/Orange/data/io.py b/Orange/data/io.py
--- a/Orange/data/io.py
+++ b/Orange/data/io.py
@@ -618,8 +618,15 @@ def data_table(cls, data, headers=None):
         # Determine maximum row length
         rowlen = max(map(len, (names, types, flags)))
 
+        strip = False
+
         def _equal_length(lst):
-            lst.extend(['']*(rowlen - len(lst)))
+            nonlocal strip
+            if len(lst) > rowlen > 0:
+                lst = lst[:rowlen]
+                strip = True
+            elif len(lst) < rowlen:
+                lst.extend(['']*(rowlen - len(lst)))
             return lst
 
         # Ensure all data is of equal width in a column-contiguous array
@@ -627,6 +634,9 @@ def _equal_length(lst):
                 for row in data if any(row)]
         data = np.array(data, dtype=object, order='F')
 
+        if strip:
+            warnings.warn("Columns with no headers were removed.")
+
         # Data may actually be longer than headers were
         try:
             rowlen = data.shape[1]


Test patch:
diff --git a/Orange/tests/test_io.py b/Orange/tests/test_io.py
--- a/Orange/tests/test_io.py
+++ b/Orange/tests/test_io.py
@@ -5,6 +5,7 @@
 import os
 import tempfile
 import shutil
+import io
 
 from Orange.data import ContinuousVariable
 from Orange.data.io import FileFormat, TabReader, CSVReader, PickleReader, \
@@ -104,6 +105,20 @@ def test_open_bad_pickle(self):
         with unittest.mock.patch("pickle.load", return_value=None):
             self.assertRaises(TypeError, reader.read, "foo")
 
+    def test_empty_columns(self):
+        """Can't read files with more columns then headers. GH-1417"""
+        samplefile = """\
+        a, b
+        1, 0,
+        1, 2,
+        """
+        c = io.StringIO(samplefile)
+        with self.assertWarns(UserWarning) as cm:
+            table = CSVReader(c).read()
+        self.assertEqual(len(table.domain.attributes), 2)
+        self.assertEqual(cm.warning.args[0],
+                         "Columns with no headers were removed.")
+
 
 class TestIo(unittest.TestCase):
     def test_sanitize_variable_deprecated_params(self):