Repository: biolab/orange3

Issue:
Table.concatenate(): Concatenating sparse and dense tables
**Describe the bug**

`Table.concatenate()` now pretends to be able to concatenate sparse tables with dense, but after concatenating, the output seems to fail elsewhere.

**To Reproduce**

Corpus - Bag of Words - Concatenate. (sparse)
Corpus - Concatenate. (dense)
Add Data Table after Concatenate.

**Orange version:**
3.26.0.dev

**Expected behavior**
All dense tables are transformed to sparse matrix and concatenated.



Discussion:
> All dense tables are transformed to sparse matrix and concatenated.

Or the opposite? If a small sparse table is attached to a large dense one, I would hope to get a dense table.

Or should `concatenate` have a parameter whose value can be `Dense`, `Sparse`, and `Auto`, which would be used in the mixed case, and where Auto would construct whichever is smaller?
Sparse.
> If a small sparse table is attached to a large dense one, I would hope to get a dense table.

I suppose so. That would make sense.
@ajdapretnar, @markotoplak opposed "auto". His argument for "sparse" was that if you create dense instead sparse, the memory requirement will grow immensely, while if you create sparse instead of dense, the memory grows by a factor of `(sizeof(int) + sizeof(double)) / sizeof(double) = (4 + 8) / 8 = 1.5`.
I think you're wrong. It seems to me that Bag of Words output a CSC and the Corpus widget outputs a CSR matrix, not a dense one!

If any table is sparse, `Table.concatenate` calls `sp.vstack`, which decides the matrix format. `sp.vstack` does the best it can and creates a COO. The Table widget (and many others) then fails when indexing it, that is, in `text.corpus.Corpus.__getitem__`.

The problem cannot be fixed in `Table.concatenate`, because it is no smarter than `vstack`. The problem is that `from_numpy` allows coo matrices, but doesn't implement `__getitem__` that would be able to handle them.

We could prohibit `coo_matrix` in `from_numpy`. But this wouldn't solve anything - your workflow would still fail.

Possible solutions:

A. `from_numpy` converts `coo_matrix` to `csr_matrix`. Row operations are only efficient on csr anyway. We keep csc, though, because some add-ons might need them. Coos ... are not useful for Table.

B. We fix row Indexing for `coo_matrix`. It would be very slow, though. Besides, `__getitem__` may not be the only operation that fails.

Comments, please!
A. Let's only support csr and csc for the Table.
And `Table.from_numpy` (silently) converts all other formats, right?

Resolving patch:
diff --git a/Orange/data/table.py b/Orange/data/table.py
--- a/Orange/data/table.py
+++ b/Orange/data/table.py
@@ -1751,6 +1751,8 @@ def ninstances(array):
                              % (ninstances(array), shape_1))
 
         if sp.issparse(array):
+            if not (sp.isspmatrix_csr(array) or sp.isspmatrix_csc(array)):
+                array = array.tocsr()
             array.data = np.asarray(array.data)
             has_inf = _check_inf(array.data)
         else:


Test patch:
diff --git a/Orange/data/tests/test_table.py b/Orange/data/tests/test_table.py
--- a/Orange/data/tests/test_table.py
+++ b/Orange/data/tests/test_table.py
@@ -1,6 +1,7 @@
 import unittest
 
 import numpy as np
+import scipy.sparse as sp
 
 from Orange.data import (
     ContinuousVariable, DiscreteVariable, StringVariable,
@@ -85,6 +86,28 @@ def test_from_numpy(self):
             self.assertRaises(ValueError, func, dom, X, Y, metas[:4])
             self.assertRaises(ValueError, func, dom, X, Y[:4])
 
+    def test_from_numpy_sparse(self):
+        domain = Domain([ContinuousVariable(c) for c in "abc"])
+        x = np.arange(12).reshape(4, 3)
+
+        t = Table.from_numpy(domain, x, None, None)
+        self.assertFalse(sp.issparse(t.X))
+
+        t = Table.from_numpy(domain, sp.csr_matrix(x))
+        self.assertTrue(sp.isspmatrix_csr(t.X))
+
+        t = Table.from_numpy(domain, sp.csc_matrix(x))
+        self.assertTrue(sp.isspmatrix_csc(t.X))
+
+        t = Table.from_numpy(domain, sp.coo_matrix(x))
+        self.assertTrue(sp.isspmatrix_csr(t.X))
+
+        t = Table.from_numpy(domain, sp.lil_matrix(x))
+        self.assertTrue(sp.isspmatrix_csr(t.X))
+
+        t = Table.from_numpy(domain, sp.bsr_matrix(x))
+        self.assertTrue(sp.isspmatrix_csr(t.X))
+
 
 class TestTableFilters(unittest.TestCase):
     def setUp(self):