Repository: biolab/orange3

Issue:
k-Means: normalization fails when mixing original and bow attributes
**Bug reports that do not sufficiently describe a bug are not helpful and will most likely be ignored.**

- [ ] **A clear and concise description.** "*I'm getting an error when I open a tree viewer*" is far from enough (what kind of error? what kind of data you give it?). "*I'm getting an error*" is even worse.

- [ ] **How to reproduce the problem.** Upload a zip with a workflow and data, or describe the steps (open this widget, click there, then add this ...)

- [ ] **Which version of Orange are you using? How did you install it?**

- [ ] **What is your operating system?**

Orange Version: 3.26.0
Operational System: macOS Mojave Version 10.14.6 - Mac Pro Mid 2010

The same file is working in my other Macbook Pro. 

How can I uninstall Orange completely to reinstall it again? Is there any way to do it? Because just removing and reinstalling the Orange maintain it's structures.

ERROR MESSAGE:

Error encountered in widget k-Means:
```
Traceback (most recent call last):
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/orangecanvas/scheme/signalmanager.py", line 936, in __process_next
    if self.__process_next_helper(use_max_active=True):
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/orangecanvas/scheme/signalmanager.py", line 974, in __process_next_helper
    self.process_node(selected_node)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/orangecanvas/scheme/signalmanager.py", line 605, in process_node
    self.send_to_node(node, signals_in)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/orangewidget/workflow/widgetsscheme.py", line 792, in send_to_node
    self.process_signals_for_widget(node, widget, signals)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/orangewidget/workflow/widgetsscheme.py", line 826, in process_signals_for_widget
    handler(*args)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/utils/sql.py", line 31, in new_f
    return f(widget, data, *args, **kwargs)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/unsupervised/owkmeans.py", line 578, in set_data
    self.invalidate(unconditional=True)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/unsupervised/owkmeans.py", line 458, in invalidate
    self.unconditional_commit()
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/unsupervised/owkmeans.py", line 444, in commit
    self.run_optimization()
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/unsupervised/owkmeans.py", line 403, in run_optimization
    self.__launch_tasks(needed_ks)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/unsupervised/owkmeans.py", line 357, in __launch_tasks
    preprocessed_data = self.preproces(self.data)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/widgets/unsupervised/owkmeans.py", line 493, in preproces
    data = Normalize()(data)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/preprocess/preprocess.py", line 356, in __call__
    return normalizer(data)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/preprocess/normalize.py", line 36, in __call__
    return data.transform(domain)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/data/table.py", line 520, in transform
    return type(self).from_table(domain, self)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/orangecontrib/text/corpus.py", line 508, in from_table
    t = super().from_table(domain, source, row_indices)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/data/table.py", line 463, in from_table
    variables=domain.attributes)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/data/table.py", line 397, in get_columns
    col_array = match_density(col(source))
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/preprocess/transformation.py", line 38, in __call__
    transformed = self.transform(col)
  File "/Applications/Orange3.app/Contents/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Orange/preprocess/transformation.py", line 127, in transform
    raise ValueError('Normalization does not work for sparse data.')
ValueError: Normalization does not work for sparse data.
```

Thank you .

  





Discussion:
First of all, there is a nice issue template at the top, which you are asked to follow. You only managed to answer the template partially. For example, how *did* you install Orange?

Secondly, you remove the packages as one removes the packages on OSX. Not much to do there (I don't know why Apple insists on this silly way of removing packages, but hey, their choice). You can search for Orange files and remove them manually.

Finally, I can't reproduce the issue.
> For example, how _did_ you install Orange?

In this case if follows from the report that Orange is (very very likely) installed as the Orange3.app package dragged into `/Applications` from the dmg.

I would be pleasantly surprised if some users tried and actually managed to build the .app from source. That is something that is hard for me to do. :)
Look, if you are happy dissecting every issue, be my guest. We have prepared the issue template so that it makes the process of analyzing issues easy. It would be nice if people respected that.
@etrauer, deleting the Orange application from your `Applications` completely removes the application, but not Orange's settings. To remove the widget settings, you could go to the Options -> Reset widget settings. To manually remove all Orange's settings, you could remove `~/Library/Application Support/Orange` and `~/Library/Caches/Orange` folders.

If you are working with the exact same (you copied it) workflow and files, both computers should give the same results. If not, it is a bug. To debug it, we need a worfklow (ows file) + data where this is happening. If your data is private, try to replicate the same behaviour with a subset of your data (maybe also anonymise them) and share that. If we can not replicate the bug we can not fix it either.
Thank you for all informations. I deleted Orange from Applications and from Library . But the "Unexpected Error" continues after a new Orange Installation [Orange3-3.26.0-Python37.dmg]
I'm attaching the Excel and Orange file
![Screen Shot 2020-09-07 at 10 36 07](https://user-images.githubusercontent.com/63246010/92393865-eef6e380-f0f6-11ea-83bd-1f40aba22936.png)
![Screen Shot 2020-09-07 at 10 42 11](https://user-images.githubusercontent.com/63246010/92393870-f0281080-f0f6-11ea-8936-e249ab0544d9.png)
[Ideias_campanha_aberta.xlsx](https://github.com/biolab/orange3/files/5183364/Ideias_campanha_aberta.xlsx)

Thank you, now I can reproduce it. The issue occurs when mixing dense and sparse data. I can reproduce this with "election-tweets-2016.tab" and simple BoW - k-Means workflow. I will rename the issue to reflect the actual bug.

Resolving patch:
diff --git a/Orange/widgets/unsupervised/owkmeans.py b/Orange/widgets/unsupervised/owkmeans.py
--- a/Orange/widgets/unsupervised/owkmeans.py
+++ b/Orange/widgets/unsupervised/owkmeans.py
@@ -2,6 +2,8 @@
 from typing import Optional, List, Dict
 
 import numpy as np
+import scipy.sparse as sp
+
 from AnyQt.QtCore import Qt, QTimer, QAbstractTableModel, QModelIndex, QThread, \
     pyqtSlot as Slot
 from AnyQt.QtGui import QIntValidator
@@ -130,6 +132,7 @@ class Warning(widget.OWWidget.Warning):
         not_enough_data = widget.Msg(
             "Too few ({}) unique data instances for {} clusters"
         )
+        no_sparse_normalization = widget.Msg("Sparse data cannot be normalized")
 
     INIT_METHODS = (("Initialize with KMeans++", "k-means++"),
                     ("Random initialization", "random"))
@@ -490,7 +493,10 @@ def select_row(self):
 
     def preproces(self, data):
         if self.normalize:
-            data = Normalize()(data)
+            if sp.issparse(data.X):
+                self.Warning.no_sparse_normalization()
+            else:
+                data = Normalize()(data)
         for preprocessor in KMeans.preprocessors:  # use same preprocessors than
             data = preprocessor(data)
         return data
@@ -570,6 +576,9 @@ def set_data(self, data):
         self.selection = None
         self._set_input_summary()
 
+        self.controls.normalize.setDisabled(
+            bool(self.data) and sp.issparse(self.data.X))
+
         # Do not needlessly recluster the data if X hasn't changed
         if old_data and self.data and array_equal(self.data.X, old_data.X):
             if self.auto_commit:


Test patch:
diff --git a/Orange/widgets/unsupervised/tests/test_owkmeans.py b/Orange/widgets/unsupervised/tests/test_owkmeans.py
--- a/Orange/widgets/unsupervised/tests/test_owkmeans.py
+++ b/Orange/widgets/unsupervised/tests/test_owkmeans.py
@@ -3,6 +3,8 @@
 from unittest.mock import patch, Mock
 
 import numpy as np
+import scipy.sparse as sp
+
 from AnyQt.QtCore import Qt
 from AnyQt.QtWidgets import QRadioButton
 from sklearn.metrics import silhouette_score
@@ -334,6 +336,44 @@ def test_select_best_row(self):
         widget.update_results()
         self.assertEqual(widget.selected_row(), None)
 
+    @patch("Orange.widgets.unsupervised.owkmeans.Normalize")
+    def test_normalize_sparse(self, normalize):
+        normalization = normalize.return_value = Mock(return_value=self.data)
+        widget = self.widget
+        widget.normalize = True
+        norm_check = widget.controls.normalize
+
+        x = sp.csr_matrix(np.random.randint(0, 2, (5, 10)))
+        data = Table.from_numpy(None, x)
+
+        self.send_signal(widget.Inputs.data, data)
+        self.assertTrue(widget.Warning.no_sparse_normalization.is_shown())
+        self.assertFalse(norm_check.isEnabled())
+        normalization.assert_not_called()
+
+        self.send_signal(widget.Inputs.data, None)
+        self.assertFalse(widget.Warning.no_sparse_normalization.is_shown())
+        self.assertTrue(norm_check.isEnabled())
+        normalization.assert_not_called()
+
+        self.send_signal(widget.Inputs.data, data)
+        self.assertTrue(widget.Warning.no_sparse_normalization.is_shown())
+        self.assertFalse(norm_check.isEnabled())
+        normalization.assert_not_called()
+
+        self.send_signal(widget.Inputs.data, self.data)
+        self.assertFalse(widget.Warning.no_sparse_normalization.is_shown())
+        self.assertTrue(norm_check.isEnabled())
+        normalization.assert_called()
+        normalization.reset_mock()
+
+        widget.controls.normalize.click()
+
+        self.send_signal(widget.Inputs.data, data)
+        self.assertFalse(widget.Warning.no_sparse_normalization.is_shown())
+        self.assertFalse(norm_check.isEnabled())
+        normalization.assert_not_called()
+
     def test_report(self):
         widget = self.widget
         widget.k = 4