Repository: biolab/orange3

Issue:
Box Plot: sparse variables are ranked wrong
<!-- 
Thanks for taking the time to report a bug!
If you're raising an issue about an add-on (i.e., installed via Options > Add-ons), raise an issue in the relevant add-on's issue tracker instead. See: https://github.com/biolab?q=orange3
To fix the bug, we need to be able to reproduce it. Please answer the following questions to the best of your ability.
-->


**What's wrong?**
<!-- Be specific, clear, and concise. Include screenshots if relevant. -->
<!-- If you're getting an error message, copy it, and enclose it with three backticks (```). -->
When analyzing topic distinctions with Box Plot, one would select a categorical variable to group by and use Order by relevance to subgroups to rank the variables by how well they split between categorical values.

In text mining it can happen (but I guess elsewhere too), that there is only one document in a specified group, so Box Plot cannot compute significance, which is fine.

The bug is that this variable is placed before one where significance can be computed. Is this expected?
A bunch of variables where all instances belong to one group are also placed very high. I, of course, expect to see these variables in a list, but probably there could be a checkbox (gosh, another one?) which would hide such variables? Or do we need a separate widget for text mining?


**How can we reproduce the problem?** 
<!-- Upload a zip with the .ows file and data. -->
<!-- Describe the steps (open this widget, click there, then add this...) -->
Corpus (book-excerpts) - Preprocess Text (default) - Bag of Words (uncheck Hide bow attributes) - Box Plot. Use Category as Subgroup, then observe the ranked order of variables.




**What's your environment?**
<!-- To find your Orange version, see "Help → About → Version" or `Orange.version.full_version` in code -->
- Operating system: OSX
- Orange version: 3.30.2dev
- How you installed Orange: pip

Box Plot: crash on numeric target
<!-- 
Thanks for taking the time to report a bug!
If you're raising an issue about an add-on (i.e., installed via Options > Add-ons), raise an issue in the relevant add-on's issue tracker instead. See: https://github.com/biolab?q=orange3
To fix the bug, we need to be able to reproduce it. Please answer the following questions to the best of your ability.
-->


**What's wrong?**
<!-- Be specific, clear, and concise. Include screenshots if relevant. -->
<!-- If you're getting an error message, copy it, and enclose it with three backticks (```). -->
Box Plot crashes when given a numeric target and split by categorical variable.

```
Traceback (most recent call last):
  File "/Users/ajda/orange/orange3/Orange/widgets/visualize/owboxplot.py", line 485, in attr_changed
    self._variables_changed(self.apply_group_sorting)
  File "/Users/ajda/orange/orange3/Orange/widgets/visualize/owboxplot.py", line 489, in _variables_changed
    self.compute_box_data()
  File "/Users/ajda/orange/orange3/Orange/widgets/visualize/owboxplot.py", line 541, in compute_box_data
    self.conts = contingency.get_contingency(
  File "/Users/ajda/orange/orange3/Orange/statistics/contingency.py", line 308, in get_contingency
    return Continuous(
  File "/Users/ajda/orange/orange3/Orange/statistics/contingency.py", line 205, in __init__
    self.from_data(dat, col_variable, row_variable)
  File "/Users/ajda/orange/orange3/Orange/statistics/contingency.py", line 231, in from_data
    conts = data._compute_contingency([col_variable], row_variable)
  File "/Users/ajda/orange/orange3/Orange/data/table.py", line 2065, in _compute_contingency
    col_data, W_, classes_ = arr[:, arr_i], W_, row_data
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
```


**How can we reproduce the problem?** 
<!-- Upload a zip with the .ows file and data. -->
<!-- Describe the steps (open this widget, click there, then add this...) -->
File (heart-disease). Set Age as target (remove Diameter Narrowing as target!). Box Plot (split by gender, select Age).




**What's your environment?**
<!-- To find your Orange version, see "Help → About → Version" or `Orange.version.full_version` in code -->
- Operating system: OSX
- Orange version: 3.31.dev
- How you installed Orange: pip



Discussion:
<img width="949" alt="Screen Shot 2021-11-22 at 14 29 17" src="https://user-images.githubusercontent.com/12524972/142869949-1f14b43c-eb06-48c1-8cfa-1ca6309df6db.png">


Upon inspection, it turns out there are two underlying issues:

1.) Sparse columns are not treated correctly. Absence of words is represented as nan, which is not considered in Box Plot when computing averages. Nans should be treated as 0 in this case.
Example: 4 documents out of 5 in category A have the word "orange", while only 1 document out of 3 in category B have this word. Box Plot reports "At least one group has just one instance", even though category B has 3 documents. The average should be 4/5 to 1/3.

2.) When there's indeed a case with one category containing just one instance, these should be ranked correctly. Now they appear in somewhat random order.


Resolving patch:
diff --git a/Orange/widgets/visualize/owboxplot.py b/Orange/widgets/visualize/owboxplot.py
--- a/Orange/widgets/visualize/owboxplot.py
+++ b/Orange/widgets/visualize/owboxplot.py
@@ -46,60 +46,27 @@ def compute_scale(min_, max_):
     return first_val, step
 
 
-def _quantiles(a, freq, q, interpolation="midpoint"):
-    """
-    Somewhat like np.quantiles, but with explicit sample frequencies.
-
-    * Only 'higher', 'lower' and 'midpoint' interpolation.
-    * `a` MUST be sorted.
-    """
-    a = np.asarray(a)
-    freq = np.asarray(freq)
-    assert a.size > 0 and a.size == freq.size
-    cumdist = np.cumsum(freq)
-    cumdist /= cumdist[-1]
-
-    if interpolation == "midpoint":  # R quantile(..., type=2)
-        left = np.searchsorted(cumdist, q, side="left")
-        right = np.searchsorted(cumdist, q, side="right")
-        # no mid point for the right most position
-        np.clip(right, 0, a.size - 1, out=right)
-        # right and left will be different only on the `q` boundaries
-        # (excluding the right most sample)
-        return (a[left] + a[right]) / 2
-    elif interpolation == "higher":  # R quantile(... type=1)
-        right = np.searchsorted(cumdist, q, side="right")
-        np.clip(right, 0, a.size - 1, out=right)
-        return a[right]
-    elif interpolation == "lower":
-        left = np.searchsorted(cumdist, q, side="left")
-        return a[left]
-    else:  # pragma: no cover
-        raise ValueError("invalid interpolation: '{}'".format(interpolation))
-
-
 ContDataRange = namedtuple("ContDataRange", ["low", "high", "group_value"])
 DiscDataRange = namedtuple("DiscDataRange", ["value", "group_value"])
 
 
 class BoxData:
-    def __init__(self, dist, group_val=None):
-        self.dist = dist
-        self.n = n = np.sum(dist[1])
-        if n == 0:
+    def __init__(self, col, group_val=None):
+        self.n = len(col) - np.sum(np.isnan(col))
+        if self.n == 0:
             return
-        self.a_min = float(dist[0, 0])
-        self.a_max = float(dist[0, -1])
-        self.mean = float(np.sum(dist[0] * dist[1]) / n)
-        self.var = float(np.sum(dist[1] * (dist[0] - self.mean) ** 2) / n)
-        self.dev = math.sqrt(self.var)
-        a, freq = np.asarray(dist)
-        q25, median, q75 = _quantiles(a, freq, [0.25, 0.5, 0.75])
-        self.median = median
-        # The code below omits the q25 or q75 in the plot when they are None
-        self.q25 = None if q25 == median else q25
-        self.q75 = None if q75 == median else q75
-        self.data_range = ContDataRange(q25, q75, group_val)
+        self.a_min = np.nanmin(col)
+        self.a_max = np.nanmax(col)
+        self.mean = np.nanmean(col)
+        self.var = np.nanvar(col)
+        self.dev = np.sqrt(self.var)
+        self.q25, self.median, self.q75 = \
+            np.nanquantile(col, [0.25, 0.5, 0.75], interpolation="midpoint")
+        self.data_range = ContDataRange(self.q25, self.q75, group_val)
+        if self.q25 == self.median:
+            self.q25 = None
+        if self.q75 == self.median:
+            self.q75 = None
 
 
 class FilterGraphicsRectItem(QGraphicsRectItem):
@@ -388,7 +355,7 @@ def compute_score(attr):
                 col = data.get_column_view(attr)[0].astype(float)
                 groups = (col[group_col == i] for i in range(n_groups))
                 groups = (col[~np.isnan(col)] for col in groups)
-                groups = [group for group in groups if len(group)]
+                groups = [group for group in groups if len(group) > 1]
                 p = f_oneway(*groups)[1] if len(groups) > 1 else 2
             else:
                 p = self._chi_square(group_var, attr)[1]
@@ -421,11 +388,8 @@ def compute_stat(group):
             if group is None:
                 return -1
             if attr.is_continuous:
-                group_col = data.get_column_view(group)[0].astype(float)
-                groups = (attr_col[group_col == i]
-                          for i in range(len(group.values)))
-                groups = (col[~np.isnan(col)] for col in groups)
-                groups = [group for group in groups if len(group)]
+                groups = self._group_cols(data, group, attr_col)
+                groups = [group for group in groups if len(group) > 1]
                 p = f_oneway(*groups)[1] if len(groups) > 1 else 2
             else:
                 p = self._chi_square(group, attr)[1]
@@ -525,6 +489,16 @@ def select_box_items(self):
             if isinstance(box, FilterGraphicsRectItem):
                 box.setSelected(box.data_range in selection)
 
+    def _group_cols(self, data, group, attr):
+        if isinstance(attr, np.ndarray):
+            attr_col = attr
+        else:
+            attr_col = data.get_column_view(group)[0].astype(float)
+        group_col = data.get_column_view(group)[0].astype(float)
+        groups = [attr_col[group_col == i] for i in range(len(group.values))]
+        groups = [col[~np.isnan(col)] for col in groups]
+        return groups
+
     def compute_box_data(self):
         attr = self.attribute
         if not attr:
@@ -538,29 +512,33 @@ def compute_box_data(self):
             return
         if self.group_var:
             self.dist = None
-            self.conts = contingency.get_contingency(
-                dataset, attr, self.group_var)
             missing_val_str = f"missing '{self.group_var.name}'"
             group_var_labels = self.group_var.values + ("",)
             if self.attribute.is_continuous:
                 stats, label_texts = [], []
-                for cont, value in zip(self.conts.array_with_unknowns,
-                                       group_var_labels):
-                    if np.sum(cont[1]):
-                        stats.append(BoxData(cont, value))
+                attr_col = dataset.get_column_view(attr)[0].astype(float)
+                for group, value in \
+                        zip(self._group_cols(dataset, self.group_var, attr_col),
+                            group_var_labels):
+                    if group.size:
+                        stats.append(BoxData(group, value))
                         label_texts.append(value or missing_val_str)
                 self.stats = stats
                 self.label_txts_all = label_texts
             else:
+                self.conts = contingency.get_contingency(
+                    dataset, attr, self.group_var)
                 self.label_txts_all = [
                     v or missing_val_str for v, c in zip(
                         group_var_labels, self.conts.array_with_unknowns)
                     if np.sum(c) > 0]
         else:
-            self.dist = distribution.get_distribution(dataset, attr)
             self.conts = None
             if self.attribute.is_continuous:
-                self.stats = [BoxData(self.dist, None)]
+                attr_col = dataset.get_column_view(attr)[0].astype(float)
+                self.stats = [BoxData(attr_col)]
+            else:
+                self.dist = distribution.get_distribution(dataset, attr)
             self.label_txts_all = [""]
         self.label_txts = [txts for stat, txts in zip(self.stats,
                                                       self.label_txts_all)
@@ -876,7 +854,7 @@ def mean_label(self, stat, attr, val_name):
     def draw_axis(self):
         """Draw the horizontal axis and sets self.scale_x"""
         misssing_stats = not self.stats
-        stats = self.stats or [BoxData(np.array([[0.], [1.]]), self.attribute)]
+        stats = self.stats or [BoxData(np.array([0.]), self.attribute)]
         mean_labels = self.mean_labels or [self.mean_label(stats[0], self.attribute, "")]
         bottom = min(stat.a_min for stat in stats)
         top = max(stat.a_max for stat in stats)


Test patch:
diff --git a/Orange/widgets/visualize/tests/test_owboxplot.py b/Orange/widgets/visualize/tests/test_owboxplot.py
--- a/Orange/widgets/visualize/tests/test_owboxplot.py
+++ b/Orange/widgets/visualize/tests/test_owboxplot.py
@@ -8,9 +8,8 @@
 
 from Orange.data import Table, ContinuousVariable, StringVariable, Domain, \
     DiscreteVariable
-from Orange.widgets.visualize.owboxplot import (
-    OWBoxPlot, FilterGraphicsRectItem, _quantiles
-)
+from Orange.widgets.visualize.owboxplot import OWBoxPlot, FilterGraphicsRectItem
+
 from Orange.widgets.tests.base import WidgetTest, WidgetOutputsTestMixin
 from Orange.tests import test_filename
 
@@ -281,8 +280,8 @@ def _select_data(self):
                  if isinstance(item, FilterGraphicsRectItem)]
         items[0].setSelected(True)
         return [100, 103, 104, 108, 110, 111, 112, 115, 116,
-                120, 123, 124, 126, 128, 132, 133, 136, 137,
-                139, 140, 141, 143, 144, 145, 146, 147, 148]
+                120, 123, 124, 128, 132, 133, 136, 137, 139,
+                140, 141, 143, 144, 145, 146, 147]
 
     def _select_list_items(self, _list):
         for name in _list.model().sourceModel():
@@ -360,42 +359,5 @@ def test_valid_data_range(self):
                 box.setSelected(True)
 
 
-class TestUtils(unittest.TestCase):
-    def test(self):
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 8 + 1), [1.] * 8, [0.0, 0.25, 0.5, 0.75, 1.0]),
-            [1., 2.5, 4.5, 6.5, 8.]
-        )
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 8 + 1), [1.] * 8, [0.0, 0.25, 0.5, 0.75, 1.0]),
-            [1., 2.5, 4.5, 6.5, 8.]
-        )
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 4 + 1), [1., 2., 1., 2],
-                       [0.0, 0.25, 0.5, 0.75, 1.0]),
-            [1.0, 2.0, 2.5, 4.0, 4.0]
-        )
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 4 + 1), [2., 1., 1., 2.],
-                       [0.0, 0.25, 0.5, 0.75, 1.0]),
-            [1.0, 1.0, 2.5, 4.0, 4.0]
-        )
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 4 + 1), [1., 1., 1., 1.],
-                       [0.0, 0.25, 0.5, 0.75, 1.0]),
-            [1.0, 1.5, 2.5, 3.5, 4.0]
-        )
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 4 + 1), [1., 1., 1., 1.],
-                       [0.0, 0.25, 0.5, 0.75, 1.0], interpolation="higher"),
-            [1, 2, 3, 4, 4]
-        )
-        np.testing.assert_array_equal(
-            _quantiles(range(1, 4 + 1), [1., 1., 1., 1.],
-                       [0.0, 0.25, 0.5, 0.75, 1.0], interpolation="lower"),
-            [1, 1, 2, 3, 4]
-        )
-
-
 if __name__ == '__main__':
     unittest.main()