Repository: biolab/orange3

Issue:
Distributions: impossible to hide bars and get just smooth kernel density lines
With the updated distributions widget, it seems impossible to get the view that is shown in the documentation:
![image](https://user-images.githubusercontent.com/8712266/67105469-d4daac80-f196-11e9-908a-b1988b879d77.png)

The closest I can get is this but I can't get rid of the bars:
![image](https://user-images.githubusercontent.com/8712266/67105600-0e131c80-f197-11e9-8133-e68b7437cb51.png)

Any suggestion?

I'm using v3.23.1



Discussion:
Distributions widget has been redesigned from scratch. It should probably also be renamed to Histogram. Documentation often lags behind, like here.

You can no longer remove the bars, but this is by design. The widget is a histogram, which can *also* fit some curves to the data. The curves themselves, especially non-parametric curves, can be misleading, so it is better, imho, to keep the histogram visible.
I disagree. Yes, sometimes hiding the bars could be misleading but keeping them makes for an extremely busy and unreadable graph with more than a few variables. The original distribution widget was one of the most useful widget for data exploration in Orange. I used it many times to wow colleagues with the nice graphs it made. Now that I can't remove the bars, I can't really use it anymore for "presentation quality" graphs.

Let me show you a concrete example:

Before (legend edited for confidential reasons):
![image](https://user-images.githubusercontent.com/8712266/67287946-0f4f8c80-f4aa-11e9-8c39-e736e7da43c8.png)

After (the input data is slightly different)
![image](https://user-images.githubusercontent.com/8712266/67288159-710ff680-f4aa-11e9-8c64-9412dc22b7af.png)

The bars here add nothing useful in this case, it's just pollution. 

Also the filled area under the curves added a nice professional touch, I'm sad that it's gone.

One last thing which should probably have its own separate issue is that the legend doesn't work as well anymore. I have an upstream Select Row widget to only keep 7 categories (the 7 lines plotted) but the legend uses all the unfiltered categories.

Your data seems to be time series, so it might be best to use Timeseries add-on for plotting such data. Moreover, I would say histograms look much more informative than distributions.
![plot](https://user-images.githubusercontent.com/12524972/67289452-eb5a6e00-f4de-11e9-8e04-c304d59945a2.png)
Please correct me if I am wrong.

As for the second part - this is just a change in setting. Tick both boxes in Purging and you will see the same results as before. 
Hello,
My data is indeed time based but it's not a time series. The X axis is "Months to failure". It's from analysis of instruments failures in the field. So the time series add-on is not really appropriate here.

I disagree with the statement that histograms are more informative (at least not always). My goal here is to show a general trend, not get exact numbers. It's much easier for the eye to follow a curve than to try to make sense of several vertical bars. Again with my example, which one is easier to understand?

![image](https://user-images.githubusercontent.com/8712266/67317508-2d33e600-f4d8-11e9-8bcd-471fc5768945.png)

or (minus the bars)
![image](https://user-images.githubusercontent.com/8712266/67317467-1b524300-f4d8-11e9-8c13-c2521b2a7d29.png)


Both types have value. Declaring that "bars are better in every single cases therefore they should always be ON" is shortsighted in my opinion. Give the user the flexibility please.

Thanks for the purging pointer. It indeed works.
Filled areas under curves indeed looked nice. I guess we should resuscitate this.

> Again with my example, which one is easier to understand?

None, in my opinion. :)

The problem with bars is that you plotted too many. When I teach data mining, I use this as an example of overfitting. There is not enough data to support this many bars. Decrease the number of bars and the image will become nice and understandable. Your example is not fair.

The problem with curves is that they suggest that, say, the failure times for the blue points are three-modal, which is probably not the case. The problem is the same as with the bars -- the kernel is too narrow for the sample size. (I also use this, just with even narrower kernels, as an example of overfitting -- I later refer to this in lessons on predictive modelling.) You should widen the kernel or, perhaps better, use some parametric curve.

The renewed widget initially didn't even have non-parametric curves because their shapes depended on the kernel width and the user has no principled way to set it. (In classification we have a similar problem, but there we can see how different regularization strengths affect the performance, while here we can't.)

Whatever the curve's shape and hyperparameters, it masks the underlying data and can present a wrong picture. Even with parametric curves, you could choose, say, Gaussian curves when your data is clearly bimodal, and without seeing the bars you would never know it.

I can however agree with your argument about aesthetics. If the curves are done right, they can convey the correct information without the bars. You have a point there.

I'm reopening the issue and adding a "Needs Discussion" tag. This marks it for discussion on next Friday's meeting of the core team.
Good point. You're right, my example wasn't fair, I used the most bars on purpose to convey my point.

Here's the same data with a bin width that makes more sense (v3.23.1)
![opt3](https://user-images.githubusercontent.com/8712266/67348614-7190a780-f513-11e9-9907-75a1b1953b08.png)

And here's how the same dataset looks like with v3.18 (not sure when the change was introduced), which in my opinion conveys the same information but is more aesthetically pleasing and easier to understand:
![opt4](https://user-images.githubusercontent.com/8712266/67348626-79504c00-f513-11e9-8241-d112d19203b2.png)

In any case, thanks for reconsidering my request.
@prdubois, I have good news for you. :)

We'll (probably, almost certainly) add a checkbox for hiding the bars, though we're not yet sure where. UX will be a bit tricky.
What about relative frequency? That's gone as well :(
It's still there and it's called show probabilities. Someone didn't update the documentation yet! 😒 
Thanks Adja, but it's not the same... previously the adjustment was to the size of each group (group by option in the previous version, now, split by), now it just normalizes the whole graph to 1. Or am I missing something?
You can use Box Plot for that.

Also, not Adja, but Ajda.
@ajdapretnar so sorry about the misspelling, Ajda.

Yep, I know about boxplot, but I need binning, which boxplot doesn't do. 

Is there anyway we can reinstall the older distribution widget on the newest version?  

I am almost absolutely sure relative frequency was from 0 to 1 in the previous version. Because that's what relative means. Do you have any example of what you wish to achieve?

No, you cannot use the old version of the widget in the newer Orange.
Two groups, absolute vs relative frequency. This was useful but apparently lost in the latest version update.


<img width="1600" alt="absolute" src="https://user-images.githubusercontent.com/3635823/71138708-3d302380-2204-11ea-9b35-7656e360dd33.png">
<img width="1600" alt="relative" src="https://user-images.githubusercontent.com/3635823/71138716-43260480-2204-11ea-9303-d3e7dc375093.png">


Resolving patch:
diff --git a/Orange/widgets/visualize/owdistributions.py b/Orange/widgets/visualize/owdistributions.py
--- a/Orange/widgets/visualize/owdistributions.py
+++ b/Orange/widgets/visualize/owdistributions.py
@@ -65,7 +65,7 @@ def mouseReleaseEvent(event):
 
 class DistributionBarItem(pg.GraphicsObject):
     def __init__(self, x, width, padding, freqs, colors, stacked, expanded,
-                 tooltip):
+                 tooltip, hidden):
         super().__init__()
         self.x = x
         self.width = width
@@ -77,8 +77,10 @@ def __init__(self, x, width, padding, freqs, colors, stacked, expanded,
         self.__picture = None
         self.polygon = None
         self.hovered = False
+        self._tooltip = tooltip
+        self.hidden = False
+        self.setHidden(hidden)
         self.setAcceptHoverEvents(True)
-        self.setToolTip(tooltip)
 
     def hoverEnterEvent(self, event):
         super().hoverEnterEvent(event)
@@ -90,7 +92,15 @@ def hoverLeaveEvent(self, event):
         self.hovered = False
         self.update()
 
+    def setHidden(self, hidden):
+        self.hidden = hidden
+        if not hidden:
+            self.setToolTip(self._tooltip)
+
     def paint(self, painter, _options, _widget):
+        if self.hidden:
+            return
+
         if self.expanded:
             tot = np.sum(self.freqs)
             if tot == 0:
@@ -264,6 +274,7 @@ class Warning(OWWidget.Warning):
     number_of_bins = settings.ContextSetting(5, schema_only=True)
 
     fitted_distribution = settings.Setting(0)
+    hide_bars = settings.Setting(False)
     show_probs = settings.Setting(False)
     stacked_columns = settings.Setting(False)
     cumulative_distr = settings.Setting(False)
@@ -292,6 +303,7 @@ def __init__(self):
         self.data = None
         self.valid_data = self.valid_group_data = None
         self.bar_items = []
+        self.curve_items = []
         self.curve_descriptions = None
         self.binnings = []
 
@@ -326,8 +338,9 @@ def __init__(self):
             label="Smoothing", orientation=Qt.Horizontal,
             minValue=2, maxValue=20, callback=self.replot)
         gui.checkBox(
-            box, self, "cumulative_distr", "Show cumulative distribution",
-            callback=self.replot)
+            box, self, "hide_bars", "Hide bars", stateWhenDisabled=False,
+            callback=self._on_hide_bars_changed,
+            disabled=not self.fitted_distribution)
 
         box = gui.vBox(self.controlArea, "Columns")
         gui.comboBox(
@@ -341,6 +354,9 @@ def __init__(self):
         gui.checkBox(
             box, self, "show_probs", "Show probabilities",
             callback=self._on_show_probabilities_changed)
+        gui.checkBox(
+            box, self, "cumulative_distr", "Show cumulative distribution",
+            callback=self.replot)
 
         gui.auto_apply(self.controlArea, self, commit=self.apply)
 
@@ -385,7 +401,7 @@ def add_new_plot(zvalue):
 
     def _setup_legend(self):
         self._legend = LegendItem()
-        self._legend.setParentItem(self.plot)
+        self._legend.setParentItem(self.plot_pdf)
         self._legend.hide()
         self._legend.anchor((1, 0), (1, 0))
 
@@ -449,9 +465,16 @@ def _on_bin_slider_released(self):
         self.apply()
 
     def _on_fitted_dist_changed(self):
+        self.controls.hide_bars.setDisabled(not self.fitted_distribution)
         self._set_smoothing_visibility()
         self.replot()
 
+    def _on_hide_bars_changed(self):
+        for bar in self.bar_items:  # pylint: disable=blacklisted-name
+            bar.setHidden(self.hide_bars)
+        self._set_curve_brushes()
+        self.plot.update()
+
     def _set_smoothing_visibility(self):
         self.smoothing_box.setVisible(
             self.Fitters[self.fitted_distribution][1] is AshCurve)
@@ -525,6 +548,7 @@ def _clear_plot(self):
         self.plot_pdf.clear()
         self.plot_mark.clear()
         self.bar_items = []
+        self.curve_items = []
         self._legend.clear()
         self._legend.hide()
 
@@ -564,9 +588,10 @@ def _call_plotting(self):
         self.plot.autoRange()
 
     def _add_bar(self, x, width, padding, freqs, colors, stacked, expanded,
-                 tooltip):
+                 tooltip, hidden=False):
         item = DistributionBarItem(
-            x, width, padding, freqs, colors, stacked, expanded, tooltip)
+            x, width, padding, freqs, colors, stacked, expanded, tooltip,
+            hidden)
         self.plot.addItem(item)
         self.bar_items.append(item)
 
@@ -618,7 +643,8 @@ def _cont_plot(self):
                 f"{freq} ({100 * freq / total:.2f} %)</p>"
             self._add_bar(
                 x0, x1 - x0, 0, [tot_freq if self.cumulative_distr else freq],
-                colors, stacked=False, expanded=False, tooltip=tooltip)
+                colors, stacked=False, expanded=False, tooltip=tooltip,
+                hidden=self.hide_bars)
 
         if self.fitted_distribution:
             self._plot_approximations(
@@ -657,6 +683,7 @@ def _cont_split_plot(self):
             self._add_bar(
                 x0, x1 - x0, 0 if self.stacked_columns else 0.1, plotfreqs,
                 gcolors, stacked=self.stacked_columns, expanded=self.show_probs,
+                hidden=self.hide_bars,
                 tooltip=self._split_tooltip(
                     self.str_int(x0, x1, not i, i == lasti),
                     np.sum(plotfreqs), total, gvalues, plotfreqs))
@@ -734,12 +761,24 @@ def _plot_approximations(self, x0, x1, fitters, colors, prior_probs):
                 tot = (y_p + (tots - y) * (1 - prior_prob))
                 tot[tot == 0] = 1
                 y = y_p / tot
-            plot.addItem(pg.PlotCurveItem(
-                x=x, y=y,
+            curve = pg.PlotCurveItem(
+                x=x, y=y, fillLevel=0,
                 pen=pg.mkPen(width=5, color=color),
-                shadowPen=pg.mkPen(width=8, color=color.darker(120))))
+                shadowPen=pg.mkPen(width=8, color=color.darker(120)))
+            plot.addItem(curve)
+            self.curve_items.append(curve)
         if not show_probs:
             self.plot_pdf.autoRange()
+        self._set_curve_brushes()
+
+    def _set_curve_brushes(self):
+        for curve in self.curve_items:
+            if self.hide_bars:
+                color = curve.opts['pen'].color().lighter(160)
+                color.setAlpha(128)
+                curve.setBrush(pg.mkBrush(color))
+            else:
+                curve.setBrush(None)
 
     @staticmethod
     def _split_tooltip(valname, tot_group, total, gvalues, freqs):


Test patch:
diff --git a/Orange/widgets/visualize/tests/test_owdistributions.py b/Orange/widgets/visualize/tests/test_owdistributions.py
--- a/Orange/widgets/visualize/tests/test_owdistributions.py
+++ b/Orange/widgets/visualize/tests/test_owdistributions.py
@@ -5,7 +5,7 @@
 from unittest.mock import Mock
 
 import numpy as np
-from AnyQt.QtCore import QItemSelection
+from AnyQt.QtCore import QItemSelection, Qt
 
 from Orange.data import Table, Domain, DiscreteVariable
 from Orange.widgets.tests.base import WidgetTest
@@ -461,6 +461,52 @@ def test_selection_grouping(self):
         self.assertEqual(
             len(out_selected.domain[ANNOTATED_DATA_FEATURE_NAME].values), 3)
 
+    def test_disable_hide_bars(self):
+        widget = self.widget
+        self.send_signal(widget.Inputs.data, self.iris)
+        domain = self.iris.domain
+
+        cb = widget.controls.hide_bars
+
+        for var in ("petal length", "iris"):
+            for fitter in (0, 1):
+                self._set_var(domain[var])
+                self._set_fitter(fitter)
+                self.assertEqual(cb.isEnabled(),
+                                 var == "petal length" and fitter == 1)
+
+    def test_hide_bars(self):
+        # bar is a valid name in this context, pylint: disable=blacklisted-name
+        widget = self.widget
+        cb = widget.controls.hide_bars
+        self._set_check(cb, True)
+        self._set_fitter(1)
+
+        self.send_signal(widget.Inputs.data, self.iris)
+
+        domain = self.iris.domain
+        self._set_var(domain["petal length"])
+
+        for cvar in (None, domain["iris"]):
+            self._set_cvar(cvar)
+            self.assertTrue(cb.isEnabled())
+            self.assertTrue(all(bar.hidden
+                                for bar in widget.bar_items))
+            self.assertTrue(all(curve.opts["brush"] is not None
+                                for curve in widget.curve_items))
+
+            self._set_check(cb, False)
+            self.assertTrue(all(not bar.hidden
+                                for bar in widget.bar_items))
+            self.assertTrue(all(curve.opts["brush"].style() == Qt.NoBrush
+                                for curve in widget.curve_items))
+
+            self._set_check(cb, True)
+            self.assertTrue(all(bar.hidden
+                                for bar in widget.bar_items))
+            self.assertTrue(all(curve.opts["brush"] is not None
+                                for curve in widget.curve_items))
+
     def test_report(self):
         """Report doesn't crash"""
         widget = self.widget