Repository: biolab/orange3

Issue:
"Statistics" output of Feature statistics is inconsistent

**What's wrong?**
For categorical variables, Feature Statistics determines the Mode as the central tendency and puts it in the Median column. That's OK with me, although it could be described a bit better in the help popup. A separate Mode column would even be better.
But if I want to put the feature statistics in a table (or save them in a file) the categories in the Median column are converted to integers. This is inconsistent, and unclear because the mapping from a category to the corresponding number isn't obvious. 

**How can we reproduce the problem?** 
See attached zip file and screenshot

**What's your environment?**
- Operating system: Mac OS 12.6 on Silicon
- Orange version: 3.32.0
- How you installed Orange: from DMG
[Feature statistics inconsistent output.ows.zip](https://github.com/biolab/orange3/files/9674384/Feature.statistics.inconsistent.output.ows.zip)
<img width="634" alt="Screen Shot 2022-09-29 at 13 54 27" src="https://user-images.githubusercontent.com/55989717/193026461-88b5fac2-c18f-4a67-87a9-7d8d95e5946e.png">



Discussion:
The name of the column shown in the widget should be renamed to something like Median/Majority.

The problem with the output is the type of the column because it mixes values from different variables. When showing it in a widget, we can print out anything. But in a table, each column must have a certain type. To be of any use for numeric variables, it must be a number. But even if all variables are categorical, the column cannot be a categorical variable because each of original variables can have different categories. Hence, it must be a textual column and therefore a meta variable.

I think the summary of our today's discussion is:
- The output will have a column Median, but the value for categorical variables will be undefined (`?`).
- We also add a column Majority (only if there are any categorical variables?) with majority values as texts.
- I believe that it could be useful to have a categorical variable with majority, but only when all categorical variables have the same categories (typically: yes or no, like Zoo without variable legs). This could be in parallel with, or instead of, the Majority in metas. Yet I wasn't able to convince others, so this bullet serves as a note to myself so that I don't forget we also had this discussion. :)
Majority can also be useful for numerical variables, especially if the number of different values is limited. In particular this can be the case when the values are integers (e.g., Likert scale). No problem if the values would be converted to text - for further processing there's always Edit Domain.

Resolving patch:
diff --git a/Orange/widgets/data/owfeaturestatistics.py b/Orange/widgets/data/owfeaturestatistics.py
--- a/Orange/widgets/data/owfeaturestatistics.py
+++ b/Orange/widgets/data/owfeaturestatistics.py
@@ -290,52 +290,45 @@ def __mode(x, *args, **kwargs):
             time_f=lambda x: ut.nanmedian(x, axis=0),
         )
 
-    def get_statistics_matrix(self, variables=None, return_labels=False):
-        """Get the numeric computed statistics in a single matrix. Optionally,
-        we can specify for which variables we want the stats. Also, we can get
-        the string column names as labels if desired.
-
-        Parameters
-        ----------
-        variables : Iterable[Union[Variable, int, str]]
-            Return statistics for only the variables specified. Accepts all
-            formats supported by `domain.index`
-        return_labels : bool
-            In addition to the statistics matrix, also return string labels for
-            the columns of the matrix e.g. 'Mean' or 'Dispersion', as specified
-            in `Columns`.
-
-        Returns
-        -------
-        Union[Tuple[List[str], np.ndarray], np.ndarray]
-
-        """
-        if self.table is None:
-            return np.atleast_2d([])
+    def get_statistics_table(self):
+        """Get the numeric computed statistics in a single matrix."""
+        if self.table is None or not self.rowCount():
+            return None
 
-        # If a list of variables is given, select only corresponding stats
-        # variables can be a list or array, pylint: disable=len-as-condition
-        if variables is not None and len(variables) != 0:
-            indices = [self.domain.index(var) for var in variables]
+        # don't match TimeVariable, pylint: disable=unidiomatic-typecheck
+        contivars = [type(var) is ContinuousVariable for var in self.variables]
+        if any(contivars):
+            def c(column):
+                return np.choose(contivars, [np.nan, column])
+
+            x = np.vstack((
+                c(self._center), c(self._median), self._dispersion,
+                c(self._min), c(self._max), self._missing,
+            )).T
+            attrs = [ContinuousVariable(column.name) for column in (
+                self.Columns.CENTER, self.Columns.MEDIAN,
+                self.Columns.DISPERSION,
+                self.Columns.MIN, self.Columns.MAX, self.Columns.MISSING)]
         else:
-            indices = ...
-
-        matrix = np.vstack((
-            self._center[indices], self._median[indices],
-            self._dispersion[indices],
-            self._min[indices], self._max[indices], self._missing[indices],
-        )).T
-
-        # Return string labels for the returned matrix columns e.g. 'Mean',
-        # 'Dispersion' if requested
-        if return_labels:
-            labels = [self.Columns.CENTER.name, self.Columns.MEDIAN.name,
-                      self.Columns.DISPERSION.name,
-                      self.Columns.MIN.name, self.Columns.MAX.name,
-                      self.Columns.MISSING.name]
-            return labels, matrix
+            x = np.vstack((self._dispersion, self._missing)).T
+            attrs = [ContinuousVariable(name)
+                     for name in ("Entropy", self.Columns.MISSING.name)]
+
+        names = [var.name for var in self.variables]
+        if any(isinstance(var, DiscreteVariable) for var in self.variables):
+            majorities = [
+                var.str_val(val) if isinstance(var, DiscreteVariable) else ""
+                for var, val in zip(self.variables, self._median)]
+            metas = np.vstack((names, majorities)).T
+            meta_attrs = [StringVariable('Feature'), StringVariable('Mode')]
+        else:
+            metas = np.atleast_2d(names).T
+            meta_attrs = [StringVariable('Feature')]
 
-        return matrix
+        domain = Domain(attributes=attrs, metas=meta_attrs)
+        statistics = Table.from_numpy(domain, x, metas=metas)
+        statistics.name = f'{self.table.name} (Feature Statistics)'
+        return statistics
 
     def __compute_stat(self, matrices, discrete_f=None, continuous_f=None,
                        time_f=None, string_f=None, default_val=np.nan):
@@ -866,14 +859,7 @@ def commit(self):
             return
 
         # Send the statistics of the selected variables to ouput
-        labels, data = self.model.get_statistics_matrix(return_labels=True)
-        var_names = np.atleast_2d([var.name for var in self.model.variables]).T
-        domain = Domain(
-            attributes=[ContinuousVariable(name) for name in labels],
-            metas=[StringVariable('Feature')]
-        )
-        statistics = Table(domain, data, metas=var_names)
-        statistics.name = '{self.data.name} (Feature Statistics)'
+        statistics = self.model.get_statistics_table()
         self.Outputs.statistics.send(statistics)
 
     def send_report(self):


Test patch:
diff --git a/Orange/widgets/data/tests/test_owfeaturestatistics.py b/Orange/widgets/data/tests/test_owfeaturestatistics.py
--- a/Orange/widgets/data/tests/test_owfeaturestatistics.py
+++ b/Orange/widgets/data/tests/test_owfeaturestatistics.py
@@ -449,6 +449,57 @@ def test_clearing_selection_sends_none_to_output(self):
         self.widget.unconditional_commit()
         self.assertIsNone(self.get_output(self.widget.Outputs.reduced_data))
 
+    def test_output_statistics(self):
+        self.widget.auto_commit = True
+
+        data = make_table([continuous_full, continuous_missing,
+                           rgb_full, rgb_missing])
+        self.send_signal(self.widget.Inputs.data, data)
+        output = self.get_output(self.widget.Outputs.statistics)
+        np.testing.assert_almost_equal(
+            output.X,
+            [[2, 2, 0.7071068, 0, 4, 0],
+             [1.75, 1.5, 0.8451543, 0, 4, 1],
+             [np.nan, np.nan, 0.9502705, np.nan, np.nan, 0],
+             [np.nan, np.nan, 1.0397208, np.nan, np.nan, 1]],
+        )
+        np.testing.assert_equal(
+            output.metas,
+            [["continuous_full", ""],
+             ["continuous_missing", ""],
+             ["rgb_full", "g"],
+             ["rgb_missing", "g"]])
+
+        data = make_table([continuous_full, continuous_missing])
+        self.send_signal(self.widget.Inputs.data, data)
+        output = self.get_output(self.widget.Outputs.statistics)
+        np.testing.assert_almost_equal(
+            output.X,
+            [[2, 2, 0.7071068, 0, 4, 0],
+             [1.75, 1.5, 0.8451543, 0, 4, 1]]
+        )
+        np.testing.assert_equal(
+            output.metas,
+            [["continuous_full"],
+             ["continuous_missing"]])
+
+        data = make_table([rgb_full, rgb_missing])
+        self.send_signal(self.widget.Inputs.data, data)
+        output = self.get_output(self.widget.Outputs.statistics)
+        np.testing.assert_almost_equal(
+            output.X,
+            [[0.9502705, 0],
+             [1.0397208, 1]],
+        )
+        np.testing.assert_equal(
+            output.metas,
+            [["rgb_full", "g"],
+             ["rgb_missing", "g"]])
+
+        self.send_signal(self.widget.Inputs.data, None)
+        output = self.get_output(self.widget.Outputs.statistics)
+        self.assertIsNone(output)
+
     def test_output_combinations(self):
         # No selection -> reduced_data is not output, statistics is present
         self.widget.unconditional_commit()