Repository: biolab/orange3

Issue:
Impute: impute with fixed value for all features
<!-- 
Thanks for taking the time to submit a feature request!
For the best chance at our team considering your request, please answer the following questions to the best of your ability.
-->

- [ ] **What's your use case?**
<!-- In other words, what's your pain point? -->
<!-- Is your request related to a problem, or perhaps a frustration? -->
<!-- Tell us the story that led you to write this request. -->
I want to impute all missing values to 0. I don't want to do it feature by feature.




- [ ] **What's your proposed solution?**
<!-- Be specific, clear, and concise. -->
Add an option to impute a specified value for all features. Custom-defined option is already available at single feature level, just add it to all features at once.




- [ ] **Are there any alternative solutions?**
Python Script.







Discussion:
Two designs (ignore the misalignments in the second). If the uses chooses this option, categorical values would be left missing (unless imputed below).

I guess I prefer the first, because it's more compact. Empty space in the second is ugly.

<img width="603" alt="Screenshot 2020-11-13 at 12 20 36" src="https://user-images.githubusercontent.com/2387315/99068290-86bbf680-25ac-11eb-9752-be75c8aca666.png">

<img width="603" alt="Screenshot 2020-11-13 at 12 33 51" src="https://user-images.githubusercontent.com/2387315/99068295-891e5080-25ac-11eb-9b70-3561bab1fae5.png">

Go with the first one. I'd change the wording to 'Fixed values' (maybe also in individual attribute settings). Also, I'd put the three labels/lineedits into their own box.

The lineedits should be disabled if Values is not chosen. Although, perhaps it would look better if the box was hidden when Values is not chosen? 
Wording: yes, Fixed values is better.

Disabling line edits: it's an option. But I also like UXs in which you can start editing the line edit and the corresponding line edit is automatically selected. As for hiding, we used to avoid it when it changes the total size of the widget.

I didn't understand the idea about boxes. Any of those? I think there are just too many boxes then.

<img width="603" alt="Screenshot 2020-11-13 at 12 59 30" src="https://user-images.githubusercontent.com/2387315/99070546-bb31b180-25b0-11eb-8242-d7ea6ce7993e.png">

<img width="603" alt="Screenshot 2020-11-13 at 12 59 30" src="https://user-images.githubusercontent.com/2387315/99070564-c4bb1980-25b0-11eb-87b9-d8fbc2fda44e.png">

One box with three Labels/LineEdits in it.

I see your point about hiding, agreed. Regarding disabling, 'Fixed values' being automatically selected when editing a LineEdit is an option, I'm just worried it'll look disjoint (it won't be visually obvious that the LineEdits are related to the Fixed values RadioButton).
This widget currently does not impute string values. Should it? Does anybody consider this useful? (Hint: I hope for a "no".)
No.

You are welcome.
I owe you one at your next PR.

Resolving patch:
diff --git a/Orange/data/io.py b/Orange/data/io.py
--- a/Orange/data/io.py
+++ b/Orange/data/io.py
@@ -270,27 +270,6 @@ def read(self):
             raise IOError("Couldn't load spreadsheet from " + self.filename)
         return table
 
-    @classmethod
-    def write_file(cls, filename, data):
-        vars = list(chain((ContinuousVariable('_w'),) if data.has_weights() else (),
-                          data.domain.attributes,
-                          data.domain.class_vars,
-                          data.domain.metas))
-        formatters = [cls.formatter(v) for v in vars]
-        zipped_list_data = zip(data.W if data.W.ndim > 1 else data.W[:, np.newaxis],
-                               data.X,
-                               data.Y if data.Y.ndim > 1 else data.Y[:, np.newaxis],
-                               data.metas)
-        headers = cls.header_names(data)
-        workbook = xlsxwriter.Workbook(filename)
-        sheet = workbook.add_worksheet()
-        for c, header in enumerate(headers):
-            sheet.write(0, c, header)
-        for i, row in enumerate(zipped_list_data, 1):
-            for j, (fmt, v) in enumerate(zip(formatters, flatten(row))):
-                sheet.write(i, j, fmt(v))
-        workbook.close()
-
 
 class ExcelReader(_BaseExcelReader):
     """Reader for .xlsx files"""
@@ -332,6 +311,27 @@ def _get_active_sheet(self) -> openpyxl.worksheet.worksheet.Worksheet:
         else:
             return self.workbook.active
 
+    @classmethod
+    def write_file(cls, filename, data):
+        vars = list(chain((ContinuousVariable('_w'),) if data.has_weights() else (),
+                          data.domain.attributes,
+                          data.domain.class_vars,
+                          data.domain.metas))
+        formatters = [cls.formatter(v) for v in vars]
+        zipped_list_data = zip(data.W if data.W.ndim > 1 else data.W[:, np.newaxis],
+                               data.X,
+                               data.Y if data.Y.ndim > 1 else data.Y[:, np.newaxis],
+                               data.metas)
+        headers = cls.header_names(data)
+        workbook = xlsxwriter.Workbook(filename)
+        sheet = workbook.add_worksheet()
+        for c, header in enumerate(headers):
+            sheet.write(0, c, header)
+        for i, row in enumerate(zipped_list_data, 1):
+            for j, (fmt, v) in enumerate(zip(formatters, flatten(row))):
+                sheet.write(i, j, fmt(v))
+        workbook.close()
+
 
 class XlsReader(_BaseExcelReader):
     """Reader for .xls files"""
diff --git a/Orange/preprocess/impute.py b/Orange/preprocess/impute.py
--- a/Orange/preprocess/impute.py
+++ b/Orange/preprocess/impute.py
@@ -7,7 +7,7 @@
 from .transformation import Transformation, Lookup
 
 __all__ = ["ReplaceUnknowns", "Average", "DoNotImpute", "DropInstances",
-           "Model", "AsValue", "Random", "Default"]
+           "Model", "AsValue", "Random", "Default", "FixedValueByType"]
 
 
 class ReplaceUnknowns(Transformation):
@@ -113,6 +113,10 @@ def __call__(self, data, variable, value=None):
         a.to_sql = ImputeSql(variable, value)
         return a
 
+    @staticmethod
+    def supports_variable(variable):
+        return variable.is_primitive()
+
 
 class ImputeSql(Reprable):
     def __init__(self, var, default):
@@ -124,7 +128,7 @@ def __call__(self):
 
 
 class Default(BaseImputeMethod):
-    name = "Value"
+    name = "Fixed value"
     short_name = "value"
     description = ""
     columns_only = True
@@ -142,6 +146,32 @@ def copy(self):
         return Default(self.default)
 
 
+class FixedValueByType(BaseImputeMethod):
+    name = "Fixed value"
+    short_name = "Fixed Value"
+    format = "{var.name}"
+
+    def __init__(self,
+                 default_discrete=np.nan, default_continuous=np.nan,
+                 default_string=None, default_time=np.nan):
+        # If you change the order of args or in dict, also fix method copy
+        self.defaults = {
+            Orange.data.DiscreteVariable: default_discrete,
+            Orange.data.ContinuousVariable: default_continuous,
+            Orange.data.StringVariable: default_string,
+            Orange.data.TimeVariable: default_time
+        }
+
+    def __call__(self, data, variable, *, default=None):
+        variable = data.domain[variable]
+        if default is None:
+            default = self.defaults[type(variable)]
+        return variable.copy(compute_value=ReplaceUnknowns(variable, default))
+
+    def copy(self):
+        return FixedValueByType(*self.defaults.values())
+
+
 class ReplaceUnknownsModel(Reprable):
     """
     Replace unknown values with predicted values using a `Orange.base.Model`
@@ -272,6 +302,9 @@ def __call__(self, data, variable):
         else:
             raise TypeError(type(variable))
 
+    @staticmethod
+    def supports_variable(variable):
+        return variable.is_primitive()
 
 class ReplaceUnknownsRandom(Transformation):
     """
@@ -354,3 +387,7 @@ def __call__(self, data, variable):
             dist[1, :] += 1 / dist.shape[1]
         return variable.copy(
             compute_value=ReplaceUnknownsRandom(variable, dist))
+
+    @staticmethod
+    def supports_variable(variable):
+        return variable.is_primitive()
diff --git a/Orange/widgets/data/owimpute.py b/Orange/widgets/data/owimpute.py
--- a/Orange/widgets/data/owimpute.py
+++ b/Orange/widgets/data/owimpute.py
@@ -11,10 +11,11 @@
 from AnyQt.QtWidgets import (
     QGroupBox, QRadioButton, QPushButton, QHBoxLayout, QGridLayout,
     QVBoxLayout, QStackedWidget, QComboBox,
-    QButtonGroup, QStyledItemDelegate, QListView, QDoubleSpinBox
-)
-from AnyQt.QtCore import Qt, QThread, QModelIndex
+    QButtonGroup, QStyledItemDelegate, QListView, QDoubleSpinBox, QLabel)
+from AnyQt.QtCore import Qt, QThread, QModelIndex, QDateTime, QLocale
 from AnyQt.QtCore import pyqtSlot as Slot
+from AnyQt.QtGui import QDoubleValidator
+
 from orangewidget.utils.listview import ListViewSearch
 
 import Orange.data
@@ -154,6 +155,8 @@ class Warning(OWWidget.Warning):
     _variable_imputation_state = settings.ContextSetting({})  # type: VariableState
 
     autocommit = settings.Setting(True)
+    default_numeric = settings.Setting("")
+    default_time = settings.Setting(0)
 
     want_main_area = False
     resizing_enabled = False
@@ -171,11 +174,13 @@ def __init__(self):
         main_layout.setContentsMargins(10, 10, 10, 10)
         self.controlArea.layout().addLayout(main_layout)
 
-        box = QGroupBox(title=self.tr("Default Method"), flat=False)
-        box_layout = QGridLayout(box)
-        box_layout.setContentsMargins(5, 0, 0, 0)
+        box = gui.vBox(None, "Default Method")
         main_layout.addWidget(box)
 
+        box_layout = QGridLayout(box)
+        box_layout.setSpacing(8)
+        box.layout().addLayout(box_layout)
+
         button_group = QButtonGroup()
         button_group.buttonClicked[int].connect(self.set_default_method)
 
@@ -186,6 +191,41 @@ def __init__(self):
             button_group.addButton(button, method)
             box_layout.addWidget(button, i % 3, i // 3)
 
+        def set_to_fixed_value():
+            self.set_default_method(Method.Default)
+
+        def set_default_time(datetime):
+            self.default_time = datetime.toSecsSinceEpoch()
+            if self.default_method_index != Method.Default:
+                set_to_fixed_value()
+            else:
+                self._invalidate()
+
+        hlayout = QHBoxLayout()
+        box.layout().addLayout(hlayout)
+        button = QRadioButton("Fixed values; numeric variables:")
+        button_group.addButton(button, Method.Default)
+        button.setChecked(Method.Default == self.default_method_index)
+        hlayout.addWidget(button)
+
+        locale = QLocale()
+        locale.setNumberOptions(locale.NumberOption.RejectGroupSeparator)
+        validator = QDoubleValidator()
+        validator.setLocale(locale)
+        le = gui.lineEdit(
+            None, self, "default_numeric",
+            validator=validator, alignment=Qt.AlignRight,
+            callback=self._invalidate, focusInCallback=set_to_fixed_value)
+        hlayout.addWidget(le)
+
+        hlayout.addWidget(QLabel(", time:"))
+
+        self.time_widget = gui.DateTimeEditWCalendarTime(self)
+        self.time_widget.setContentsMargins(0, 0, 0, 0)
+        self.default_time = QDateTime.currentDateTime().toSecsSinceEpoch()
+        self.time_widget.dateTimeChanged.connect(set_default_time)
+        hlayout.addWidget(self.time_widget)
+
         self.default_button_group = button_group
 
         box = QGroupBox(title=self.tr("Individual Attribute Settings"),
@@ -267,6 +307,17 @@ def create_imputer(self, method, *args):
             m = AsDefault()
             m.method = default
             return m
+        elif method == Method.Default and not args:  # global default values
+            if self.default_numeric == "":
+                default_num = np.nan
+            else:
+                default_num, ok = QLocale().toDouble(self.default_numeric)
+                if not ok:
+                    default_num = np.nan
+            return impute.FixedValueByType(
+                default_continuous=default_num,
+                default_time=self.default_time or np.nan
+            )
         else:
             return METHODS[method](*args)
 
@@ -302,6 +353,8 @@ def set_data(self, data):
         if data is not None:
             self.varmodel[:] = data.domain.variables
             self.openContext(data.domain)
+            self.time_widget.set_datetime(
+                QDateTime.fromSecsSinceEpoch(self.default_time))
             # restore per variable imputation state
             self._restore_state(self._variable_imputation_state)
 
@@ -660,5 +713,19 @@ def storeSpecificSettings(self):
         super().storeSpecificSettings()
 
 
+def __sample_data():  # pragma: no cover
+    domain = Orange.data.Domain(
+        [Orange.data.ContinuousVariable(f"c{i}") for i in range(3)]
+        + [Orange.data.TimeVariable(f"t{i}") for i in range(3)],
+        [])
+    n = np.nan
+    x = np.array([
+        [1, 2, n, 1000, n, n],
+        [2, n, 1, n, 2000, 2000]
+    ])
+    return Orange.data.Table(domain, x, np.empty((2, 0)))
+
+
 if __name__ == "__main__":  # pragma: no cover
+    # WidgetPreview(OWImpute).run(__sample_data())
     WidgetPreview(OWImpute).run(Orange.data.Table("brown-selected"))


Test patch:
diff --git a/Orange/preprocess/tests/test_impute.py b/Orange/preprocess/tests/test_impute.py
--- a/Orange/preprocess/tests/test_impute.py
+++ b/Orange/preprocess/tests/test_impute.py
@@ -1,7 +1,12 @@
 import unittest
 
-from Orange.data import DiscreteVariable, ContinuousVariable
-from Orange.preprocess.impute import ReplaceUnknownsRandom, ReplaceUnknowns
+import numpy as np
+
+from Orange.data import \
+    Domain, Table, \
+    DiscreteVariable, ContinuousVariable, TimeVariable, StringVariable
+from Orange.preprocess.impute import ReplaceUnknownsRandom, ReplaceUnknowns, \
+    FixedValueByType
 from Orange.statistics.distribution import Discrete
 
 
@@ -52,5 +57,59 @@ def test_equality(self):
         self.assertNotEqual(hash(t1), hash(t1a))
 
 
+class TestFixedValuesByType(unittest.TestCase):
+    def setUp(self):
+        domain = Domain(
+            [DiscreteVariable("d", values=tuple("abc")),
+             ContinuousVariable("c"),
+             TimeVariable("t")],
+            [],
+            [StringVariable("s")]
+        )
+        n = np.nan
+        self.data = Table(
+            domain,
+            np.array([[1, n, 15], [n, 42, n]]),
+            np.empty((2, 0)),
+            np.array([["foo"], [""]]))
+
+    def test_none_defined(self):
+        d, c, t = self.data.domain.attributes
+        s,  = self.data.domain.metas
+
+        imputer = FixedValueByType()
+        for var in (d, c, t):
+            imp = imputer(self.data, var)
+            self.assertIsInstance(imp.compute_value, ReplaceUnknowns)
+            self.assertTrue(np.isnan(imp.compute_value.value))
+        imp = imputer(self.data, s)
+        self.assertIsInstance(imp.compute_value, ReplaceUnknowns)
+        self.assertIsNone(imp.compute_value.value)
+
+    def test_all_defined(self):
+        d, c, t = self.data.domain.attributes
+        s, = self.data.domain.metas
+
+        imputer = FixedValueByType(
+            default_discrete=1, default_continuous=42,
+            default_string="foo", default_time=3.14)
+
+        self.assertEqual(imputer(self.data, d).compute_value.value, 1)
+        self.assertEqual(imputer(self.data, c).compute_value.value, 42)
+        self.assertEqual(imputer(self.data, t).compute_value.value, 3.14)
+        self.assertEqual(imputer(self.data, s).compute_value.value, "foo")
+
+    def test_with_default(self):
+        s, = self.data.domain.metas
+
+        imputer = FixedValueByType(
+            default_discrete=1, default_continuous=42,
+            default_string="foo", default_time=3.14)
+
+        self.assertEqual(
+            imputer(self.data, s, default="bar").compute_value.value,
+            "bar")
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/Orange/tests/test_xlsx_reader.py b/Orange/tests/test_xlsx_reader.py
--- a/Orange/tests/test_xlsx_reader.py
+++ b/Orange/tests/test_xlsx_reader.py
@@ -16,11 +16,11 @@ def get_dataset(name):
 
 
 def get_xlsx_reader(name: str) -> io.ExcelReader:
-    return io.ExcelReader(get_dataset(name))
+    return io.ExcelReader(get_dataset(name + ".xlsx"))
 
 
 def get_xls_reader(name: str) -> io.XlsReader:
-    return io.XlsReader(get_dataset(name))
+    return io.XlsReader(get_dataset(name + ".xls"))
 
 
 def read_file(reader: Callable, name: str) -> Table:
@@ -37,7 +37,7 @@ def wrapper(self):
 
 class TestExcelReader(unittest.TestCase):
     def test_read_round_floats(self):
-        table = read_file(get_xlsx_reader, "round_floats.xlsx")
+        table = read_file(get_xlsx_reader, "round_floats")
         domain = table.domain
         self.assertIsNone(domain.class_var)
         self.assertEqual(len(domain.metas), 0)
@@ -50,7 +50,7 @@ def test_read_round_floats(self):
 class TestExcelHeader0(unittest.TestCase):
     @test_xlsx_xls
     def test_read(self, reader: Callable[[str], io.FileFormat]):
-        table = read_file(reader, "header_0.xlsx")
+        table = read_file(reader, "header_0")
         domain = table.domain
         self.assertIsNone(domain.class_var)
         self.assertEqual(len(domain.metas), 0)
@@ -68,13 +68,13 @@ def test_read(self, reader: Callable[[str], io.FileFormat]):
 class TextExcelSheets(unittest.TestCase):
     @test_xlsx_xls
     def test_sheets(self, reader: Callable[[str], io.FileFormat]):
-        reader = reader("header_0_sheet.xlsx")
+        reader = reader("header_0_sheet")
         self.assertSequenceEqual(reader.sheets,
                                  ["Sheet1", "my_sheet", "Sheet3"])
 
     @test_xlsx_xls
     def test_named_sheet(self, reader: Callable[[str], io.FileFormat]):
-        reader = reader("header_0_sheet.xlsx")
+        reader = reader("header_0_sheet")
         reader.select_sheet("my_sheet")
         table = reader.read()
         self.assertEqual(len(table.domain.attributes), 4)
@@ -96,7 +96,7 @@ def test_named_sheet_table_xls(self):
 class TestExcelHeader1(unittest.TestCase):
     @test_xlsx_xls
     def test_no_flags(self, reader: Callable[[str], io.FileFormat]):
-        table = read_file(reader, "header_1_no_flags.xlsx")
+        table = read_file(reader, "header_1_no_flags")
         domain = table.domain
         self.assertEqual(len(domain.metas), 0)
         self.assertEqual(len(domain.attributes), 4)
@@ -115,7 +115,7 @@ def test_no_flags(self, reader: Callable[[str], io.FileFormat]):
 
     @test_xlsx_xls
     def test_flags(self, reader: Callable[[str], io.FileFormat]):
-        table = read_file(reader, "header_1_flags.xlsx")
+        table = read_file(reader, "header_1_flags")
         domain = table.domain
 
         self.assertEqual(len(domain.attributes), 1)
@@ -146,7 +146,7 @@ def test_flags(self, reader: Callable[[str], io.FileFormat]):
 class TestExcelHeader3(unittest.TestCase):
     @test_xlsx_xls
     def test_read(self, reader: Callable[[str], io.FileFormat]):
-        table = read_file(reader, "header_3.xlsx")
+        table = read_file(reader, "header_3")
         domain = table.domain
 
         self.assertEqual(len(domain.attributes), 2)
diff --git a/Orange/tests/xlsx_files/header_0.xls b/Orange/tests/xlsx_files/header_0.xls
new file mode 100644
Binary files /dev/null and b/Orange/tests/xlsx_files/header_0.xls differ
diff --git a/Orange/tests/xlsx_files/header_1_flags.xls b/Orange/tests/xlsx_files/header_1_flags.xls
new file mode 100644
Binary files /dev/null and b/Orange/tests/xlsx_files/header_1_flags.xls differ
diff --git a/Orange/tests/xlsx_files/header_1_no_flags.xls b/Orange/tests/xlsx_files/header_1_no_flags.xls
new file mode 100644
Binary files /dev/null and b/Orange/tests/xlsx_files/header_1_no_flags.xls differ
diff --git a/Orange/tests/xlsx_files/header_3.xls b/Orange/tests/xlsx_files/header_3.xls
new file mode 100644
Binary files /dev/null and b/Orange/tests/xlsx_files/header_3.xls differ
diff --git a/Orange/widgets/data/tests/test_owimpute.py b/Orange/widgets/data/tests/test_owimpute.py
--- a/Orange/widgets/data/tests/test_owimpute.py
+++ b/Orange/widgets/data/tests/test_owimpute.py
@@ -3,10 +3,10 @@
 from unittest.mock import Mock
 import numpy as np
 
-from AnyQt.QtCore import Qt, QItemSelection
+from AnyQt.QtCore import Qt, QItemSelection, QLocale
 from AnyQt.QtTest import QTest
 
-from Orange.data import Table, Domain
+from Orange.data import Table, Domain, ContinuousVariable, TimeVariable
 from Orange.preprocess import impute
 from Orange.widgets.data.owimpute import OWImpute, AsDefault, Learner, Method
 from Orange.widgets.tests.base import WidgetTest
@@ -119,6 +119,31 @@ def test_select_method(self):
         self.assertIsInstance(widget.get_method_for_column(0), AsDefault)
         self.assertIsInstance(widget.get_method_for_column(2), AsDefault)
 
+    def test_overall_default(self):
+        domain = Domain(
+            [ContinuousVariable(f"c{i}") for i in range(3)]
+            + [TimeVariable(f"t{i}") for i in range(3)],
+            [])
+        n = np.nan
+        x = np.array([
+            [1, 2, n, 1000, n, n],
+            [2, n, 1, n, 2000, 2000]
+        ])
+        data = Table(domain, x, np.empty((2, 0)))
+
+        widget = self.widget
+        widget.default_numeric = QLocale().toString(3.14)
+        widget.default_time = 42
+        widget.default_method_index = Method.Default
+
+        self.send_signal(self.widget.Inputs.data, data)
+        imp_data = self.get_output(self.widget.Outputs.data)
+        np.testing.assert_almost_equal(
+            imp_data.X,
+            [[1, 2, 3.14, 1000, 42, 42],
+             [2, 3.14, 1, 42, 2000, 2000]]
+        )
+
     def test_value_edit(self):
         data = Table("heart_disease")[::10]
         self.send_signal(self.widget.Inputs.data, data)