Repository: biolab/orange3

Issue:
Venn diagram should allow matching by equality
Mathematicall a venn diagram is kwalitative,
this option is present in the widget (duplicates) for the output.

the problem: identical rows are not recognized as such
the warning given is: Some variables have been renamed to avoid duplicates. status
Dataset #1 has no suitable identifiers.


a few screenshots to clarify:

the data:
![image](https://user-images.githubusercontent.com/17875724/110925548-a0cfd080-8323-11eb-9108-a210b266a365.png)

the venn diagram with the given warning (and wrong count):
![image](https://user-images.githubusercontent.com/17875724/110925614-b6dd9100-8323-11eb-9201-65cc6549750f.png)

a part of a datatable behind the venn diagram (with duplicates, even though this was not selected in the venn diagram)
![image](https://user-images.githubusercontent.com/17875724/110926064-379c8d00-8324-11eb-93d2-06ec63605ba0.png)

Orange version 3.27.1 on linux installed using pipenv

maybe related to
https://github.com/biolab/orange3/issues/4759



Discussion:
A couple of things here:
1. Venn diagram is currently not intended for the task of filtering duplicates on a single data set as you intended. This can be done with Unique, where a unique identifier can be used. What is the issue here? Venn looks at "Instance identity", where each row is unique (hash). So it works correctly for the task - it outputs unique rows, where by definition each row is unique (not by values, but by identity). Perhaps we could add other options (variables to Rows, matched by), but this would replicate the functionality of Unique (which is intended to be moved to core Orange).
2. The first warning is strange and probably wrong?
>     1. Venn diagram is currently not intended for the task of filtering duplicates on a single data set as you intended. This can be done with Unique, where a unique identifier can be used. What is the issue here? Venn looks at "Instance identity", where each row is unique (hash). So it works correctly for the task - it outputs unique rows, where by definition each row is unique (not by values, but by identity). Perhaps we could add other options (variables to Rows, matched by), but this would replicate the functionality of Unique (which is intended to be moved to core Orange).

I'm down for adding this, even though it may duplicate Unique's functionality.

Sidenote: What would happen if we omitted the Selected column in the Data signal, should there be no selection?
@ajdapretnar, this reminds me that Unique is still in Prototypes. Can you take a look at https://github.com/biolab/orange3-prototypes/pull/241 (I know you already have a lot on your plate today!), merge if OK, and then I'd create a PR in the core repo?

It has an icon, but no docs, obviously. :)
the reason for adding this would be the meaning of venn diagram in mathematics,
to avoid confusion, using the same terminology,
in earlier versions of orange this was the case, the behaviour of this widget changed at some point.

what seems strange is that the hashes for identical rows are different,
maybe the arbitrary row number is included in the hash?
(with arbitrary i mean, the same dataset but in a different order)
And how would you define the meaning of Venn diagram in mathematics? In what sense does Orange misuse mathematical terminology?

I guess your problem is in the definition of sameness. It is neither about hashes nor about row numbers nor about equality.
generally speaking, if you add an element x to a set A, and if you add the element x again to the set A, it should only appear once.

indeed as you say: the definition of sameness. (one can debate about this applied definition in this case, not my intention)
(the above remark about the hashing was only meant for the background, how it is implemented, i haven't checked it though)

in this case of datamining, my practical opinion would be: the suggestion of having several options for defining what is the same, would help a lot, like @irgolic changed the title of the topic, allowing to match for equality)


Resolving patch:
diff --git a/Orange/widgets/visualize/owvenndiagram.py b/Orange/widgets/visualize/owvenndiagram.py
--- a/Orange/widgets/visualize/owvenndiagram.py
+++ b/Orange/widgets/visualize/owvenndiagram.py
@@ -26,7 +26,9 @@
 
 from Orange.data import Table, Domain, StringVariable, RowInstance
 from Orange.data.util import get_unique_names_duplicates
-from Orange.widgets import widget, gui, settings
+from Orange.widgets import widget, gui
+from Orange.widgets.settings import (
+    DomainContextHandler, ContextSetting, Setting)
 from Orange.widgets.utils import itemmodels, colorpalettes
 from Orange.widgets.utils.annotated_data import (create_annotated_table,
                                                  ANNOTATED_DATA_SIGNAL_NAME)
@@ -39,6 +41,24 @@
 _InputData = namedtuple("_InputData", ["key", "name", "table"])
 _ItemSet = namedtuple("_ItemSet", ["key", "name", "title", "items"])
 
+IDENTITY_STR = "Instance identity"
+EQUALITY_STR = "Instance equality"
+
+
+class VennVariableListModel(itemmodels.VariableListModel):
+    def __init__(self):
+        super().__init__([IDENTITY_STR, EQUALITY_STR])
+        self.same_domains = True
+
+    def set_variables(self, variables, same_domains):
+        self[2:] = variables
+        self.same_domains = same_domains
+
+    def flags(self, index):
+        if index.row() == 1 and not self.same_domains:
+            return Qt.NoItemFlags
+        return Qt.ItemIsSelectable | Qt.ItemIsEnabled
+
 
 class OWVennDiagram(widget.OWWidget):
     name = "Venn Diagram"
@@ -66,15 +86,15 @@ class Warning(widget.OWWidget.Warning):
 
     selection: list
 
-    settingsHandler = settings.DomainContextHandler()
+    settingsHandler = DomainContextHandler()
     # Indices of selected disjoint areas
-    selection = settings.Setting([], schema_only=True)
+    selection = Setting([], schema_only=True)
     #: Output unique items (one output row for every unique instance `key`)
     #: or preserve all duplicates in the output.
-    output_duplicates = settings.Setting(False)
-    autocommit = settings.Setting(True)
-    rowwise = settings.Setting(True)
-    selected_feature = settings.ContextSetting(None)
+    output_duplicates = Setting(False)
+    autocommit = Setting(True)
+    rowwise = Setting(True)
+    selected_feature = ContextSetting(IDENTITY_STR)
 
     want_main_area = False
     graph_name = "scene"
@@ -130,11 +150,12 @@ def __init__(self):
                             Qt.Horizontal,
                             addSpaceBefore=False),
             self, "selected_feature",
-            model=itemmodels.VariableListModel(
-                placeholder="Instance identity"
-            ),
-            callback=self._on_inputAttrActivated
-        )
+            model=VennVariableListModel(),
+            callback=self._on_inputAttrActivated,
+            tooltip="Instances are identical if originally coming from the "
+                    "same row of the same table.\n"
+                    "Instances can be check for equality only if described by "
+                    "the same variables.")
         box.layout().setSpacing(6)
         box.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Fixed)
 
@@ -146,12 +167,13 @@ def __init__(self):
         self.output_duplicates_cb = gui.checkBox(
             box, self, "output_duplicates", "Output duplicates",
             callback=lambda: self.commit(),  # pylint: disable=unnecessary-lambda
+            stateWhenDisabled=False,
             attribute=Qt.WA_LayoutUsesWidgetRect)
         auto = gui.auto_send(box, self, "autocommit",
                              box=False,
                              contentsMargins=(0, 0, 0, 0))
         gui.rubber(box)
-        self.output_duplicates_cb.setEnabled(bool(self.rowwise))
+        self._update_duplicates_cb()
         self._queue = []
 
     def resizeEvent(self, event):
@@ -230,7 +252,6 @@ def handleNewSignals(self):
         if not self.autocommit:
             self.unconditional_commit()
 
-        self._updateInfo()
         super().handleNewSignals()
 
     def set_input_summary(self):
@@ -246,31 +267,52 @@ def set_input_summary(self):
             kwargs = {"format": Qt.RichText}
         self.info.set_input_summary(summary, details, **kwargs)
 
-    def intersectionStringAttrs(self):
-        sets = [set(string_attributes(data_.table.domain)) for data_ in self.data.values()]
+    def _intersection_string_attrs(self):
+        sets = [set(string_attributes(data_.table.domain))
+                for data_ in self.data.values()]
         if sets:
-            return reduce(set.intersection, sets)
-        return set()
+            return list(reduce(set.intersection, sets))
+        return []
+
+    def _all_domains_same(self):
+        domains = [data_.table.domain for data_ in self.data.values()]
+        # Domain.__hash__ is hacky, let's not use a set here, just for the case
+        return not domains or all(domain == domains[0] for domain in domains)
+
+    def _uses_feature(self):
+        return isinstance(self.selected_feature, StringVariable)
 
     def _setInterAttributes(self):
         model = self.controls.selected_feature.model()
-        model[:] = [None] + list(self.intersectionStringAttrs())
-        if self.selected_feature:
-            names = (var.name for var in model if var)
-            if self.selected_feature.name not in names:
-                self.selected_feature = model[0]
+        same_domains = self._all_domains_same()
+        variables = self._intersection_string_attrs()
+        model.set_variables(variables, same_domains)
+        if self.selected_feature == EQUALITY_STR and not same_domains \
+                or self._uses_feature() and \
+                self.selected_feature.name not in (var.name for var in variables):
+            self.selected_feature = IDENTITY_STR
+
+    @staticmethod
+    def _hashes(table):
+        # In the interest of space, we compare hashes. If this is not OK,
+        # concatenate bytes instead of xoring hashes. Renaming a method
+        # brings bonus points.
+        return [hash(inst.x.data.tobytes())
+                ^ hash(inst.y.data.tobytes())
+                ^ hash(inst.metas.data.tobytes()) for inst in table]
 
     def _itemsForInput(self, key):
         """
         Calculates input for venn diagram, according to user's settings.
         """
         table = self.data[key].table
-        attr = self.selected_feature
-        if attr:
-            return [str(inst[attr]) for inst in table
-                    if not np.isnan(inst[attr])]
-        else:
+        if self.selected_feature == IDENTITY_STR:
             return list(table.ids)
+        if self.selected_feature == EQUALITY_STR:
+            return self._hashes(table)
+        attr = self.selected_feature
+        return [str(inst[attr]) for inst in table
+                if not np.isnan(inst[attr])]
 
     def _createItemsets(self):
         """
@@ -329,19 +371,13 @@ def _createDiagram(self):
                 area.setText("{0}".format(len(area_items)))
 
             label = disjoint_set_label(i, n, simplify=False)
-            head = "<h4>|{}| = {}</h4>".format(label, len(area_items))
-            if len(area_items) > 32:
-                items_str = ", ".join(map(escape, area_items[:32]))
-                hidden = len(area_items) - 32
-                tooltip = ("{}<span>{}, ...</br>({} items not shown)<span>"
-                           .format(head, items_str, hidden))
-            elif area_items:
-                tooltip = "{}<span>{}</span>".format(
-                    head,
-                    ", ".join(map(escape, area_items))
-                )
-            else:
-                tooltip = head
+            tooltip = "<h4>|{}| = {}</h4>".format(label, len(area_items))
+            if self._uses_feature() or not self.rowwise:
+                # Nothing readable to show when matching by identity or equality
+                tooltip += "<span>" + ", ".join(map(escape, area_items[:32]))
+                if len(area_items) > 32:
+                    tooltip += f"</br>({len(area_items) - 32} items not shown)"
+                tooltip += "</span>"
 
             area.setToolTip(tooltip)
 
@@ -352,21 +388,6 @@ def _createDiagram(self):
         self._updating = False
         self._on_selectionChanged()
 
-    def _updateInfo(self):
-        # Clear all warnings
-        self.warning()
-
-        if self.selected_feature is None:
-            no_idx = ["#{}".format(i + 1)
-                      for i, key in enumerate(self.data)
-                      if not source_attributes(self.data[key].table.domain)]
-            if len(no_idx) == 1:
-                self.warning("Dataset {} has no suitable identifiers."
-                             .format(no_idx[0]))
-            elif len(no_idx) > 1:
-                self.warning("Datasets {} and {} have no suitable identifiers."
-                             .format(", ".join(no_idx[:-1]), no_idx[-1]))
-
     def _on_selectionChanged(self):
         if self._updating:
             return
@@ -375,17 +396,20 @@ def _on_selectionChanged(self):
         self.selection = [i for i, area in enumerate(areas) if area.isSelected()]
         self.invalidateOutput()
 
+    def _update_duplicates_cb(self):
+        self.output_duplicates_cb.setEnabled(
+            self.rowwise and self._uses_feature())
+
     def _on_matching_changed(self):
-        self.output_duplicates_cb.setEnabled(bool(self.rowwise))
+        self._update_duplicates_cb()
         if not self.settings_compatible():
             self.invalidateOutput()
             return
         self._createItemsets()
         self._createDiagram()
-        self._updateInfo()
 
     def _on_inputAttrActivated(self):
-        self.rowwise = 1
+        self.rowwise = True
         self._on_matching_changed()
 
     def _on_itemTextEdited(self, index, text):
@@ -589,7 +613,7 @@ def extract_rowwise(self, var_dict, ids=None, selection=False):
 
         if renamed:
             self.Warning.renamed_vars(', '.join(renamed))
-        ids = None if self.selected_feature else np.array(all_ids)
+        ids = None if self._uses_feature() else np.array(all_ids)
         table = self.merge_data(domain, values, ids)
         if selection:
             mask = [idx in self.selected_items for idx in all_ids]
@@ -598,18 +622,18 @@ def extract_rowwise(self, var_dict, ids=None, selection=False):
 
     def get_indices(self, table, selection):
         """Returns mappings of ids (be it row id or string) to indices in tables"""
-        if self.selected_feature:
+        if self.selected_feature == IDENTITY_STR:
+            items = table.ids
+            ids = range(len(table))
+        elif self.selected_feature == EQUALITY_STR:
+            items, ids = np.unique(self._hashes(table), return_index=True)
+        else:
+            items = getattr(table[:, self.selected_feature], 'metas')
             if self.output_duplicates and selection:
-                items, inverse = np.unique(getattr(table[:, self.selected_feature], 'metas'),
-                                           return_inverse=True)
+                items, inverse = np.unique(items, return_inverse=True)
                 ids = [np.nonzero(inverse == idx)[0] for idx in range(len(items))]
             else:
-                items, ids = np.unique(getattr(table[:, self.selected_feature], 'metas'),
-                                       return_index=True)
-
-        else:
-            items = table.ids
-            ids = range(len(table))
+                items, ids = np.unique(items, return_index=True)
 
         if selection:
             return {item: idx for item, idx in zip(items, ids)
@@ -744,18 +768,19 @@ def get_disjoint(self, sets):
 
         return disjoint_sets, included_tables
 
+    @classmethod
+    def migrate_settings(cls, settings, version):
+        if version < 3:
+            if settings.pop("selected_feature", None) is None:
+                settings["selected_feature"] = IDENTITY_STR
+
+
 def string_attributes(domain):
     """
     Return all string attributes from the domain.
     """
     return [attr for attr in domain.variables + domain.metas if attr.is_string]
 
-def source_attributes(domain):
-    """
-    Return all suitable attributes for the venn diagram.
-    """
-    return string_attributes(domain)  # + discrete_attributes(domain)
-
 
 def disjoint_set_label(i, n, simplify=False):
     """


Test patch:
diff --git a/Orange/widgets/visualize/tests/test_owvenndiagram.py b/Orange/widgets/visualize/tests/test_owvenndiagram.py
--- a/Orange/widgets/visualize/tests/test_owvenndiagram.py
+++ b/Orange/widgets/visualize/tests/test_owvenndiagram.py
@@ -7,6 +7,8 @@
 
 import numpy as np
 
+from AnyQt.QtCore import Qt
+
 from orangewidget.widget import StateInfo
 
 from Orange.data import (Table, Domain, StringVariable,
@@ -15,9 +17,9 @@
 from Orange.widgets.utils.annotated_data import (ANNOTATED_DATA_FEATURE_NAME)
 from Orange.widgets.utils.state_summary import (format_multiple_summaries,
                                                 format_summary_details)
-from Orange.widgets.visualize.owvenndiagram import (OWVennDiagram, get_perm,
-                                                    arrays_equal, pad_columns)
-
+from Orange.widgets.visualize.owvenndiagram import (
+    OWVennDiagram, get_perm, arrays_equal, pad_columns,
+    IDENTITY_STR, EQUALITY_STR)
 
 
 class TestOWVennDiagram(WidgetTest, WidgetOutputsTestMixin):
@@ -42,7 +44,7 @@ def test_rows_id(self):
         data1[:, 1] = 1
         self.widget.rowwise = True
         self.send_signal(self.signal_name, data1[:10], 1)
-        self.widget.selected_feature = None
+        self.widget.selected_feature = IDENTITY_STR
         self.send_signal(self.signal_name, data[5:10], 2)
 
         self.assertIsNone(self.get_output(self.widget.Outputs.selected_data))
@@ -110,6 +112,70 @@ def test_multiple_input_rows_id(self):
         self.assertIsNone(self.get_output(self.widget.Outputs.selected_data))
         self.assertIsNone(self.get_output(self.widget.Outputs.annotated_data))
 
+    def test_equality(self):
+        self.widget.rowwise = True
+        data1 = Table.from_numpy(None, np.arange(20).reshape(20, 1))
+        data2 = Table.from_numpy(None, np.arange(10, 25).reshape(15, 1))
+        self.send_signal(self.signal_name, data1, 1)
+        self.send_signal(self.signal_name, data2, 2)
+
+        self.widget.vennwidget.vennareas()[3].setSelected(True)
+        self.assertFalse(bool(self.get_output(self.widget.Outputs.selected_data)))
+
+        self.widget.selected_feature = EQUALITY_STR
+        self.widget._on_inputAttrActivated()
+        self.assertEqual(
+            set(self.get_output(self.widget.Outputs.selected_data).X.flatten()),
+            set(range(10, 20)))
+
+    def test_disable_duplicates(self):
+        cb = self.widget.output_duplicates_cb
+        data1 = Table.from_numpy(None, np.arange(20).reshape(20, 1))
+        data2 = Table.from_numpy(None, np.arange(10, 25).reshape(15, 1))
+        self.send_signal(self.signal_name, data1, 1)
+        self.send_signal(self.signal_name, data2, 2)
+        self.assertFalse(cb.isEnabled())
+
+        data = Table("zoo")
+        self.send_signal(self.signal_name, data, 1)
+        self.send_signal(self.signal_name, data, 2)
+        for self.widget.selected_feature in (IDENTITY_STR, EQUALITY_STR):
+            self.widget._on_inputAttrActivated()
+            self.assertFalse(cb.isEnabled())
+
+        self.widget.selected_feature = \
+            self.widget.controls.selected_feature.model()[2]
+        self.widget._on_inputAttrActivated()
+        self.assertTrue(cb.isEnabled())
+
+        self.widget.rowwise = False
+        self.widget._on_matching_changed()
+        self.assertFalse(cb.isEnabled())
+
+    def test_disable_match_equality(self):
+        model = self.widget.controls.selected_feature.model()
+        row0 = model.index(0, 0)
+        row1 = model.index(1, 0)
+        data = Table("zoo")
+
+        self.assertTrue(model.flags(row0) & Qt.ItemIsEnabled)
+        self.assertTrue(model.flags(row1) & Qt.ItemIsEnabled)
+
+        self.send_signal(self.signal_name, data, 1)
+        self.assertTrue(model.flags(row0) & Qt.ItemIsEnabled)
+        self.assertTrue(model.flags(row1) & Qt.ItemIsEnabled)
+
+        self.send_signal(self.signal_name, data, 2)
+        self.assertTrue(model.flags(row0) & Qt.ItemIsEnabled)
+        self.assertTrue(model.flags(row1) & Qt.ItemIsEnabled)
+
+        data1 = Table.from_numpy(None, np.arange(20).reshape(20, 1))
+        self.widget.selected_feature = EQUALITY_STR
+        self.send_signal(self.signal_name, data1, 3)
+        self.assertEqual(self.widget.selected_feature, IDENTITY_STR)
+        self.assertTrue(model.flags(row0) & Qt.ItemIsEnabled)
+        self.assertFalse(model.flags(row1) & Qt.ItemIsEnabled)
+
     def test_multiple_input_over_cols(self):
         self.widget.rowwise = False
         selected_atr_name = 'Selected'
@@ -301,6 +367,16 @@ def test_summary(self):
         self.assertIsInstance(info._StateInfo__output_summary, StateInfo.Empty)
         self.assertEqual(info._StateInfo__output_summary.details, no_output)
 
+    def test_migration_to_3(self):
+        settings = {"selected_feature": None}
+        OWVennDiagram.migrate_settings(settings, 1)
+        self.assertEqual(settings["selected_feature"], IDENTITY_STR)
+
+        settings = {}
+        OWVennDiagram.migrate_settings(settings, 1)
+        self.assertEqual(settings["selected_feature"], IDENTITY_STR)
+
+
 class TestVennUtilities(unittest.TestCase):
 
     def test_array_equals_cols(self):