Repository: qutebrowser/qutebrowser

Issue:
Consider better data structures for bindings
e.g. a trie - from #4652 by @user202729:

> @The-Compiler Regarding the problem you mentioned: I implemented trie data structure for keybindings (https://paste.the-compiler.org/view/c15f385c), however the implementation is incomplete:
> 
> * Partial key sequence suggestion or config bind/unbind (`on_config_changed`) still traverse the whole dict.
> * `update` method is not implemented (although this one is easy to implement, just loop over all items)




Discussion:
Note that this approach will have some extra cost when adding bindings (eg:
when hinting, the hint mode dict is rewritten on the fly). I don't think that
will outweigh the improvements from not iterating over everything though (even
in hint mode).

Another approach I thought of (which is sloppier than a trie but maybe easier)
is to just add every combination leading to a complete key to the binding. Eg:
'abc' would store 'a->partial, ab->partial, abc->complete' in a dict. A trie
is better and cleaner though, in my opinion (wouldn't waste a bunch of space,
although it might be a bit slower).

@jgkamat Why would adding bindings have extra cost? If implemented correctly it's only necessary to traverse the trie, which has time complexity == sequence length, which is about as good as it can get.
user202729 writes:

> @jgkamat Why would adding bindings have extra cost? If implemented correctly it's only necessary to traverse the trie, which has time complexity == sequence length, which is about as good as it can get.

If I understood the old logic correctly, the old logic had an insertion cost
scaling with only the number of bindings, while a trie will be the key
sequence length * number of bindings (just looking at time complexity alone).
If, for example, there was some logic that inserted extremely long bindings
repeatedly, it will probably get slower.

Of course, the lookup cost will probably be so dramatically better that it
won't matter at all.

@jgkamat 

* Actually, I think the old logic is also key sequence length * number of bindings, because hash takes linear time. Try this:

```
>>> class a:
...  def __hash__(self):
...   print('h')
...   return 1
... 
>>> x=a()
>>> hash(x)
h
1
>>> hash(x)
h
1
```

* Ideally, there should be some way for the config system to notify the key parser exactly which one is bound/unbound so that the key parser only have to update that one, in that case both would takes (amortized) time only (key sequence length).
@jgkamat Regarding your proposed alternative: it would need to store how many "complete" bindings may result in a "partial" binding, otherwise delete a binding would requires traversing the whole dict. (that is, assume the `config_updated`/`read_config` logic is updated so that it won't just delete and rebuild the whole dict)

Resolving patch:
diff --git a/qutebrowser/keyinput/basekeyparser.py b/qutebrowser/keyinput/basekeyparser.py
--- a/qutebrowser/keyinput/basekeyparser.py
+++ b/qutebrowser/keyinput/basekeyparser.py
@@ -20,6 +20,7 @@
 """Base class for vim-like key sequence parser."""
 
 import string
+import collections
 
 from PyQt5.QtCore import pyqtSignal, QObject
 from PyQt5.QtGui import QKeySequence
@@ -28,6 +29,87 @@
 from qutebrowser.utils import usertypes, log, utils
 from qutebrowser.keyinput import keyutils
 
+MYPY = False
+if MYPY:
+    from typing import MutableMapping, Optional
+
+
+class BindingTrie(collections.abc.MutableMapping):
+
+    """Helper class for key parser. Represents a set of bindings.
+
+    Except for the root item, there is no children BindingTrie with no bound
+    commands.
+
+    This class works like a dict[keyutils.KeySequence, str], but with matches
+    method added.
+
+    Attributes:
+        child: A map. Keys of this map can be get from the KeyInfo.to_int
+               method.
+        command: Command associated with this trie node.
+    """
+
+    __slots__ = 'child', 'command'
+
+    def __init__(self):
+        self.child = {}  # type: MutableMapping[int, BindingTrie]
+        self.command = None  # type: Optional[str]
+
+    def __getitem__(self, sequence: keyutils.KeySequence):
+        matchtype, command = self.matches(sequence)
+        if matchtype != QKeySequence.ExactMatch:
+            raise KeyError(sequence)
+        return command
+
+    def __setitem__(
+            self, sequence: keyutils.KeySequence, command: str):
+        node = self
+        for key in sequence:
+            key_int = key.to_int()
+            if key_int not in node.child:
+                node.child[key_int] = BindingTrie()
+            node = node.child[key_int]
+
+        node.command = command
+
+    def __delitem__(self, sequence: keyutils.KeySequence):
+        raise NotImplementedError
+
+    def __iter__(self):
+        raise NotImplementedError
+
+    def __len__(self):
+        raise NotImplementedError
+
+    def matches(self, sequence: keyutils.KeySequence):
+        """Try to match a given keystring with any bound keychain.
+
+        Args:
+            sequence: The command string to find.
+
+        Return:
+            A tuple (matchtype, binding).
+                matchtype: QKeySequence.ExactMatch, QKeySequence.PartialMatch
+                           or QKeySequence.NoMatch.
+                binding: - None with QKeySequence.PartialMatch or
+                           QKeySequence.NoMatch.
+                         - The found binding with QKeySequence.ExactMatch.
+        """
+        node = self
+        for key in sequence:
+            try:
+                node = node.child[key.to_int()]
+            except KeyError:
+                return QKeySequence.NoMatch, None
+
+        if node.command is not None:
+            return QKeySequence.ExactMatch, node.command
+        elif node.child:
+            return QKeySequence.PartialMatch, None
+        else:  # This can only happen when there is no bindings
+            return QKeySequence.NoMatch, None
+
 
 class BaseKeyParser(QObject):
 
@@ -75,7 +157,7 @@ def __init__(self, win_id, parent=None, supports_count=True):
         self._sequence = keyutils.KeySequence()
         self._count = ''
         self._supports_count = supports_count
-        self.bindings = {}
+        self.bindings = BindingTrie()
         config.instance.changed.connect(self._on_config_changed)
 
     def __repr__(self):
@@ -104,17 +186,8 @@ def _match_key(self, sequence):
         """
         assert sequence
         assert not isinstance(sequence, str)
-        result = QKeySequence.NoMatch
-
-        for seq, cmd in self.bindings.items():
-            assert not isinstance(seq, str), seq
-            match = sequence.matches(seq)
-            if match == QKeySequence.ExactMatch:
-                return match, cmd
-            elif match == QKeySequence.PartialMatch:
-                result = QKeySequence.PartialMatch
 
-        return result, None
+        return self.bindings.matches(sequence)
 
     def _match_without_modifiers(self, sequence):
         """Try to match a key with optional modifiers stripped."""
@@ -216,6 +289,9 @@ def handle(self, e, *, dry_run=False):
 
     @config.change_filter('bindings')
     def _on_config_changed(self):
+        # Note: This function is called which erases and rebuild the whole
+        # self.bindings object, even if it only needs to add or remove one
+        # item.
         self._read_config()
 
     def _read_config(self, modename=None):
@@ -234,7 +310,7 @@ def _read_config(self, modename=None):
             modename = self._modename
         else:
             self._modename = modename
-        self.bindings = {}
+        self.bindings = BindingTrie()
 
         for key, cmd in config.key_instance.get_bindings_for(modename).items():
             assert not isinstance(key, str), key


Test patch:
diff --git a/tests/unit/keyinput/test_bindingtrie.py b/tests/unit/keyinput/test_bindingtrie.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/keyinput/test_bindingtrie.py
@@ -0,0 +1,93 @@
+# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
+
+# Copyright 2019 Jay Kamat <jaygkamat@gmail.com>:
+#
+# This file is part of qutebrowser.
+#
+# qutebrowser is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# qutebrowser is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with qutebrowser.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Tests for the BindingTrie."""
+
+import string
+import itertools
+
+import pytest
+
+from PyQt5.QtGui import QKeySequence
+
+from qutebrowser.keyinput import basekeyparser
+from qutebrowser.keyinput import keyutils
+from unit.keyinput import test_keyutils
+
+
+@pytest.mark.parametrize('entered, configured, expected',
+                         test_keyutils.TestKeySequence.MATCH_TESTS)
+def test_matches_single(entered, configured, expected):
+    entered = keyutils.KeySequence.parse(entered)
+    configured = keyutils.KeySequence.parse(configured)
+    trie = basekeyparser.BindingTrie()
+    trie[configured] = "eeloo"
+    ret_expected = None
+    if expected == QKeySequence.ExactMatch:
+        ret_expected = "eeloo"
+    assert trie.matches(entered) == (expected, ret_expected)
+
+
+@pytest.mark.parametrize(
+    'configured, expected_tuple',
+    (((),
+      # null match
+      (('a', QKeySequence.NoMatch),
+       ('', QKeySequence.NoMatch))),
+     (('abcd',),
+      (('abcd', QKeySequence.ExactMatch),
+       ('abc', QKeySequence.PartialMatch))),
+     (('aa', 'ab', 'ac', 'ad'),
+      (('ac', QKeySequence.ExactMatch),
+       ('a', QKeySequence.PartialMatch),
+       ('f', QKeySequence.NoMatch),
+       ('acd', QKeySequence.NoMatch))),
+     (('aaaaaaab', 'aaaaaaac', 'aaaaaaad'),
+      (('aaaaaaab', QKeySequence.ExactMatch),
+       ('z', QKeySequence.NoMatch))),
+     (tuple(string.ascii_letters),
+      (('a', QKeySequence.ExactMatch),
+       ('!', QKeySequence.NoMatch)))))
+def test_matches_tree(configured, expected_tuple, benchmark):
+    trie = basekeyparser.BindingTrie()
+    trie.update(dict.fromkeys(
+        map(keyutils.KeySequence.parse, configured), "eeloo"))
+
+    def _run():
+        for entered, expected in expected_tuple:
+            entered = keyutils.KeySequence.parse(entered)
+            ret_expected = None
+            if expected == QKeySequence.ExactMatch:
+                ret_expected = "eeloo"
+            assert trie.matches(entered) == (expected, ret_expected)
+    benchmark(_run)
+
+
+@pytest.mark.parametrize(
+    'configured',
+    ('a',
+     itertools.permutations('asdfghjkl', 3)))
+def test_bench_create(configured, benchmark):
+    configured = dict.fromkeys(
+        map(keyutils.KeySequence.parse, configured), "dres")
+
+    def _run():
+        trie = basekeyparser.BindingTrie()
+        trie.update(configured)
+    benchmark(_run)
diff --git a/tests/unit/keyinput/test_keyutils.py b/tests/unit/keyinput/test_keyutils.py
--- a/tests/unit/keyinput/test_keyutils.py
+++ b/tests/unit/keyinput/test_keyutils.py
@@ -317,7 +317,7 @@ def test_getitem_slice(self):
         assert s1[3:5] == s2
         assert seq[3:5] == expected
 
-    @pytest.mark.parametrize('entered, configured, expected', [
+    MATCH_TESTS = [
         # config: abcd
         ('abc', 'abcd', QKeySequence.PartialMatch),
         ('abcd', 'abcd', QKeySequence.ExactMatch),
@@ -339,8 +339,9 @@ def test_getitem_slice(self):
         # empty strings
         ('', '', QKeySequence.ExactMatch),
         ('', 'a', QKeySequence.PartialMatch),
-        ('a', '', QKeySequence.NoMatch),
-    ])
+        ('a', '', QKeySequence.NoMatch)]
+
+    @pytest.mark.parametrize('entered, configured, expected', MATCH_TESTS)
     def test_matches(self, entered, configured, expected):
         entered = keyutils.KeySequence.parse(entered)
         configured = keyutils.KeySequence.parse(configured)