Repository: nvaccess/nvda

Issue:
Use braille display keys as modifier keys
The feature I propose is best described with the following example.

There are several braille displays with function keys assigned to specific keyboard modifier keys, one of which is the Hims Braille Edge. This display has 8 function keys, from left to right meant to be assigned to Esc, Tab, Ctrl, Alt, Shift, Insert, Windows, and the application key. This display also has two circular groups of four buttons, meant to be arrow keys.

The mentioned braille display function keys are especially handy if you combine them with other function keys or arrow keys. So when you press advance4 (ctrl) + advance1 (tab), you should be able to simulate the ctrl+tab gesture on the keyboard.

IN the current situation, every single combination has to be manually defined in NVDA. Thus, if you assign br(hims):advance3 to kb:control and br(hims):leftSideLeftArrow to kb:leftArrow, the combination advance3+leftSideLeftArrow isn't automatically assigned to kb:control+leftArrow, unless you specifically do this, either manually or in the driver itself. Especially the in-driver assignments consume a lot of code and are error-prone, since both the keyboard key and braille key assignments reside in strings and there is currently no meganism to test them other than with the braille display in front of you.

This leads to the idea of adding a way to use braille display keys assigned to keyboard modifier keys as modifier keys themselves. In the case of Hims, if advance3 is assigned to ctrl and advance4 to alt, br(hims):advance3+advance4+leftSideLeftArrow should sum up to kb:ctrl+alt+leftArrow without the need of manually assigning this combination. Using these modifier keys in other composite key strokes (containing arrow key emulators or even braille input combinations) should be possible without manual assignments. This would open great possibilities, pressing key strokes like ctrl+c, ctrl+v, ctrl+space without keyboard intervention and manual assignments. 

I understand that this proposal can conflict with current key assignments in certain display drivers, which is a point to discuss. I see the following options:
1. Always give assigned combinations precedense. When advance3=ctrl and leftSideLeftArrow=leftArrow, it should be possible to assign advance3+leftSideLeftArrow to something entirely different, like moving the review cursor left
2. Ignore those assigned combinations. If advance3+leftSideLeftArrow is assigned for moving the review cursor left, that assignment should be ignored and the combination should consistently simulate ctrl+leftArrow

In other words, point 1 stands for freedom, point 2 stands for concistency. Since I see reasons why both options could have preference, I would vote for making this an option in the braille settings dialog



Discussion:
Some research revealed the following:
1. Implementing combinations like advance3+dot1+dot2+dot3 = ctrl+l will be quite hard at first sight, as it will require the concatenation of two types of gestures, namely a brailleInput.BrailleInputGesture and a braille.BrailleDisplayGesture
2. The other type of composite commands wouldn't be that difficult. We can:
   - Implement a system in inputCore GlobalGestureMap to dynamically create new mappings for possible combinations. So, if advance3=control and leftSiteDownArrow=downArrow, dynamically create advance3+leftSideDownArrow=control+downArrow. This would have the advantage that all these dynamically created keys can be seen and removed in the input gestures dialog.
- Patch scriptHandler._getObjScript to see whether a composisite keyboard emulation script can be returned if scriptName is None. However, inputCore._AllGestureMappingsRetriever won't know them this way so it is not possible to review them in input gestures dialog, for example. This one is probably more ugly.


Resolving patch:
diff --git a/source/braille.py b/source/braille.py
--- a/source/braille.py
+++ b/source/braille.py
@@ -26,6 +26,7 @@
 import inputCore
 import brailleTables
 from collections import namedtuple
+import scriptHandler
 
 roleLabels = {
 	# Translators: Displayed in braille for an object which is a
@@ -1948,6 +1949,25 @@ def getPossiblePorts(cls):
 	#: @type: L{inputCore.GlobalGestureMap}
 	gestureMap = None
 
+	@classmethod
+	def _getModifierGestures(cls):
+		"""Retrieves modifier gestures from this display driver's L{gestureMap}
+		that are bound to modifier only keyboard emulate scripts.
+		@return: the ids of the display keys and the associated generalised modifier names
+		@rtype: generator of (set, set)
+		"""
+		import globalCommands
+		# Ignore the locale gesture map when searching for braille display gestures
+		globalMaps = [inputCore.manager.userGestureMap]
+		if cls.gestureMap:
+			globalMaps.append(cls.gestureMap)
+		for globalMap in globalMaps:
+			for scriptCls, gesture, scriptName in globalMap.getScriptsForAllGestures():
+				if gesture.startswith("br({source})".format(source=cls.name)) and scriptCls is globalCommands.GlobalCommands and scriptName.startswith("kb"):
+					emuGesture = keyboardHandler.KeyboardInputGesture.fromName(scriptName.split(":")[1])
+					if emuGesture.isModifier:
+						yield set(gesture.split(":")[1].split("+")), set(emuGesture._keyNamesInDisplayOrder)
+
 class BrailleDisplayGesture(inputCore.InputGesture):
 	"""A button, wheel or other control pressed on a braille display.
 	Subclasses must provide L{source} and L{id}.
@@ -1997,6 +2017,49 @@ def _get_scriptableObject(self):
 			return display
 		return super(BrailleDisplayGesture, self).scriptableObject
 
+	def _get_script(self):
+		# Overrides L{inputCore.InputGesture._get_script} to support modifier keys.
+		script=scriptHandler.findScript(self)
+		if script:
+			self.script = script
+			return self.script
+		# No script for this gesture has been found, so process this gesture for possible modifiers. 
+		# For example, if L{self.id} is 'key1+key2',
+		# key1 is bound to 'kb:control' and key2 to 'kb:tab',
+		# this gesture should execute 'kb:control+tab'.
+		# Combining modifiers with braille input (#7306) is not yet supported.
+		gestureKeys = set(self.keyNames)
+		gestureModifiers = set()
+		for keys, modifiers in handler.display._getModifierGestures():
+			if keys<gestureKeys:
+				gestureModifiers |= modifiers
+				gestureKeys -= keys
+		if not gestureModifiers:
+			# No modifier assignments found in this gesture.
+			return None
+		# Find a script for L{gestureKeys}.
+		fakeGestureId = u"br({source}):{id}".format(source=self.source, id="+".join(gestureKeys))
+		scriptNames = []
+		globalMaps = [inputCore.manager.userGestureMap, handler.display.gestureMap]
+		for globalMap in globalMaps:
+			scriptNames.extend(scriptName for cls, scriptName in globalMap.getScriptsForGesture(fakeGestureId) if scriptName.startswith("kb"))
+		if not scriptNames:
+			# Gesture contains modifiers, but no keyboard emulate script exists for the gesture without modifiers
+			return None
+		# We can't bother about multiple scripts for a gesture, we will just use the first one
+		scriptName = "kb:{modifiers}+{keys}".format(
+			modifiers="+".join(gestureModifiers),
+			keys=scriptNames[0].split(":")[1]
+		)
+		self.script = scriptHandler._makeKbEmulateScript(scriptName)
+		return self.script
+
+	def _get_keyNames(self):
+		"""The names of the keys that are part of this gesture.
+		@rtype: list
+		"""
+		return self.id.split("+")
+
 	@classmethod
 	def getDisplayTextForIdentifier(cls, identifier):
 		return handler.display.description, identifier.split(":", 1)[1]


Test patch: