Repository: nvaccess/nvda

Issue:
NVDA does not announce F-keys as Keystrokes of JMenuItems (Java Access Bridge)
<!-- Please read the text in this edit field before filling it in.
Please thoroughly read NVDA's wiki article on how to fill in this template, including how to provide the required files.
Issues may be closed if the required information is not present.
https://github.com/nvaccess/nvda/blob/master/devDocs/githubIssueTemplateExplanationAndExamples.md
Please also note that the NVDA project has a Citizen and Contributor Code of Conduct which can be found at https://github.com/nvaccess/nvda/blob/master/CODE_OF_CONDUCT.md. NV Access expects that all contributors and other community members read and abide by the rules set out in this document while participating or contributing to this project. This includes creating or commenting on issues and pull requests. 

Each of the questions and sections below start with multiple hash symbols (#). Place your answers and information on the blank line below each question.
-->
When selecting a JMenuItem which has an F-Key in its keystroke (accelerator key), NVDA does not announce the F-key although the information is transmitted via the Java Access Bridge. Other screen readers announce the F-key in a JMenuItem's keystroke. This issue affects also JCheckBoxMenuItem and JRadioButtonMenuItem as sub classes of JMenuItem.

If a keystroke is assigned to a JMenuItem, the user can fire the assigned action directly by hitting the keystroke. To use this quick access, the user needs to know the keystroke.

I wrote a simple application Test_JMenuItem_KeyStrokes.jar with a menu containing five JCheckBoxMenuItems with different keystrokes and attached it to this bug report.

### Steps to reproduce:
1. Extract the Test_JMenuItem_KeyStrokes.jar file from the linked .zip file: 
[Test_JMenuItem_KeyStrokes.zip](https://github.com/nvaccess/nvda/files/8564139/Test_JMenuItem_KeyStrokes.zip)
3. Execute the Test_JMenuItem_KeyStrokes.jar file which opens a simple frame with a checkbox and a single menu in the menu bar. (If you prefer a .java file, you can find it within the .jar file.)
-> A Java window with title "A Frame with Menu" opens.
-> The checkbox with label "A checkbox" has the keyboard focus.
4. Press F10 to open the single menu.
-> The menu "Menu with Keystrokes" opens and NVDA announces it and its keystroke Alt+M.
Note: You can also open this menu with its keystroke Alt+M.
-> The first menu item is selected and NVDA anounces it.
5. Navigate with arrow down/up keys within the menu.

### Actual behavior:
NVDA announces the menu items with its keystrokes as follows (leaving out the announced state information in detail):

- "First Menu Item", its state, "Ctrl a"
- "Second Menu Item", its state, "Ctrl Shift a"
- "Third Menu Item". its state
- "Fourth Menu Item", its state, "Ctrl"
- "Fifth Menu Item", its state, "Ctrl Shift"

Note, that the F-key (F4) is missing in the third, fourth and fifth menu item.

### Expected behavior:
For the third, fourth and fifth menu item, NVDA should announce the following:

- "Third Menu Item", its state, "F4"
- "Fourth Menu Item", its state, "Ctrl F4"
- "Fifth Menu Item", its state, "Ctrl Shift F4"

### System configuration
#### NVDA installed/portable/running from source:
installed

#### NVDA version:
Version: 2021.3.5

#### Windows version:
Version: 10.0.19042 Build 19042

#### Name and version of other software in use when reproducing the issue:
java version "18.0.1" 2022-04-19
Java(TM) SE Runtime Environment (build 18.0.1+10-24)
Java HotSpot(TM) 64-Bit Server VM (build 18.0.1+10-24, mixed mode, sharing)

#### Other information about your system:
Java Access Bridge is enabled.

### Other questions
#### Does the issue still occur after restarting your computer?
Yes.

#### Have you tried any other versions of NVDA? If so, please report their behaviors.
Yes. Same behavior.

#### If NVDA add-ons are disabled, is your problem still occurring?
Not applicable.

#### Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
Not applicable.



Discussion:

Resolving patch:
diff --git a/source/JABHandler.py b/source/JABHandler.py
--- a/source/JABHandler.py
+++ b/source/JABHandler.py
@@ -4,6 +4,7 @@
 # This file is covered by the GNU General Public License.
 # See the file COPYING for more details.
 
+from enum import IntEnum, IntFlag
 import os
 import queue
 from ctypes import (
@@ -226,14 +227,61 @@ class AccessibleTableCellInfo(Structure):
 	]
 
 MAX_KEY_BINDINGS=50
-ACCESSIBLE_SHIFT_KEYSTROKE=1
-ACCESSIBLE_CONTROL_KEYSTROKE=2
-ACCESSIBLE_META_KEYSTROKE=4
-ACCESSIBLE_ALT_KEYSTROKE=8
-ACCESSIBLE_ALT_GRAPH_KEYSTROKE=16
-ACCESSIBLE_BUTTON1_KEYSTROKE=32
-ACCESSIBLE_BUTTON2_KEYSTROKE=64
-ACCESSIBLE_BUTTON3_KEYSTROKE=128
+
+
+class AccessibleKeystroke(IntFlag):
+	"""
+	Defined in the JDK in header include/win32/bridge/AccessBridgePackages.h
+	"""
+	SHIFT = 1
+	CONTROL = 2
+	META = 4
+	ALT = 8
+	ALT_GRAPH = 16
+	BUTTON1 = 32
+	BUTTON2 = 64
+	BUTTON3 = 128
+	FKEY = 256  # F key pressed, character contains 1-24
+	CONTROLCODE = 512
+	"""
+	Control code key pressed, character contains control code.
+	Refer to AccessibleVK.
+	"""
+
+
+# Keep for backwards compatibility
+ACCESSIBLE_SHIFT_KEYSTROKE = AccessibleKeystroke.SHIFT
+ACCESSIBLE_CONTROL_KEYSTROKE = AccessibleKeystroke.CONTROL
+ACCESSIBLE_META_KEYSTROKE = AccessibleKeystroke.META
+ACCESSIBLE_ALT_KEYSTROKE = AccessibleKeystroke.ALT
+ACCESSIBLE_ALT_GRAPH_KEYSTROKE = AccessibleKeystroke.ALT_GRAPH
+ACCESSIBLE_BUTTON1_KEYSTROKE = AccessibleKeystroke.BUTTON1
+ACCESSIBLE_BUTTON2_KEYSTROKE = AccessibleKeystroke.BUTTON2
+ACCESSIBLE_BUTTON3_KEYSTROKE = AccessibleKeystroke.BUTTON3
+# Do not extend this list
+
+
+class AccessibleVK(IntEnum):
+	"""
+	The supported control code keys related to AccessibleKeystroke.CONTROLCODE.
+	Defined in the JDK in header include/win32/bridge/AccessBridgePackages.h
+	"""
+	BACK_SPACE = 8
+	DELETE = 127
+	DOWN = 40
+	END = 35
+	HOME = 36
+	INSERT = 155
+	KP_DOWN = 225
+	KP_LEFT = 226
+	KP_RIGHT = 227
+	KP_UP = 224
+	LEFT = 37
+	PAGE_DOWN = 34
+	PAGE_UP = 33
+	RIGHT = 39
+	UP = 38
+
 
 class AccessibleKeyBindingInfo(Structure):
 	_fields_=[
@@ -827,3 +875,46 @@ def terminate():
 	bridgeDll=None
 	windll.kernel32.FreeLibrary(h)
 	isRunning=False
+
+
+JABKeyControlCodesToLabels = {
+	AccessibleVK.UP: "uparrow",
+	AccessibleVK.DOWN: "downarrow",
+	AccessibleVK.LEFT: "leftarrow",
+	AccessibleVK.RIGHT: "rightarrow",
+	AccessibleVK.KP_UP: "numpad8",
+	AccessibleVK.KP_DOWN: "numpad2",
+	AccessibleVK.KP_LEFT: "numpad4",
+	AccessibleVK.KP_RIGHT: "numpad6",
+	AccessibleVK.BACK_SPACE: "backspace",
+	AccessibleVK.INSERT: "insert",
+	AccessibleVK.DELETE: "delete",
+	AccessibleVK.HOME: "home",
+	AccessibleVK.END: "end",
+	AccessibleVK.PAGE_UP: "pageup",
+	AccessibleVK.PAGE_DOWN: "pagedown"
+}
+
+# Do not include AccessibleKeystroke.FKEY_KEYSTROKE and AccessibleKeystroke.CONTROLCODE
+# as these are not really modifiers
+JABKeyModifiersToLabels = {
+	AccessibleKeystroke.BUTTON3: "button3",
+	AccessibleKeystroke.BUTTON2: "button2",
+	AccessibleKeystroke.BUTTON1: "button1",
+	AccessibleKeystroke.ALT_GRAPH: "altgraph",
+	AccessibleKeystroke.ALT: "alt",
+	AccessibleKeystroke.META: "meta",
+	AccessibleKeystroke.CONTROL: "control",
+	AccessibleKeystroke.SHIFT: "shift"
+}
+
+
+def _getKeyLabels(modifiers, character):
+	keys = [v for m, v in JABKeyModifiersToLabels.items() if modifiers & m]
+	if modifiers & AccessibleKeystroke.FKEY:
+		keys.append("F{}".format(ord(character)))
+	elif modifiers & AccessibleKeystroke.CONTROLCODE:
+		keys.append(JABKeyControlCodesToLabels.get(ord(character), character))
+	else:
+		keys.append(character)
+	return keys
diff --git a/source/NVDAObjects/JAB/__init__.py b/source/NVDAObjects/JAB/__init__.py
--- a/source/NVDAObjects/JAB/__init__.py
+++ b/source/NVDAObjects/JAB/__init__.py
@@ -18,6 +18,7 @@
 from .. import InvalidNVDAObject
 from locationHelper import RectLTWH
 
+
 JABRolesToNVDARoles={
 	"alert":controlTypes.Role.DIALOG,
 	"column header":controlTypes.Role.TABLECOLUMNHEADER,
@@ -95,6 +96,8 @@
 	"editable":controlTypes.State.EDITABLE,
 }
 
+
+
 re_simpleXmlTag=re.compile(r"\<[^>]+\>")
 
 
@@ -268,18 +271,23 @@ def _get_keyboardShortcut(self):
 		for index in range(bindings.keyBindingsCount):
 			binding=bindings.keyBindingInfo[index]
 			# We don't support these modifiers
-			if binding.modifiers&(JABHandler.ACCESSIBLE_META_KEYSTROKE|JABHandler.ACCESSIBLE_ALT_GRAPH_KEYSTROKE|JABHandler.ACCESSIBLE_BUTTON1_KEYSTROKE|JABHandler.ACCESSIBLE_BUTTON2_KEYSTROKE|JABHandler.ACCESSIBLE_BUTTON3_KEYSTROKE):
+			if binding.modifiers & (
+				JABHandler.AccessibleKeystroke.META
+				| JABHandler.AccessibleKeystroke.ALT_GRAPH
+				| JABHandler.AccessibleKeystroke.BUTTON1
+				| JABHandler.AccessibleKeystroke.BUTTON2
+				| JABHandler.AccessibleKeystroke.BUTTON3
+			):
 				continue
-			keyList=[]
+			modifiers = binding.modifiers
 			# We assume alt  if there are no modifiers at all and its not a menu item as this is clearly a nmonic
-			if (binding.modifiers&JABHandler.ACCESSIBLE_ALT_KEYSTROKE) or (not binding.modifiers and self.role!=controlTypes.Role.MENUITEM):
-				keyList.append(keyLabels.localizedKeyLabels['alt'])
-			if binding.modifiers&JABHandler.ACCESSIBLE_CONTROL_KEYSTROKE:
-				keyList.append(keyLabels.localizedKeyLabels['control'])
-			if binding.modifiers&JABHandler.ACCESSIBLE_SHIFT_KEYSTROKE:
-				keyList.append(keyLabels.localizedKeyLabels['shift'])
-			keyList.append(binding.character)
-		shortcutsList.append("+".join(keyList))
+			if not modifiers and self.role != controlTypes.Role.MENUITEM:
+				modifiers |= JABHandler.AccessibleKeystroke.ALT
+			keyList = [
+				keyLabels.localizedKeyLabels.get(l, l)
+				for l in JABHandler._getKeyLabels(modifiers, binding.character)
+			]
+			shortcutsList.append("+".join(keyList))
 		return ", ".join(shortcutsList)
 
 	def _get_name(self):


Test patch:
diff --git a/tests/unit/test_javaAccessBridge.py b/tests/unit/test_javaAccessBridge.py
--- a/tests/unit/test_javaAccessBridge.py
+++ b/tests/unit/test_javaAccessBridge.py
@@ -7,6 +7,8 @@
 
 import unittest
 from NVDAObjects import JAB
+import JABHandler
+from JABHandler import AccessibleKeystroke
 
 
 class TestJavaAccessBridge(unittest.TestCase):
@@ -26,3 +28,74 @@ def test_htmlStringHasTagsRemoved(self):
 		htmlStr = "<html><body><p>Some <b>bold</b> text.</p></body></html>"
 		expected = "   Some  bold  text.   "
 		self.assertEqual(expected, JAB._processHtml(htmlStr))
+
+
+MODIFIER_COMBINATIONS = [
+	{},
+	{AccessibleKeystroke.ALT: "alt"},
+	{AccessibleKeystroke.SHIFT: "shift"},
+	{AccessibleKeystroke.CONTROL: "control"},
+	{AccessibleKeystroke.CONTROL: "control", AccessibleKeystroke.SHIFT: "shift"},
+	{
+		AccessibleKeystroke.ALT: "alt",
+		AccessibleKeystroke.CONTROL: "control",
+		AccessibleKeystroke.SHIFT: "shift"
+	},
+	{AccessibleKeystroke.ALT: "alt", AccessibleKeystroke.SHIFT: "shift"},
+	{AccessibleKeystroke.ALT: "alt", AccessibleKeystroke.CONTROL: "control"},
+	{AccessibleKeystroke.ALT_GRAPH: "altgraph"},
+	{AccessibleKeystroke.META: "meta"},
+	{AccessibleKeystroke.BUTTON1: "button1"},
+	{AccessibleKeystroke.BUTTON2: "button2"},
+	{AccessibleKeystroke.BUTTON3: "button3"},
+	{
+		AccessibleKeystroke.BUTTON3: "button3",
+		AccessibleKeystroke.BUTTON2: "button2",
+		AccessibleKeystroke.BUTTON1: "button1",
+		AccessibleKeystroke.ALT_GRAPH: "altgraph",
+		AccessibleKeystroke.ALT: "alt",
+		AccessibleKeystroke.META: "meta",
+		AccessibleKeystroke.CONTROL: "control",
+		AccessibleKeystroke.SHIFT: "shift"
+	}
+]
+BASIC_SHORTCUT_KEYS = [chr(x) for x in range(ord('A'), ord('Z') + 1)]
+FKEY_SHORTCUTS = [chr(x) for x in range(1, 25)]
+
+
+class TestJavaAccessBridgeShortcutKeys(unittest.TestCase):
+	def testBasicShortcut(self):
+		for c in BASIC_SHORTCUT_KEYS:
+			for modifierCombination in MODIFIER_COMBINATIONS:
+				modifiers = 0
+				modLabels = []
+				for m, l in modifierCombination.items():
+					modifiers |= m
+					modLabels.append(l)
+				with self.subTest(character=c, modifiers=modifiers, modLabels=modLabels):
+					expected = modLabels + [c]
+					self.assertListEqual(expected, JABHandler._getKeyLabels(modifiers, c))
+
+	def testFKeyShortcut(self):
+		for c in FKEY_SHORTCUTS:
+			for modifierCombination in MODIFIER_COMBINATIONS:
+				modifiers = AccessibleKeystroke.FKEY
+				modLabels = []
+				for m, l in modifierCombination.items():
+					modifiers |= m
+					modLabels.append(l)
+				with self.subTest(fkey=ord(c), modifiers=modifiers, modLabels=modLabels):
+					expected = modLabels + ["F{}".format(ord(c))]
+					self.assertListEqual(expected, JABHandler._getKeyLabels(modifiers, c))
+
+	def testControlCodeShortcut(self):
+		for c, v in JABHandler.JABKeyControlCodesToLabels.items():
+			for modifierCombination in MODIFIER_COMBINATIONS:
+				modifiers = AccessibleKeystroke.CONTROLCODE
+				modLabels = []
+				for m, l in modifierCombination.items():
+					modifiers |= m
+					modLabels.append(l)
+				with self.subTest(controlCode=c, label=v, modifiers=modifiers, modLabels=modLabels):
+					expected = modLabels + [v]
+					self.assertListEqual(expected, JABHandler._getKeyLabels(modifiers, chr(c)))