Repository: nvaccess/nvda

Issue:
support the new emoji panel in windows 10
### Steps to reproduce:
In windows 10 windows insider build 16215 Microsoft introduced the Emoji panel. 
1. focus on any text field and then press the windows key + ; or + . 
2. the emoji panel appears. 
### Expected behavior:
NVDA doesn't speak anything, and moving with the right and left keys moves between Emojies but NVDA doesn't read any of them. if you press enter the emoji would get entered as expected and announced.
### Actual behavior:
NVDA doesn't tell us that the panel has opned nor does it read when moving between items. Note that Narrator reads those fine as expected.
### System configuration:
NVDA version:
NVDA Version: next-14106,059e7f02

NVDA Installed or portable:
installed

Other information:
none needed

Windows version:
Windows Version 1703 (OS Build 16215.1000)

the other questions in the template don't apply here.


Discussion:
Hi, partly because we need to find out if the new XAML touch input panel fires useful events and announce changes if any. CC @MichaelDCurran

 

From: Mohammed Al Shara [mailto:notifications@github.com] 
Sent: Sunday, June 11, 2017 3:24 AM
To: nvaccess/nvda <nvda@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
Subject: [nvaccess/nvda] support the new emoji panel in windows 10 (#7273)

 


Steps to reproduce:


In windows 10 windows insider build 16215 Microsoft introduced the Emoji panel.

1.	focus on any text field and then press the windows key + ; or + .
2.	the emoji panel appears.


Expected behavior:


NVDA doesn't speak anything, and moving with the right and left keys moves between Emojies but NVDA doesn't read any of them. if you press enter the emoji would get entered as expected and announced.


Actual behavior:


NVDA doesn't tell us that the panel has opned nor does it read when moving between items. Note that Narrator reads those fine as expected.


System configuration:


NVDA version:
NVDA Version: next-14106,059e7f02

NVDA Installed or portable:
installed

Other information:
none needed

Windows version:
Windows Version 1703 (OS Build 16215.1000)

the other questions in the template don't apply here.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <https://github.com/nvaccess/nvda/issues/7273> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkAuT1WKls9FDnqfppS_coloDfqWqks5sC8A0gaJpZM4N2WjF> .


I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard.

The emoji panel does work with Narrator.

The emoji panel is beneath a top level window named "Microsoft Text Input Application". [This MSDN article about IMEs in Windows Store apps](https://msdn.microsoft.com/en-us/library/windows/apps/hh967425.aspx#accessibility) says the following about accessibility:

> Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention.
> 
> - The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates.
> - When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively
> - When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE.
> - The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId.

However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids:

1. There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList.
2. The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji.
3. The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works.
4. The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters.
5. When you press tab, the selected item in the second list changes (corresponding to the selected category).
6. Note that Narrator doesn't report the categories in any way.

CC @MichaelDCurran.
Hi, yes, element selected event is indeed raised. I’ll come up with a temporary workaround in the WinTenApps add-on. Thanks.

 

From: James Teh [mailto:notifications@github.com] 
Sent: Sunday, June 11, 2017 3:52 PM
To: nvaccess/nvda <nvda@noreply.github.com>
Cc: Joseph Lee <joseph.lee22590@gmail.com>; Comment <comment@noreply.github.com>
Subject: Re: [nvaccess/nvda] support the new emoji panel in windows 10 (#7273)

 

I looked into this briefly a couple of days ago. Note that this can be used even if you don't use the touch keyboard.

The emoji panel does work with Narrator.

The emoji panel is beneath a top level window named "Microsoft Text Input Application". This MSDN article about IMEs in Windows Store apps <https://msdn.microsoft.com/en-us/library/windows/apps/hh967425.aspx#accessibility>  says the following about accessibility:

Implement the following convention to make your IMEs conform to the accessibility requirements and to work with Narrator. To make candidate lists accessible, your IMEs must follow this convention.

*	The candidate list must have a UIA_AutomationIdPropertyId equal to "IME_Candidate_Window" for lists of conversion candidates or "IME_Prediction_Window" for lists of prediction candidates.
*	When the candidate list appears and disappears, it raises events of type UIA_MenuOpenedEventId and UIA_MenuClosedEventId, respectively
*	When the current selected candidate changes, the candidate list raises a UIA_SelectionItem_ElementSelectedEventId. The selected element should have a property UIA_SelectionItemIsSelectedPropertyId equal to TRUE.
*	The UIA_NamePropertyId for each item in the candidate list must be the name of the candidate. Optionally, you can provide additional information to disambiguate candidates through UIA_HelpTextPropertyId.

However, it seems Microsoft doesn't follow their own guidelines here with regard to automation ids:

1.	There are two lists in that window. The first list has an automation id of TEMPLATE_PART_ExpressionFullViewItemsGrid. The second has an automation id of TEMPLATE_PART_ExpressionFullViewGroupsList.
2.	The first list is the list of emoji. The name of each item in this lit is the Unicode representation of the emoji.
3.	The selected item in the first list does change when you use the arrow keys. That means it probably does raise selection events. I think Narrator probably listens for selection events globally, which explains how that works.
4.	The second is the list of categories.Unfortunately, the category list is all private use (and thus not standardised) Unicode characters.
5.	When you press tab, the selected item in the second list changes (corresponding to the selected category).
6.	Note that Narrator doesn't report the categories in any way.

CC @michaelDCurran <https://github.com/michaeldcurran> .

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <https://github.com/nvaccess/nvda/issues/7273#issuecomment-307662382> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkBvEEImKFNu715dhZZOgr--uliXBks5sDG-VgaJpZM4N2WjF> .  <https://github.com/notifications/beacon/AHgLkI3mjWLjQoneQdU1kZ2IiXUqo84gks5sDG-VgaJpZM4N2WjF.gif> 


Hi,

Technical: these are "floating suggestion items" in that they are not part of the app in question. Rather, they appear as part of a floating window with its own app executable name. Numerous solutions come to my mind:

* #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).
* A request event for UIA objects should be added for the app module for the new XAML input panel. Among other things, this opens up numerous possibilities such as working with floating windows.

Important note: the implementation @JCSTeh described and I wrote a workaround for can change in future Insider builds.

Thanks.
Hi,

Although the suggestions list item is in master, I'll hold off until at least two slow ring builds of Fall Creators Update Insider Preview are released in order to gauge how it'll work in the wild. Thanks.
in build 16237 some changes were made to the Emoji panel, in particular this may be beneficial to the implementation of this ticket:

"• Have you ever wondered “What the heck is this emoji supposed to be”? Wonder no more! If you hover your mouse over any of the emoji in the Emoji Panel, you’ll now see a handy showing you the exact Unicode character name. Bonus: we’ve added tooltips to the categories so they’re no longer a mystery."
Experientially speaking, in build 16237, there appears to be no difference in how Narrator or NVDA treat the emoji panel than originally reported by @jcsteh. I filed a bug in the Feedback Center last week. See the following. https://aka.ms/fk2cna

Hi, and a possible modification might be in order, as the panel title is announced when I use arrow keys (at least the implementation in my add-on may require some edits). Thanks.

 

From: PratikP1 [mailto:notifications@github.com] 
Sent: Saturday, July 8, 2017 3:19 AM
To: nvaccess/nvda <nvda@noreply.github.com>
Cc: Joseph Lee <joseph.lee22590@gmail.com>; Comment <comment@noreply.github.com>
Subject: Re: [nvaccess/nvda] support the new emoji panel in windows 10 (#7273)

 

Experientially speaking, in build 16237, there appears to be no difference in how Narrator or NVDA treat the emoji panel than originally reported by @jcsteh. I filed a bug in the Feedback Center last week. See the following. https://aka.ms/fk2cna 



—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <https://github.com/nvaccess/nvda/issues/7273#issuecomment-313847389> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgLkAuAJN3aEW6nDCSwrkhKHjTDMTtHks5sL1ehgaJpZM4N2WjF> .


Here is a log snippet of what NVDA says when Win 10 Apps Essential 2017-07-05-dev is running. Window Title is not announced There’s nothing without the add-on running.

IO - inputCore.InputManager.executeGesture (12:20:07.668):
Input: kb(laptop):windows+.
IO - inputCore.InputManager.executeGesture (12:20:09.502):
Input: kb(laptop):tab
IO - speech.speak (12:20:09.595):
Speaking [LangChangeCommand (u'en_US'), u'1 inches']
IO - speech.speak (12:20:09.693):
Speaking [LangChangeCommand (u'en_US'), u'blank']
IO - speech.speak (12:20:10.374):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f600  1 of 256']
IO - inputCore.InputManager.executeGesture (12:20:12.174):
Input: kb(laptop):shift+tab
IO - speech.speak (12:20:12.278):
Speaking [LangChangeCommand (u'en_US'), u'1 inches']
IO - speech.speak (12:20:12.394):
Speaking [LangChangeCommand (u'en_US'), u'blank']
IO - speech.speak (12:20:12.746):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f602  1 of 45']
IO - inputCore.InputManager.executeGesture (12:20:14.892):
Input: kb(laptop):rightArrow
IO - speech._speakSpellingGen (12:20:15.151):
Speaking character u'line feed'
IO - speech.speak (12:20:15.158):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f600  2 of 45']
IO - inputCore.InputManager.executeGesture (12:20:15.964):
Input: kb(laptop):rightArrow
IO - speech._speakSpellingGen (12:20:16.204):
Speaking character u'line feed'
IO - speech.speak (12:20:16.213):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f601  3 of 45']
IO - inputCore.InputManager.executeGesture (12:20:17.252):
Input: kb(laptop):rightArrow
IO - speech._speakSpellingGen (12:20:17.484):
Speaking character u'line feed'
IO - speech.speak (12:20:17.492):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f923  4 of 45']
IO - inputCore.InputManager.executeGesture (12:20:17.948):
Input: kb(laptop):tab
IO - speech.speak (12:20:18.045):
Speaking [LangChangeCommand (u'en_US'), u'1 inches']
IO - speech.speak (12:20:18.147):
Speaking [LangChangeCommand (u'en_US'), u'blank']
IO - speech.speak (12:20:18.783):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f600  1 of 256']
IO - inputCore.InputManager.executeGesture (12:20:20.069):
Input: kb(laptop):tab
IO - speech.speak (12:20:20.177):
Speaking [LangChangeCommand (u'en_US'), u'1 inches']
IO - speech.speak (12:20:20.285):
Speaking [LangChangeCommand (u'en_US'), u'blank']
IO - speech.speak (12:20:21.020):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f469  1 of 175']
IO - inputCore.InputManager.executeGesture (12:20:22.428):
Input: kb(laptop):tab
IO - speech.speak (12:20:22.526):
Speaking [LangChangeCommand (u'en_US'), u'1 inches']
IO - speech.speak (12:20:22.638):
Speaking [LangChangeCommand (u'en_US'), u'blank']
IO - speech.speak (12:20:23.562):
Speaking [LangChangeCommand (u'en_US'), u'\U0001f388  1 of 273']
IO - inputCore.InputManager.executeGesture (12:20:25.029):
Input: kb(laptop):end
IO - speech._speakSpellingGen (12:20:25.796):
Speaking character u'line feed'
IO - inputCore.InputManager.executeGesture (12:20:27.059):
Input: kb(laptop):leftArrow
IO - speech._speakSpellingGen (12:20:27.286):
Speaking character u'line feed'
IO - speech.speak (12:20:27.296):
Speaking [LangChangeCommand (u'en_US'), u'\u23f2  272 of 273']


@josephsl, I think this would be good to support in core. If you'd be willing to submit a PR, that'd be great. We should think about the best implementation, though:

[**@josephsl**](https://github.com/josephsl) commented on [12 Jun 2017, 12:05 GMT+10](https://github.com/nvaccess/nvda/issues/7273#issuecomment-307674396 "2017-06-12T02:05:14Z - Replied by Github Reply Comments"):
> * #6241: I can add a new FloatingSuggestionsListItem class that will do what SuggestionsListItem does except for checking for controller for event (this is the solution employed in latest WinTenApps snapshot as of June 12th).

1. Any thoughts on the likelihood of other floating suggestions like this?
2. How do you determine what should get this behaviour? Do you use the automation id or similar or is it much more generic? I'd be concerned about unintentional reporting if it were too generic.
3. Is "suggestions" the right way to think about this? This is probably more like candidates in an input method.

> * A request event for UIA objects should be added for the app module for the new XAML input panel.

Does this need requestEvent? Is it blocked by shouldAcceptEvent by default?

I'm inclined to go more for an app module specific implementation here. Aside from the open questions above, handling the categories properly will require specific code, I think.
Hi,

App module method: I'll report back if I find anything new (using slow ring 16232 on my physical desktop, which should make it easier for me to test this).
Candidate or not: I see why it could be labeled that way. If so, I guess it makes implementation super easy unless @MichaelDCurran says otherwise.
Other floating suggestions: not that I know of. I'll ask a contact from MS Shell team about this, as well as asking if the Emoji panel implementation is stable, as the answer to that changes when the PR would be ready; if the panel code is stable, I can clean things up and send a PR as early as Tuesday, if not, I can wait until September.

Thanks.

Resolving patch:
diff --git a/source/appModules/windowsinternal_composableshell_experiences_textinput_inputapp.py b/source/appModules/windowsinternal_composableshell_experiences_textinput_inputapp.py
new file mode 100644
--- /dev/null
+++ b/source/appModules/windowsinternal_composableshell_experiences_textinput_inputapp.py
@@ -0,0 +1,102 @@
+# App module for Composable Shell (CShell) input panel
+#A part of NonVisual Desktop Access (NVDA)
+#Copyright (C) 2017-2018 NV Access Limited, Joseph Lee
+#This file is covered by the GNU General Public License.
+#See the file COPYING for more details.
+
+"""App module for Windows 10 Modern Keyboard aka new touch keyboard panel.
+The chief feature is allowing NVDA to announce selected emoji when using the keyboard to search for and select one.
+Another feature is to announce candidates for misspellings if suggestions for hardware keyboard is selected.
+This is applicable on Windows 10 Fall Creators Update and later."""
+
+import appModuleHandler
+import api
+import speech
+import braille
+import ui
+import config
+import winVersion
+from NVDAObjects.UIA import UIA
+
+class AppModule(appModuleHandler.AppModule):
+
+	# Cache the most recently selected item.
+	_recentlySelected = None
+
+	def event_UIA_elementSelected(self, obj, nextHandler):
+		# #7273: When this is fired on categories, the first emoji from the new category is selected but not announced.
+		# Therefore, move the navigator object to that item if possible.
+		# However, in recent builds, name change event is also fired.
+		# For consistent experience, report the new category first by traversing through controls.
+		# #8189: do not announce candidates list itself (not items), as this is repeated each time candidate items are selected.
+		if obj.UIAElement.cachedAutomationID == "CandidateList": return
+		speech.cancelSpeech()
+		# Sometimes clipboard candidates list gets selected, so ask NvDA to descend one more level.
+		if obj.UIAElement.cachedAutomationID == "TEMPLATE_PART_ClipboardItemsList":
+			obj = obj.firstChild
+		candidate = obj
+		# Sometimes, due to bad tree traversal, something other than the selected item sees this event.
+		parent = obj.parent
+		if obj.UIAElement.cachedClassName == "ListViewItem" and isinstance(parent, UIA) and parent.UIAElement.cachedAutomationID != "TEMPLATE_PART_ClipboardItemsList":
+			# The difference between emoji panel and suggestions list is absence of categories/emoji separation.
+			# Turns out automation ID for the container is different, observed in build 17666 when opening clipboard copy history.
+			candidate = obj.parent.previous
+			if candidate is not None:
+				# Emoji categories list.
+				ui.message(candidate.name)
+				obj = candidate.firstChild
+		if obj is not None:
+			api.setNavigatorObject(obj)
+			obj.reportFocus()
+			braille.handler.message(braille.getBrailleTextForProperties(name=obj.name, role=obj.role, positionInfo=obj.positionInfo))
+			# Cache selected item.
+			self._recentlySelected = obj.name
+		else:
+			# Translators: presented when there is no emoji when searching for one in Windows 10 Fall Creators Update and later.
+			ui.message(_("No emoji"))
+		nextHandler()
+
+	def event_UIA_window_windowOpen(self, obj, nextHandler):
+		# Make sure to announce most recently used emoji first in post-1709 builds.
+		# Fake the announcement by locating 'most recently used" category and calling selected event on this.
+		# However, in build 17666 and later, child count is the same for both emoji panel and hardware keyboard candidates list.
+		if winVersion.winVersion.build <= 17134 and obj.childCount == 3:
+			self.event_UIA_elementSelected(obj.lastChild.firstChild, nextHandler)
+		# Handle hardware keyboard suggestions.
+		# Treat it the same as CJK composition list - don't announce this if candidate announcement setting is off.
+		# This is also the case for emoji panel in build 17666 and later.
+		elif obj.childCount == 1:
+			childAutomationID = obj.firstChild.UIAElement.cachedAutomationID
+			if childAutomationID == "CandidateWindowControl" and config.conf["inputComposition"]["autoReportAllCandidates"]:
+				try:
+					self.event_UIA_elementSelected(obj.firstChild.firstChild.firstChild, nextHandler)
+				except AttributeError:
+					# Because this is dictation window.
+					pass
+			# Emoji panel in build 17666 and later (unless this changes).
+			elif childAutomationID == "TEMPLATE_PART_ExpressionGroupedFullView":
+				self._emojiPanelJustOpened = True
+				self.event_UIA_elementSelected(obj.firstChild.firstChild.next.next.firstChild.firstChild, nextHandler)
+		nextHandler()
+
+	# Argh, name change event is fired right after emoji panel opens in build 17666 and later.
+	_emojiPanelJustOpened = False
+
+	def event_nameChange(self, obj, nextHandler):
+		# #49: reported by a user: on some systems, touch keyboard keys keeps firing name change event.
+		# Argh, in build 17704, whenever skin tones are selected, name change is fired by emoji entries (GridViewItem).
+		if ((obj.UIAElement.cachedClassName in ("CRootKey", "GridViewItem"))
+		or (obj.UIAElement.cachedAutomationID == "TEMPLATE_PART_ClipboardItemsList")
+		# And no, emoji entries should not be announced here.
+		or (self._recentlySelected is not None and self._recentlySelected in obj.name)):
+			return
+		# The word "blank" is kept announced, so suppress this on build 17666 and later.
+		if winVersion.winVersion.build > 17134:
+			# In build 17672 and later, return immediatley when element selected event on clipboard item was fired just prior to this.
+			if obj.UIAElement.cachedAutomationID == "TEMPLATE_PART_ClipboardItemIndex" or obj.parent.UIAElement.cachedAutomationID == "TEMPLATE_PART_ClipboardItemsList": return
+			if not self._emojiPanelJustOpened or obj.UIAElement.cachedAutomationID != "TEMPLATE_PART_ExpressionGroupedFullView": speech.cancelSpeech()
+			self._emojiPanelJustOpened = False
+		# Don't forget to add "Microsoft Candidate UI" as something that should be suppressed.
+		if obj.UIAElement.cachedAutomationID not in ("TEMPLATE_PART_ExpressionFullViewItemsGrid", "TEMPLATE_PART_ClipboardItemIndex", "CandidateWindowControl"):
+			ui.message(obj.name)
+		nextHandler()


Test patch: