Repository: nvaccess/nvda

Issue:
nvda alpha no longer works with math player
### Steps to reproduce:
with recent builds of nvda 2021.3 alpha, after commiting "make UI automation for recent office365 builds" install math type, (demo will be sufficient), and math player 4. 
try to read math, once you focus a math equasion, nvda just announces "graphic, math type x.x equasion"
### Actual behavior:
math is not recognized nor can it be navigated.
### Expected behavior:
NVDA should still be able to work with math in word
### System configuration
#### NVDA installed/portable/running from source:
installed
#### NVDA version:
(NVDA)
Version: alpha-23953,f8df880a

#### Windows version:
windows 11
#### Name and version of other software in use when reproducing the issue:
Math type 6.x or 7.x and math player 4
#### Other information about your system:
I think this is errelivant to the issue.
### Other questions
#### Does the issue still occur after restarting your computer?
yes
#### Have you tried any other versions of NVDA? If so, please report their behaviors.
yes, nvda 2021.2 works as expected as long as "use UI automation with word" is not checked. 
#### If NVDA add-ons are disabled, is your problem still occurring?
yes
#### Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
yes


Discussion:
cc @michaelDCurran 
There are two parts to this.
1. When NVDA uses UIA to access MS Word documents, there is no mechanism for NVDA to collect mathML from mathType equations, and I doubt there will be in the future. However, the user can highlight each mathtype equation graphic, and in the context menu, choose Convert to Office math. then, the user can use the arrow keys to move through the equation -- MS Word now does a pretty good job at exposing character navigation for equations, though it is still not as good as mathPlayer.
2. Insider builds of MS Word now do apparently have a UIA custom property which can be used to fetch raw mathML from an Office math equation. So when NVDA supports this custom property, the user could convert their mathtype equation to an Office math equation, and then NVDA should just handle it with mathplayer as it once did with the object model.  

As to what to do for NVDA 2021.3 however: I think it is looking more and more likely we should hold back on using UIA by default, or at least, limit it to a build of MS word with the mathML custom property. But, we would then have to work around the bug that breaks modern comments in MS Word for builds 13901 up to the build that supports the mathML custom property.
I shall do some further investigation on this. 
"MS Word now does a pretty good job at exposing character navigation for
equations, though it is still not as good as mathPlayer." Can you try to
elaborate on this a bit more? How does it lag behind mathplayer? what
should the user expect in case of reading MS word equations with NVDA?

Thank you.


On Mon, Oct 18, 2021 at 7:29 AM Michael Curran ***@***.***>
wrote:

> There are two parts to this.
>
>    1. When NVDA uses UIA to access MS Word documents, there is no
>    mechanism for NVDA to collect mathML from mathType equations, and I doubt
>    there will be in the future. However, the user can highlight each mathtype
>    equation graphic, and in the context menu, choose Convert to Office math.
>    then, the user can use the arrow keys to move through the equation -- MS
>    Word now does a pretty good job at exposing character navigation for
>    equations, though it is still not as good as mathPlayer.
>    2. Insider builds of MS Word now do apparently have a UIA custom
>    property which can be used to fetch raw mathML from an Office math
>    equation. So when NVDA supports this custom property, the user could
>    convert their mathtype equation to an Office math equation, and then NVDA
>    should just handle it with mathplayer as it once did with the object model.
>
> As to what to do for NVDA 2021.3 however: I think it is looking more and
> more likely we should hold back on using UIA by default, or at least, limit
> it to a build of MS word with the mathML custom property. But, we would
> then have to work around the bug that breaks modern comments in MS Word for
> builds 13901 up to the build that supports the mathML custom property.
> I shall do some further investigation on this.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/nvaccess/nvda/issues/12941#issuecomment-945332276>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABSCL7FCFQIG6JZHIU6HES3UHOIARANCNFSM5GDXQZIA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

> then, the user can use the arrow keys to move through the equation -- MS Word now does a pretty good job at exposing character navigation for equations, though it is still not as good as mathPlayer.


What about  displaying in braille - last time I looked MS Word has absolutely nothing to offer in that regard and that is one of the biggest selling points of Math Player?

Resolving patch:
diff --git a/source/NVDAObjects/UIA/wordDocument.py b/source/NVDAObjects/UIA/wordDocument.py
--- a/source/NVDAObjects/UIA/wordDocument.py
+++ b/source/NVDAObjects/UIA/wordDocument.py
@@ -5,6 +5,7 @@
 
 from comtypes import COMError
 from collections import defaultdict
+import mathPres
 from scriptHandler import isScriptWaiting
 import textInfos
 import eventHandler
@@ -24,6 +25,7 @@
 	WordDocument as WordDocumentBase,
 	WordDocumentTextInfo as LegacyWordDocumentTextInfo
 )
+from NVDAObjects import NVDAObject
 from scriptHandler import script
 
 
@@ -130,6 +132,12 @@ def label(self):
 
 class WordDocumentTextInfo(UIATextInfo):
 
+	def getMathMl(self, field):
+		mathml = field.get('mathml')
+		if not mathml:
+			raise LookupError("No MathML")
+		return mathml
+
 	def _ensureRangeVisibility(self):
 		try:
 			inView = self.pointAtStart in self.obj.location
@@ -192,6 +200,8 @@ def _getControlFieldForUIAObject(self, obj, isEmbedded=False, startOfNode=False,
 		elif obj.UIAElement.cachedControlType==UIAHandler.UIA_GroupControlTypeId and obj.name:
 			field['role']=controlTypes.Role.EMBEDDEDOBJECT
 			field['alwaysReportName']=True
+		elif obj.role == controlTypes.Role.MATH:
+			field['mathml'] = obj.mathMl
 		elif obj.UIAElement.cachedControlType==UIAHandler.UIA_CustomControlTypeId and obj.name:
 			# Include foot note and endnote identifiers
 			field['content']=obj.name
@@ -287,6 +297,32 @@ def getTextWithFields(self,formatConfig=None):
 		if len(fields)==0: 
 			# Nothing to do... was probably a collapsed range.
 			return fields
+
+		# MS Word tries to produce speakable math content within equations.
+		# However, using mathPlayer with the exposed mathml property on the equation is much nicer.
+		# But, we therefore need to remove the inner math content if reading by line
+		if not formatConfig or not formatConfig.get('extraDetail'):
+			# We really only want to remove content if we can guarantee that mathPlayer is available.
+			mathPres.ensureInit()
+			if mathPres.speechProvider or mathPres.brailleProvider:
+				curLevel = 0
+				mathLevel = None
+				mathStartIndex = None
+				mathEndIndex = None
+				for index in range(len(fields)):
+					field = fields[index]
+					if isinstance(field, textInfos.FieldCommand) and field.command == "controlStart":
+						curLevel += 1
+						if mathLevel is None and field.field.get('mathml'):
+							mathLevel = curLevel
+							mathStartIndex = index
+					elif isinstance(field, textInfos.FieldCommand) and field.command == "controlEnd":
+						if curLevel == mathLevel:
+							mathEndIndex = index
+						curLevel -= 1
+				if mathEndIndex is not None:
+					del fields[mathStartIndex + 1:mathEndIndex]
+
 		# Sometimes embedded objects and graphics In MS Word can cause a controlStart then a controlEnd with no actual formatChange / text in the middle.
 		# SpeakTextInfo always expects that the first lot of controlStarts will always contain some text.
 		# Therefore ensure that the first lot of controlStarts does contain some text by inserting a blank formatChange and empty string in this case.
@@ -368,16 +404,22 @@ def getTextWithFields(self,formatConfig=None):
 
 class WordBrowseModeDocument(UIABrowseModeDocument):
 
-	def shouldSetFocusToObj(self,obj):
+	def _shouldSetFocusToObj(self, obj: NVDAObject) -> bool:
 		# Ignore strange editable text fields surrounding most inner fields (links, table cells etc) 
 		if obj.role==controlTypes.Role.EDITABLETEXT and obj.UIAElement.cachedAutomationID.startswith('UIA_AutomationId_Word_Content'):
 			return False
-		return super(WordBrowseModeDocument,self).shouldSetFocusToObj(obj)
+		elif obj.role == controlTypes.Role.MATH:
+			# Don't set focus to math equations otherwise they cannot be interacted  with mathPlayer.
+			return False
+		return super()._shouldSetFocusToObj(obj)
 
 	def shouldPassThrough(self,obj,reason=None):
 		# Ignore strange editable text fields surrounding most inner fields (links, table cells etc) 
 		if obj.role==controlTypes.Role.EDITABLETEXT and obj.UIAElement.cachedAutomationID.startswith('UIA_AutomationId_Word_Content'):
 			return False
+		elif obj.role == controlTypes.Role.MATH:
+			# Don't  activate focus mode for math equations otherwise they cannot be interacted  with mathPlayer.
+			return False
 		return super(WordBrowseModeDocument,self).shouldPassThrough(obj,reason=reason)
 
 	def script_tab(self,gesture):
@@ -403,7 +445,16 @@ def _iterNodesByType(self,nodeType,direction="next",pos=None):
 class WordDocumentNode(UIA):
 	TextInfo=WordDocumentTextInfo
 
+	def _get_mathMl(self):
+		try:
+			return self._getUIACacheablePropertyValue(self._UIACustomProps.word_mathml.id)
+		except COMError:
+			pass
+		return None
+
 	def _get_role(self):
+		if self.mathMl:
+			return controlTypes.Role.MATH
 		role=super(WordDocumentNode,self).role
 		# Footnote / endnote elements currently have a role of unknown. Force them to editableText so that theyr text is presented correctly
 		if role==controlTypes.Role.UNKNOWN:
diff --git a/source/_UIACustomProps.py b/source/_UIACustomProps.py
--- a/source/_UIACustomProps.py
+++ b/source/_UIACustomProps.py
@@ -88,3 +88,11 @@ def __init__(self):
 			programmaticName="ItemCount",
 			uiaType=UIAutomationType.INT,
 		)
+
+		# A property for fetching raw MathML from an equation node in Microsoft Word.
+		# Available in MS Word build 14326 and higher.
+		self.word_mathml = CustomPropertyInfo(
+			guid=GUID("{FA170AB3-3229-4E7C-827F-DD05EE0481D9}"),
+			programmaticName="Word.MathML",
+			uiaType=UIAutomationType.STRING,
+		)


Test patch: