Repository: nvaccess/nvda

Issue:
LibreOffice Writer: Unusable coordinates announced
### Steps to reproduce:
1. Open Writer.
2. Working with screenreader NVDA hit NVDA + Delete.

### Actual behavior:
The Screeader announces a two part number, withou anyone knoing what this is about to be.

### Expected behavior:
A clear position shound be announced (lef, right, high and low) position should be announced.

### System configuration
#### NVDA installed/portable/running from source:
installed
#### NVDA version:
2020.2

#### Windows version:
Edition	Windows 	10 Home
Version				1909
Operating system build	18363.1016

#### Name and version of other software in use when reproducing the issue:
LibreOffice
Version:	7.0.0.3 (x64)
Build ID:	8061b3e9204bef6b321a21033174034a5e2ea88e
CPU-Threads: 	4; BS: Windows 10.0 Build 18363; UI-Render: Skia/Vulkan; VCL: win
Locale: 	de-DE (de_DE); UI: de-DE
Calc: 	threaded

#### Other information about your system:
---
### Other questions
#### Does the issue still occur after restarting your computer?
Yes.

#### Have you tried any other versions of NVDA? If so, please report their behaviors.
No.

#### If addons are disabled, is your problem still occuring?
Not tested.

#### Did you try to run the COM registry fixing tool in NVDA menu / tools?
Not tested.



Discussion:
The issue has also been reported to the Document Foundation; see https://bugs.documentfoundation.org/show_bug.cgi?id=136760
> ### Steps to reproduce:
> 
>     1. Open Writer.
> 
>     2. Working with screenreader NVDA hit NVDA + Delete.

Note to those not so familiar with NVDA (like myself): shortcut is NVDA + **numpadDelete** on desktop computers (and at least for myself on my laptop as well).
NVDA user documentation at https://www.nvaccess.org/files/nvda/documentation/userGuide.html#ObjectNavigation says this about the functionality:


Name | Desktop key | Laptop key | Touch | Description
-- | -- | -- | -- | --
Report review cursor location | NVDA+numpadDelete | NVDA+delete | none | Reports information about the location of the text or object at the  review cursor. For example, this might include the percentage through  the document, the distance from the edge of the page or the exact screen  position. Pressing twice may provide further detail.

In a quick test with Word, NVDA e.g. says "X inches from left edge of page, Y inches from top edge of page"

Hello Michael,

the NVDA key combination actually works in Word as you describe. But in
LibreOffice Writer, unfortunately, this procedure does not work. LO does
provide "coordinates", but these seem to be some kind of pixel coordinates
and would have to be converted into centimetres/inches to get a similar
result as in Word. The "conversion" is therefore the problem...
Presumably LO Writer would have to query system data (how big the screen
and its resolution is) so that you can finally get centimetre/inch
information - but I'm not sure.
Unfortunately, only the current coordinates do not help a blind person in
LO Writer...

With best regards


Am Mi., 27. Okt. 2021 um 16:22 Uhr schrieb Michael Weghorn <
***@***.***>:

> Steps to reproduce:
>
> 1. Open Writer.
>
> 2. Working with screenreader NVDA hit NVDA + Delete.
>
> Note to those not so familiar with NVDA (like myself): shortcut is NVDA +
> *numpadDelete* on desktop computers (and at least for myself on my laptop
> as well).
> NVDA user documentation at
> https://www.nvaccess.org/files/nvda/documentation/userGuide.html#ObjectNavigation
> says this about the functionality:
> Name Desktop key Laptop key Touch Description
> Report review cursor location NVDA+numpadDelete NVDA+delete none Reports
> information about the location of the text or object at the review cursor.
> For example, this might include the percentage through the document, the
> distance from the edge of the page or the exact screen position. Pressing
> twice may provide further detail.
>
> In a quick test with Word, NVDA e.g. says "X inches from left edge of
> page, Y inches from top edge of page"
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/nvaccess/nvda/issues/11696#issuecomment-952981466>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARFL6CAYHDSCPHO55XX7ZW3UJAKJ5ANCNFSM4R4KF22Q>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

From a first analysis, I haven't seen any way to expose/retrieve the information at what position in the page the cursor currently is via the accessibility interfaces (`IAccessible2` in the case of LibreOffice).

The `IAccessibleComponent` interface has a way to [get the position in pixels relative to the parent](https://accessibility.linuxfoundation.org/a11yspecs/ia2/docs/html/interface_i_accessible_component.html#a8709fdbf3f51ae079af274854a8bffd4) , but that's not really helpful here. From a first look, the Writer page itself also has no a11y object in the a11y tree.

What NVDA seems to do for MS Word is to use the Microsoft Office COM API to interact with MS Office, not the accessibility APIs:
https://github.com/nvaccess/nvda/blob/d2d4bfce2d31de8fca3c6df16caf93350b26374d/source/NVDAObjects/window/winword.py#L597-L614

It queries for the `wdHorizontalPositionRelativeToPage` and `wdVerticalPositionRelativeToPage` attributes described here:
https://learn.microsoft.com/en-us/office/vba/api/word.wdinformation

This page has some more infos on using the COM with NVDA and Word:
https://github.com/nvaccess/nvda/wiki/Using-COM-with-NVDA-and-Microsoft-Word

So if there's no way to retrieve the information via the accessibility APIs, I see no "easy" way of implementing this to work with LibreOffice Writer as well.

What *might* work is to not use the accessibility APIs, but LibreOffice's UNO API (which also has a Python binding), but adding support for doing this would presumably be quite a lot of work.

Possibly, there are ways to retrieve the required information (or retrieve information from which the position in the page could then be calculated). Potentially helpful UNO interfaces might be (but that would need a closer look):

* `XTextViewCursor::getPosition()` to retrieve cursor pos relative to top left position of the first page of the document
* `XPageCursor::getPage()` can be used to retrieve the page that the cursor is currently on
* `offapi/com/sun/star/style/PageProperties.idl` to access properties of the page,...
* ...

Looking into the implementation of `SwXTextViewCursor::getPosition` (which is the LO-internal implementation of one of the interfaces) might give some more ideas.

If that's an approach to investigate further, it would probably be a good idea to start with a macro to run from within LO to see whether it's possible to calculate the position from the information accessible via the UNO API.
cc: @michaelDCurran, @seanbudd maybe you can assist with some hints here.
This is what I could find for now:

```
190
/** These constants define which coordinate system a point is located in.
 This enum is used in IAccessible2::scrollToPoint, IAccessibleImage::imagePosition,
 IAccessibleText::characterExtents, and IAccessibleText::offsetAtPoint, and 
 IAccessibleText::scrollSubstringToPoint.
*/

enum IA2CoordinateType {

  /// The coordinates are relative to the screen.
  IA2_COORDTYPE_SCREEN_RELATIVE, 

  /** The coordinates are relative to the upper left corner of the bounding box
   of the immediate parent. */
  IA2_COORDTYPE_PARENT_RELATIVE

/** Special offsets for use in IAccessibleText and IAccessibleEditableText methods
  Refer to @ref _specialOffsets 
  "Special Offsets for use in the IAccessibleText and IAccessibleEditableText Methods" 
  for more information.
*/

enum IA2TextSpecialOffsets {
  IA2_TEXT_OFFSET_LENGTH = -1,	/**< This offset is equivalent to the length of the string.  It eliminates
								 the need to call IAccessibleText::nCharacters. */

  IA2_TEXT_OFFSET_CARET = -2	/**< This offset signifies that the text related to the physical location
								 of the caret should be used. */
````

Source:
https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/iaccessible2/ia2_api_all.idl

This one here is also very explanatory and quite good for understanding the API in my opinion:
https://opensource.apple.com/source/WebKit/WebKit-7601.6.17/win/Interfaces/Accessible2/AccessibleText.idl.auto.html

@Adriani90 : Thanks! I've briefly looked at this before and more intensely now, but I still don't see any methods that could be used for the purpose of getting the position relative to the document page. I might be missing something though, and would be very thankful for any hints/suggestions on how to do it.
Hmm, yeah or unless you define the document page as the parent object of the editable text. then you could retrieve the coordinates relative to the parent object. But not sure how the document page is currently located in the object hierarchy in Libreoffice.

 

 

Von: Michael Weghorn ***@***.***> 
Gesendet: Freitag, 10. März 2023 20:50
An: nvaccess/nvda ***@***.***>
Cc: Adriani90 ***@***.***>; Mention ***@***.***>
Betreff: Re: [nvaccess/nvda] LibreOffice Writer: Unusable coordinates announced (#11696)

 

@Adriani90 <https://github.com/Adriani90>  : Thanks! I've briefly looked at this before and more intensely now, but I still don't see any methods that could be used for the purpose of getting the position relative to the document page. I might be missing something though, and would be very thankful for any hints/suggestions on how to do it.

—
Reply to this email directly, view it on GitHub <https://github.com/nvaccess/nvda/issues/11696#issuecomment-1464332411> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGVCP4OSNQN3UWRVIS6RQWDW3OAXJANCNFSM4R4KF22Q> .
You are receiving this because you were mentioned.  <https://github.com/notifications/beacon/AGVCP4LJVBBRK54RRE7K27LW3OAXJA5CNFSM4R4KF222YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOK5D7A6Y.gif> Message ID: ***@***.*** ***@***.***> >


> Hmm, yeah or unless you define the document page as the parent object of the editable text. then you could retrieve the coordinates relative to the parent object. But not sure how the document page is currently located in the object hierarchy in Libreoffice.

From what I can see, there is no a11y object representing a page in the a11y tree for a Writer document at all.
> From what I can see, there is no a11y object representing a page in the a11y tree for a Writer document at all.
>

That's interesting. I tried to find out how NVDA interprets what a page in Libreoffice is but I couldn't find any useful information. For now NVDA reports the current page while navigating through the document, so it must retrieve the page information from somewhere.
> For now NVDA reports the current page while navigating through the document, so it must retrieve the page information from somewhere.

That's a very helpful hint! I looked into what happens for page reporting. As far as I can see, this is done via custom attributes on the document object:

https://github.com/nvaccess/nvda/blob/d2d4bfce2d31de8fca3c6df16caf93350b26374d/source/appModules/soffice.py#L115-L127

From a quick test, it should technically be possible to do the same for the relative cursor position in the document.

Demo change in LibreOffice to expose two new attributes for horizontal and vertical cursor position (change on top of current git master as of commit 9b38beadf9eaf027b201cdf0ecb2bce5611014dd):

```
diff --git a/sw/source/core/access/accdoc.cxx b/sw/source/core/access/accdoc.cxx
index 618425a68f9d..c188b64cec5b 100644
--- a/sw/source/core/access/accdoc.cxx
+++ b/sw/source/core/access/accdoc.cxx
@@ -533,6 +533,22 @@ uno::Any SAL_CALL SwAccessibleDocument::getExtendedAttributes()
             ";total-pages:" +
             OUString::number( pCursorShell->GetPageCnt() ) + ";";

+
+        // DEMO for nvda#11696/tdf#136760: set current cursor pos coordinates as extended attributes
+        Point aCursorPos = pFEShell->GetCursorDocPos();
+        // NOTE: maybe the current implementation could be optimized from
+        // iterating over all the pages to passing the correct page right away,
+        // or just subtracting the cursor pos from the base pos of the current
+        // page, since that is retrieved anyway, s. the "page-number" attribute
+        // below
+        Point aRelativePagePos = pFEShell->GetRelativePagePosition(aCursorPos);
+        // TODO: use and set attribute to sth that makes sense
+        // TODO: consider performance impact
+        SAL_WARN("mw.test", "relative page pos: " << aRelativePagePos);
+        sValue += "cursor-horizontal-position-in-page:" + OUString::number(aRelativePagePos.getX())
+                + ";cursor-vertical-position-in-page:" + OUString::number(aRelativePagePos.getY()) + ";";
+
+
         SwContentFrame* pCurrFrame = pCursorShell->GetCurrFrame();
         SwPageFrame* pCurrPage=static_cast<SwFrame*>(pCurrFrame)->FindPageFrame();
         sal_uLong nLineNum = 0;
```
Demo change in NVDA (on top of git master as of d2d4bfce2d31de8fca3c6df16caf93350b26374d):

```
diff --git a/source/appModules/soffice.py b/source/appModules/soffice.py
index e007570ad..ccedb0f66 100755
--- a/source/appModules/soffice.py
+++ b/source/appModules/soffice.py
@@ -122,6 +122,10 @@ class SymphonyTextInfo(IA2TextTextInfo):
                                pass
                        else:
                                try:
+                                       # NOTE: This is where the page number is retrieved from^M
+                                       # extended attribute for doc^M
+                                       # s.a. SwAccessibleDocument::getExtendedAttributes which looks^M
+                                       # like this might be the place where this is set on LO side^M
                                        formatField["page-number"] = docAttribs["page-number"]
                                except KeyError:
                                        pass
@@ -129,6 +133,16 @@ class SymphonyTextInfo(IA2TextTextInfo):
                                        formatField["line-number"] = docAttribs["line-number"]
                                except KeyError:
                                        pass
+                               try:^M
+                                       val = docAttribs["cursor-horizontal-position-in-page"]^M
+                                       print("horizontal cursor pos in page: {}".format(val) )^M
+                               except KeyError:^M
+                                       pass^M
+                               try:^M
+                                       val = docAttribs["cursor-vertical-position-in-page"]^M
+                                       print("vertical cursor pos in page: {}".format(val) )^M
+                               except KeyError:^M
+                                       pass^M

                return formatField,(startOffset,endOffset)

```

With these demo changes in place, the cursor pos in the page is printed to the NVDA log, e.g.:

```
WARNING - stdout (21:14:25.145) - MainThread (5692):
horizontal cursor pos in page: 1974
WARNING - stdout (21:14:25.145) - MainThread (5692):
vertical cursor pos in page: 1410

```

The position is in the unit that LO Writer uses: "twips", with 1440 twips being 1 inch.

Resolving patch:
diff --git a/source/appModules/soffice.py b/source/appModules/soffice.py
--- a/source/appModules/soffice.py
+++ b/source/appModules/soffice.py
@@ -16,13 +16,14 @@
 from controlTypes import TextPosition
 import textInfos
 import colors
-from compoundDocuments import CompoundDocument
+from compoundDocuments import CompoundDocument, TreeCompoundTextInfo
 from NVDAObjects.IAccessible import IAccessible, IA2TextTextInfo
 from NVDAObjects.behaviors import EditableText
 from logHandler import log
 import speech
 import api
 import braille
+import languageHandler
 import vision
 
 
@@ -275,6 +276,43 @@ class SymphonyParagraph(SymphonyText):
 	value=None
 	description=None
 
+
+def getDistanceTextForTwips(twips):
+	"""Returns a text representation of the distance given in twips,
+	converted to the local measurement unit."""
+	if languageHandler.useImperialMeasurements():
+		val = twips / 1440.0
+		# Translators: a measurement in inches
+		valText = _("{val:.2f} inches").format(val=val)
+	else:
+		val = twips * 0.0017638889
+		# Translators: a measurement in centimetres
+		valText = _("{val:.2f} centimetres").format(val=val)
+	return valText
+
+
+class SymphonyDocumentTextInfo(TreeCompoundTextInfo):
+
+	def _get_locationText(self):
+		try:
+			# if present, use document attributes to get cursor position relative to page
+			docAttribs = self.obj.rootNVDAObject.IA2Attributes
+			horizontalPos = int(docAttribs["cursor-position-in-page-horizontal"])
+			horizontalDistanceText = getDistanceTextForTwips(horizontalPos)
+			verticalPos = int(docAttribs["cursor-position-in-page-vertical"])
+			verticalDistanceText = getDistanceTextForTwips(verticalPos)
+			return _(
+				# Translators: LibreOffice, report cursor position in the current page
+				"cursor positioned {horizontalDistance} from left edge of page, {verticalDistance} from top edge of page"
+			).format(horizontalDistance=horizontalDistanceText, verticalDistance=verticalDistanceText)
+		except (AttributeError, KeyError):
+			return super(SymphonyDocumentTextInfo, self)._get_locationText()
+
+
+class SymphonyDocument(CompoundDocument):
+	TextInfo = SymphonyDocumentTextInfo
+
+
 class AppModule(appModuleHandler.AppModule):
 
 	def chooseNVDAObjectOverlayClasses(self, obj, clsList):
@@ -301,4 +339,4 @@ def event_NVDAObject_init(self, obj):
 		if windowClass in ("SALTMPSUBFRAME", "SALFRAME") and obj.role in (controlTypes.Role.DOCUMENT,controlTypes.Role.TEXTFRAME) and obj.description:
 			# This is a word processor document.
 			obj.description = None
-			obj.treeInterceptorClass = CompoundDocument
+			obj.treeInterceptorClass = SymphonyDocument


Test patch: